Wednesday, March 12, 2014

Coconut Headphones: Why Agile Has Failed

The 2001 agile manifesto was an attempt to replace rigid, process and management heavy, development methodologies with a more human and software-centric approach. They identified that the programmer is the central actor in the creation of software, and that the best software grows and evolves organically in contact with its users.

My first real contact with the ideas of agile software development came from reading Bob Martin’s book ‘Agile Software Development’. I still think it’s one of the best books about software I’ve read. It’s a tour-de-force survey of modern (at the time) techniques; a recipe book of how to create flexible but robust systems. What might surprise people familiar with how agile is currently understood, is that the majority of the book is about software engineering, not management practices.

So what happened? Why is agile now about stand-ups, retrospectives, two-week iterations and planning poker?

Somehow, over the decade or so since the original agile manifesto, agile has come to mean ‘management agile’. It’s been captured by management consultants and distilled as a small set of non-technical rituals that emerged from the much larger, richer, but often deeply technical set of agile practices.

It’s often said that ‘bad agile’ resembles a cargo cult. James Shore has an excellent post, Cargo Cult Agile, that describes how rigid adherence to the ritualistic forms of agile methodologies closely resemble South Pacific cargo cults:

“The tragedy of the cargo cult is its adherence to the superficial, outward signs of some idea combined with ignorance of how that idea actually works. In the story, the islanders replicated all the elements of cargo drops--the airstrip, the controller, the headphones--but didn't understand where the airplanes actually came from.

I see the same tragedy occurring with Agile.”

Current non-technical agile practitioners still don’t understand where the airplanes come from. They stand in their bamboo control towers with their coconut headphones on and wonder why their software projects still fail.

Agile has indeed become a cargo cult. Stripped of actual software engineering practices and conducted by ‘agile practitioners’ with no understanding of software engineering, it merely becomes a set of meaningless rituals that are mostly impediments and distractions to creating successful software.

well-ask-them-for-estimates

The core problem is that non-technical managers of software projects will always fail, or at best be counter productive, whatever the methodology. Developing software is a deeply technical endeavour. Sending your managers on an agile course to learn how to beat developers over the head with planning poker, two week iterations and stand-ups will do nothing to save spaghetti code and incompetent teams. You might have software projects that succeed despite the agile nonsense, but that would be coincidence, not causation.

Because creating good software is so much about technical decisions and so little about management process, I believe that there is very little place for non-technical managers in any software development organisation. If your role is simply asking for estimates and enforcing the agile rituals: stand-ups, fortnightly sprints, retrospectives; then you are an impediment rather than an asset to delivery.

Please don’t put non-technical managers in charge of software developers.

I don’t have an answer, or an alternative methodology to offer you, but here are some things that any software development organisation must address:

  • The skills and talents of individual programmers are the main determinant of software quality. No amount of management, methodology, or high-level architecture astronautism can compensate for a poor quality team.
  • The motivation and empowerment of programmers has a direct and strong relationship to the quality of  the software.
  • Hard deadlines, especially micro-deadlines will result in poor quality software that will take longer to deliver.
  • The consequences of poor design decisions multiply rapidly.
  • It will usually take multiple attempts to arrive at a viable design.
  • You  should make it easy to throw away code and start again.
  • Latency kills. Short feedback loops to measurable outcomes create good software.
  • Estimates are guess-timates; they are mostly useless. There is a geometric relationship between the length of an estimate and its inaccuracy.
  • Software does not scale. Software teams do not scale. Architecture should be as much about enabling small teams to work on small components as the technical requirements of the software.

Because the technical and motivational aspects of software development are so key, I’m very intrigued by the zero-management approaches of organisations such as Valve and GitHub. I thoroughly recommend reading the Valve employee handbook and Michael Abrash’s blog.  Maybe that’s the way forward? The original agile manifesto was very much about self organizing teams, it would be great if we could get back to that. In the meantime, the word ‘agile’ has become so abused, that we should stop using it.

bellware-bury-agile

81 comments:

David Chapdelaine said...

Awesome post. I totally agree.

Unknown said...

Learn about all good and important methodologies, and take what fits your character and team. Any religious treatment of any method is not natural. Thanks for the good read.

Unknown said...

I say take ownership of it and reclaim it for what it was promised to be! I whole heartedly agree that the fake-agile you are describing is an easy trap to fall into. As a product owner who used to be a developer and scrum master I fight battles every day to keep things properly agile. An empowered team that pulls from the backlog and uses best practices to deliver quality code.
We should come up with a name for the fake agile. How about FRAgile? F****ing Ruined Agile.

Kurt Häusler said...

Good rant, and I agree that what you have observed happens too much, and is not ideal.

It is a little unclear whether you are attacking "agile" or the non-technical manager.

One of the ideas behind the agile management practices, is servant leadership and self organising teams.

The agile community recognised the problems you raise, so that is why "agile", unlike traditional methods, fights against the concept of "non-technical managers being in charge of teams".

But even where managers have understood this (and at some point along the chain there will be managers that do important work that isn't technical), developers have sat there and said, we don't care about management, we don't care about contracts, we don't care about pricing, we don't care about how knowledge work flows through a system, we just want to program.

So what are you to do?

Non-technical managers don't need to know to program, but they do need to know something about the economics of knowledge work and servant leadership, they need domain knowledge, and extra soft skills like team leadership, negotiations, and dealing with customers.

Then they need to ditch the idea of "being in charge of a team". The customer is in charge, and if he is not technical I fully expect the programmers to complain just as much about the customer as the non-technical manager, but that is ok.

The team is second ranking after the customer. They do all the work, and are more important than the managers.

The manager should definitely be a third place role. They should simply introduce the customer to the team, do the paperwork, keep an eye on non-technical things, get out of the way and serve the team.

(Of course then the programers will whinge that they have too much customer contact and just want to hide in the basement and program, and start pleading for some manager to be in charge of them again)

Anonymous said...

Don't forget that there are technical managers who are simply outdated! I've been in and seen too many situations where and out of touch manager is dictating practises and implementing plans they don't understand.

Jonathan Swift said...

I have been at first contemplating, then looking into then actually studying Cult Phenomena since 1974 or maybe 1975, when from time to time I used to read about the weird news coming out of Communist Cambodia in Jack Anderson's editorial page columns.

While an arch-conservative, Anderson was really quite lucid, and thoughtful, as compared to today's conservative columnists.

He at first reported that quite suddenly and completely out of nowhere, all of the Cambodian children were turning their own parents into the authorities.

A month or two later, everyone in the entire nation who wore eyeglasses disappeared overnight.

Later I started learning about NAZI Germany.

When Jim Jones managed to move nine hundred cups of Cyanide-laced Kool-Ade from his Jonestown, Guyana refreshing sports drink stand, I had a pretty good idea as to how that came to pass.

I learned how to pull that same stunt off myself when I joined the Human Potential Movement by enrolling in the Lifespring Course in Los Angeles over the Summer and early fall of 1984.

(LA LP19 - that it, the nineteenth session of LP's Leadership Program, which was free of charge however we were highly pressured to enroll five other people. I myself enrolled three fellow Caltech students, two of whom went completely out of their trees, the third had a real hard time yet managed to graduate, however he is no longer in Science.)

When I started pointing out to other members of "The Caltech Community" that just maybe it was bad for us that no one from "The Outside World" ever so much as set foot on campus - not even the Pasadena Police! - I was beaten senseless then expelled for sleeping on a couch in a Ricketts House hallway after the Institute's Master of Student Housing told me not to.

I learned about how all manner of cults work - including cargo cults - as a result of taking Cultural Field Anthropolist Stuart Schlegel's most-excellent "Anthropology of Religion" course at the University of California Santa Cruz.

Dr. Schlegel later wrote that I had written the most-moving term paper he had read in twenty years, just twelve rather spontaneously composed pages entitled "Surviving Suicide", which informed "Stu" - as he preferred to be called - that his discussion of Kant's Construction of Reality enabled me to overcome quite profoundly suicidal depression.

The world recoiled in horror as a result of the Spring 1997 San Diego mass suicide of the Heaven's Gate UFO Cult.

They all thought that comet Hale-Bopp was an incoming alien spacecraft, coming to destroy the Earth. While Heaven's Gate was adamantly opposed to suicide, they considered themselves not to be dying, but to be surviving the oncoming onslaught, when they "Shed Their Vehicles" by eating Phenobarbital-laced Applesauce and Pudding.

Drop "Marshall Applewhite" into YouTube.

If you want to know how cult leaders work, watch Applewhite's eyes closely. Note that his eyes are wide open and he never blinks. That's a rather extreme form of NLP - Neurolinguistic Programming. While it is easy to resist if one knows what one is doing, if one practices NLP on unsuspecting people it is quite easy to manipulate their minds.

Whereas most were aghast because they didn't understand, I myself damn near checked into the nuthouse because I understood all too well, and so went almost off the deep end myself.

What concerned me wasn't that the same would happen to me, but that all of the Heaven's Gate followers other than Marshall Applewhite were, to the very best of my knowledge, perfectly happy normal young people before they came to work as web designers for the Heaven's Gate web design consultancy.

It was to explain to others, that all is not as it seems, the reality is not concrete but fluid, that I commenced what been now been seventeen years of writing for the most part on the topic of mental illness, not just my own but that of others, as well as anthropology, sociology and the like

Blaine Reeve said...

Don't work for companies where your boss sucks, and I am saying this as an executive and software manager. Just leave. A well formed team is all that is necessary to hit deadlines and meet customer expectations. Their success has nothing to do with the development methodology, if they are not a cohesive and talented team, they will fail under any methodology or under the slightest customer pressure. My job is to build a great team, not necessarily filled by A+ players, who can get along and work together. I will take 4 junior guys over 4 senior guys who think they are software geniuses (clearly software is a deeply technical discipline, not the playground of neophytes..hahaha) who can work together. My job is to find the roadblocks, including asshole developers, overbearing peer executives driving a deadline agenda, and technology/methodology evangelists and reduce or remove their daily influence on the team that is well performing otherwise. Do I let non-technical Project Managers have access to my team? Of course,
At the end of the day, very few professionals, developers included, have the skill and capacity to do breakthrough work, lets not paint them as gods that need no supervision, blocking, tackling, or code reviews. Could managers improve, both technical and non-technical, absolutely. Its a skill just like software development, pattern recognition and problem solving, some can do it, some can't.

Anonymous said...

But what's the alternative ? It's easy to criticize something , without an alternative.

Julian Kennedy said...

Fantastic post! I have seen this "cargo cult" too many times.

Anonymous said...

Got out of a six month agile catastrophe recently. Non technical agile cult management put a completely unqualified former tech support worker as our "scrum master." Everyone quit or was so pissed off they simply stopped working. The product was essential to the success of the company...it will probably take another year for them to ship it.

That product really needed good product management and design, which we didnt have. Sprinkling magic agile fairy dust on a fundamentally broken team with bad players and missing talent wont fix anything. No amount of process can make the wrong people into the right ones.

Its sad but you are dead on. There is no room for non technical people in this space. The most successful teams I have seen made use of "tech leads" who were engineers capable of running a project. Good luck hiring people that good who want to work at your dumb company run by lawyers, sales and marketing. Anyone capable of being a tech lead is probably ripping down double six figures at google.

Anonymous said...

"Please don’t put non-technical managers in charge of software developers." - Never a truer word spoken!

Chris Jones said...

I feel like this is a strawman argument... The role of non-technical manager doesn't exist in Scrum, XP, Kanban or any flavor of agile that I've ever heard of. So arguing that "Agile has failed" and then describing a not very agile process seems a bit misleading.

In our organisation agile principles mean that the team is empowered and face-to-face communication with non-technical stakeholders is highly valued.

I'd suggest anyone who is struggling to deliver effectivly go back to basics and look at the underlying principles of the Agile movement and try to understand how practices like iterations, stand-ups and retrospectives support these and take positive steps to improve your work environment for technical and non-technical folks.

Anonymous said...

This is the biggest and most self pretentious pile of crap I read in ages, and only self indulgent, bad coders could be so condescending with themselves. Software does not scale? Crap - it's because you can only focus on the nuts and bolts and not the overall picture. A project managers needs to have some degree of understanding of the technical complexity - but his/her main role is to deliver a product within a defined budget and timescale - all things that programmers cannot do. Someone always pays the programmers' salaries - and yet you seem to refuse and deny this reality and pretend to be self sufficient gods. Do grow up, people - you eat and pay rent like anyone else. Cristiano

Anonymous said...

Nice article!!

DE said...

http://www.halfarsedagilemanifesto.org/

Jan Wedekind said...

Well said!

Jan Wedekind said...
This comment has been removed by the author.
Jan Wedekind said...
This comment has been removed by the author.
Andrew Jerrison said...

Whenever I have encountered 'Agile' it does always seem to be a management tool, without real understanding of the technical side. It is not usually implemented in the favour of the engineer. As you say, this is probably what has led to its negative image in the eyes of those who it is meant to help!

It seems to have been hijacked by an unholy cabal of non-IT literate managers who think they've got a way to measure performance of their 'resources', and the kind of developers who are happy to sit at a desk code-bashing like a robot all day.

There doesn't seem to be much room for the human side of things such as a varied job such as design work and customer liaison, or career advancement, or even swapping team/jobs easily (thanks to the 'no docs' rules). Whatever happened to the old 'software engineer' role, i.e. senior technical people who managed both the whole project and the people? It kind of worked you know.

Moan over. :-)

Dave Aronson said...

"If your role is simply asking for estimates and enforcing the agile rituals"... then you can be replaced by a small shell script. And then we can delete it.

Anonymous said...

Great comment, Blaine!

Patrick McElhaney said...

Let's be careful not to look at all the failed "agile" projects and assume airstrips, controllers, and headphones prevent planes from landing.

As Chris Jones said, there's no place in the agile practices for a "non-technical manager", or even a manager. Teams are self organizing.

Scrum can be considered a "starter kit" -- practices that have proved in many different teams to be highly effective. Over time the team is supposed tweak the system through the practice of kaizen (continuous improvement).

Unfortunately, the "agile" most people experience today is the same old management practices with new labels: status meetings are now called "stand ups." Schedules handed down from above are now "estimates." Big design up front and long bug fixing cycles don't have a place in Scrum, so we've added "Sprint 0" and "hardening sprints".

The retrospective, IMHO the most valuable of the four Scrum rituals, is usually the first to be tossed out.

Believe it or not, cargo planes really do land when there are people on the ground with real radios and a plan.

Peter said...

Awesome post. reading this made my day

Mike said...

I come from the consulting world and in that environment, a good non-technical manager is a huge asset. What does a good non-technical manager look like? They trust their technical team (and especially the technical team lead) to deliver based on design and to monitor the development process and report up to them when things are going off the rails in terms of time.

They also control scope. All too often development projects come in over budget and time because scope increases go out of control.

I've always been suspicious of agile due to it's lack of project control. Little long term planning or design. In a consulting environment where the customer is paying for the product based on the hours it took to develop, these controls are critical to success. It always seemed that the agile approach was a reaction to programmers not enjoying being managed.

I do agree that it seems the superficial elements of agile (two week sprints, stand-ups, etc) were adopted thinking they were the magic ingredients for success. In fact, every successful project I've been on has been the result of talented people doing their job and recognizing where they have to adapt the process they use to the unique situation they are in.

4193cf56-aad9-11e3-9b49-000bcdcb2996 said...

I'm also interested in the management approaches of Valve and Github. But I believe that's much closer to political analysis than software engineering.

Valve's structure is an anomaly. If the software industry was substantially structured around workers making their own decisions, that's a direct affront to capitalism.

I believe we should pursue structures like this, including starting new cooperatively-run enterprises. But we should not pretend this is simply a matter of methodology: it's at the very core of politics.

Anonymous said...

Agree with your bulleted points, mostly, except for the bits that attack deadlines and estimates. While I nod my head in agreement that estimates are guesses, what alternative do you have? Are you against all estimation or only against estimation of long-timeline efforts? And hard deadlines, they don't guarantee poor quality software. In fact, they can prevent poor quality software. If a product manager asks me for X by date Y, and a customer won't buy unless it's there by date Y, and I know we can't deliver by that date, I can go ahead and focus on those things that have a chance of success.

Anonymous said...

Well said. Techies will get this i9nstantly, non technical managers will no doubt huff and puff and spout justifications.
This stuff really should not be controversial…and it should be standard practise but the "culture" of non-functional IT (ie. Architects and non-coding managers) has now conquered most office environments. Not only is doing software badly very profitable for some, it is also a much easier career than actually working.

We clearly need less coxswains for every rower and less peripheral BS, however programming well is *hard* and it appears that and easier career path is to forgo any technical competence and just stay "high level". Not only is this much easier, it allows time for the important grooming activities so critical in a multi-layer primate hierarchy. Techies appear to be exchangeable with cheap imports and can be linked directly with stuff ups whereas once in a high level "team" clique you are relatively safe. More pay, more safety, less work...with a bleak, soulless existence, but that's a trade off many will make to pay the mortgage.
Automation's wealth effect has led to many adult day care roles…this is just another example.

Ross said...

Reminded me of Dave Thomas post http://pragdave.me/blog/2014/03/04/time-to-kill-agile/

Cliff Berg said...

I agree completely with your sentiment.

But interestingly, the things that you say that "any software development organisation must address" are all core agile ideas.

So agile - as a set of ideas - is not the problem.

Agilists will then say, "The problem is that these organizations are not really doing agile: they are 'doing agile' - not 'being agile'".

No. That is the wrong way to look at it. The truth is that while agile is a great set of ideas, the agile community has not answered the very important question of how agile should work in a large, complex, traditional organization.

Just saying that a big company/agency needs to be truer to agile value is not saying anything: one might as well say that the reason we can't have world peace is because world governments are not more trusting. Saying that accomplishes nothing.

The fact also is that large organizations cannot adopt agile that way that it is practiced in small ones.

But there is a-lot of truth to what you say about too much focus on agile ceremonies. Most of the ceremonies that agile coaches pay so much attention to don't really matter. What matters is servant leaders. What matters is small teams. What matters is customer involvement. What matters is having small demonstrable units of work. What matters is having smart people on the team, or at least people with the right skills and a willingness to learn. What matters is having someone who is technically astute making sure that everything works as a whole. All the ceremonies don't matter.

Anonymous said...

Agree with this comment from the article and many others posted, but - most of you 'dev' guys just aren't that good. 1 in 5 or maybe 10 can develop, most can barely code. You obviously have your own headphones on and they are up your arse...
The skills and talents of individual programmers are the main determinant of software quality. No amount of management, methodology, or high-level architecture astronautism can compensate for a poor quality team.

Anonymous said...

Agile is dead! Long live Agile2!

Anonymous said...

How true !! Certainly the projects success depends on capability its technical team

Anonymous said...

What a horrible article. By extension of your logic we would have to say that technical managers should never be in a position to run the business. I've seen dozens of "non-technical" managers who have done excellent jobs managing software teams (Agile and not). The failure of Agile, and waterfall, and XP, and whatever, lies not in the methodology and not with non-technical managers. It lies squarely with the developer not taking personal responsibility for producing quality code that provides business value.

Anonymous said...

"Software does not scale." -- WTF really?

"Estimates are guess-timates" -- when last dit you look at the definition of "Estimate", http://lmgtfy.com/?q=Estimates+define

Please understand 'agile' and the words you use before you say 'agile has failed'

Anonymous said...

http://www.dummies.com/how-to/content/the-12-agile-principles.html

Is that what any system strive for?

Do I need to say more?

Felipe Brito said...

I have to agree with others that this feels like a straw man...

The recently published 2013 State of Agile Report states (in page 7) that "respondents reported year-over-year improvements in ALL areas measured" from adopting agile methodologies and "73% reporting faster time to complete a project"and "92% reporting improved ability to manage changing priorities" (page 8). And in page 9, "51% of respondents said that the majority, if not all, of their agile projects have been successful."

You may claim that the purveyor of the numbers usually has a vested interest in their interpretation, but you may find similar numbers in various others reports from Forrester and Gartner...

Agility is now a success and the teams and companies that have not embraced it yet are laggards.

I started with RUP back in 2000. I wrote hundreds of use cases, created an infinite number of sequence diagrams and led dozens of projects from inception through transition. We had successful projects (and unsuccessful ones too), but we were always asking ourselves whether there were leaner and more valuable ways of bringing value to our clients. We were CMMI3, then CMMI5. We learned to be open-minded, inspect, adapt and encourage our organization to conduct experiments and gather lessons from the results. In 2006, we started our agile journey and two years later we were already a 100%-Agile company - full enterprise-wide adoption: from development and maintenance teams, through operation, human resources, marketing and sales. We use Scrum and Kanban and now have lean principles in our DNA. Our success rate is 97%.

We have technical Scrum Masters that obviously started their careers developing and understand the nuts and bolts of the code, but we do have scrum masters that work greatly in solving blocks and mastering communication with the business, guaranteeing integration, providing coaching, cadence and supporting the team without a technical background. It is possible.

I ask you: what is your vision for the future? How can we make sure that agility gets to the next level? How can we build a new Boeing jetliner in an agile way for example? How can we break the silos of large organizations and engage dozens or even hundreds of people in a large program? How can we make sure that one agile team of seven people learn with lessons from other teams and avoid waste? And how can they all work together to build something bigger - keeping the values and principles we so very much respect?

Best!

Marc Ziman said...

I think the problem is the emphasis Scrum puts on project management and the associated rituals. XP focusses on technical practices (TDD/Pair programming etc). Without strong technical practices the project management aspects are next to useless. Martin Fowler calls this Flacid Agile. Too few developers seem interested in good technical practices in my experience. This is a major impediment.

Eryts said...

Great post and I'm living a similar situation in my project.
This is my first "Agile" project and I can see there are many deficencies on how we are doing things. Probably is the management that always want it quick and dirty or is the lack of knowledge from the whole team on what "Agile" means.
The thing is that we should start thinking more on how to make our developers more happy instead of how to make them more efficient, because want we really want are great resources doing software, not robots producing code. If we want robots, let's code some robots to do it for us.

Anonymous said...

Whether you develop software or build skyscrapers, it all boils down to good project management. A construction project manager does not need to operate a payloader to build architecture wonders nor does a software pm need to know Java to the detail to develop a breakthrough application. Projects fail because of a number of factors and it's surely not because the Manager does not know how to operate a large hydraulic excavator nor code classes or methods. Compared to construction, software development is still in its infancy.

ForceCrate said...

Agile defenders use management metrics to defend agile?! This is another problem in itself; the belief that you can accurately measure human activity! Non-technical managers love to measure because they are incapable of judgement. Agile encourages this stupid behavior. One day I may see a deadline disappear because of programmer input but I won't hold my breath waiting.

Anonymous said...

I agree with your view of what Agile has become in many organizations. Partly I believe it is our own fault. As developers, we want to write code not "organize" ourselves. So someone with less technical knowledge is appointed SCRM Master. (Why waste a good developer, right.) So we abrogate our responsibility to organize our team.

The other issue is one of management. Someone, not developers, will be managing the company where we write software. CEOs, CFOs, accountants and they will all want to know what the development team is doing, and they should. It's their job like ours is to write code.
So there will always be a "new thing". (waterfall was new once) and it will work well until it becomes form over substance. That takes diligent, consistent, non-code related work on the part of all developers.

Martin Fulop said...

According to my experience (SW dev, technical mng), the success of an software development project depends on far more that just the methodology.

It depends on the environment, which is used to deliver the product.

Environment
With environment I mean all methodologies, human skills, human nature, tools, offices, languages and anything you can think of having an influence on the software development project.

I see the key to success in creating the environment fitting the software purpose you want to develop and deliver, and then continuously improving the environment.

Responsibility
Blaming non-technical management for failed "agile" software in general is actually denying your own responsibility. The responsibility does not originate from the "agile", it comes with the contract you have signed for delivering the job. Some of us tend to forget that sometimes.
In agile the technical roles are responsible for the technical richness of the environment. So if you feel, that it's all about meetings, sprints, pokers and all the technical practices are lost than you have failed to keep your responsibility. To be honest, you cannot wait for a product owner to tell you i.e. that you should set-up some continuous integration server.

Constant improvement
Everybody is responsible, thus if anyone sees an impediment, than he should point at it. If impediments are ignored, than you fail to improve your environment and in the end fail to deliver the product. That's continuous improvement of the environment. Actually, it's not an invention of scrum or agile, it's the common strategy of survival in a constantly changing world and should be practiced in any methodology (even those waterfalls).

Summary
If you feel, that the agile approach combined with all the kinds of meetings and reviews and missing technical richness and talk and talk and talk is making your work ineffective, then
1. it's actually you in the first place, who is not fitting the specific agile implementation,
2a. you could try to find the impediments in you and improve your self,
2b. you could look for something more fitting to your nature (incl. the amount of responsibility you are willing to keep),
2c. you could keep things as they are, writing blogs to let out the stress,
(this is not a test, it's just a choice ;)

In fact, agile principles are NOT fitting for every software, every customer, every team ...

Misha said...

I think the single best thing about agile perspectives in software development is that it encourages frequent and direct connections between a software developer and those for whom the software is built. Those involved in direct and frequent connections should be: decision-makers for priority questions; functional experts; and technical experts. A manager without functional or technical expertise would have nothing to offer and could only get in the way of progress.

philipoakley said...

Can I recommend "The life Cycle of a Silver Bullet" by Sarah Sheard from a few years ago http://www.crosstalkonline.org/storage/issue-archives/2003/200307/200307-Sheard.pdf.

It's still a great article that fully reflects the thesis of the blog post.

Allan Sitte said...

Dude!!
Did you read my mind while I was riding in an airplane or metro/subway?

Excellent post! +1 times 5 worthy.
It was almost as if my thoughts just popped onto the web page.

:-)

ScarvedOne said...

Every decade, another new methodology gets turned into a cult and run into the ground. I know! Let's invent some more!

Hey, Rocky, watch me pull a rabbit out of my hat!

Mike said...

Agile is no different from any other business, manufacturing, technical or management "methodology du jour". Problems arise when the buzzword is adopted but the methodology is inadequately adopted or perhaps even applied where it does not belong.

Anonymous said...

Huh. We've been doing Agile for about three years now after 12 years of basically unstructured ad hoc development practices. The productivity increase is massive. But we don't have non-technical managers leading technical teams. The non-technial (product owners) interface with the tech lead who organizes the teams. All that said, we are not rigid on any aspect of it, but have found organically that daily standups, retrospectives, and, particularly, shorter sprints (for us usually four weeks) has resulted in massively increased productivity. We went through formalized "scrum master" training phase, but have largely abandoned that as voodoo training. As others have said, use what works for you. Agile can work, even the more modern agile. Just don't force it to try to work if it isn't for you.

Steve Bjorg said...

There is some truth to the rant, but leaving programmers in charge of a project is just as bad (as evidenced by Fred Brooks who few would challenge on technical chops). Someone needs to understand the objectives of what the product needs to accomplish and that person must be at some distance from the coding effort to not be embroiled in it. Otherwise, the outcome is an over-engineered, poorly market-focused product that is a jewel of human ingenuity that serves no particular need.

Kevin O'Shaughnessy said...

Some very good comments here that I agree with. Recently seems to be even more fashionable than usual to criticise Agile and I think it is often a good thing because, as explained in this article: http://everydaylean.info/2013/09/heart-agile/ the heart of agile is all about the pursuit of better ways of working, not about any specific practices or methodologies. Most flavours of Agile are not methodologies for example Scrum is an empirical process framework, nothing more or less.

Yes, there are countless companies that adopt the agile buzzwords but don't have a clue how to manage their projects or their staff. The same applies to waterfall etc.

This is the only thing I can recommend for everyone:
http://alistair.cockburn.us/Oath+of+Non-Allegiance

Anonymous said...

It appears to me you do not know Agile well or have worked with a poor implementation of Agile. The key here is collaboration. Collaboration with your end-user, architects, testers, developers. The technical people write the code, the clients identify the problem.

This sounds like a bitter rant fem someone stuck in the past.

Anonymous said...

The author identifies a real problem in organizations where they try and use the 'Agile process' as a golden hammer, without the right people and in the spirit of Agile. If a technology business places leadership in folks (managers, architects, engineers, people, insert title here) who do not understand technology, the results will be poor. Agile does not fix lack of vision and incompetence. In large organizations, Agile is another 'fad' like Six Sigma that has been evangelized and commoditized. We can only imagine how much $$$ is spent on training and consulting just to quote learn the process quote. But then the spirit of Agile is not there- estimations become deadlines, poor code is written, re-factoring does not happen. BUT the processes like two week sprints and retrospectives occur that act as checkboxes so the business can claim it is up and coming and lean. Keep in mind this blog post is most applicable to big, entrenched companies that in many cases have corrupted the Agile philosophy.

Unknown said...

great retrospective. Let's use that to improve the next iteration.

Anonymous said...

I'd like to suggest a twist on the "project management" aspect of this discussion. I find that whether the project manager is actually technical or not may not be that critical (although I agree having a technical PM is an asset in most cases). I find that the ability of the PM to manage *expectations* can make the difference, and I'm referring to ALL expectations here - those of the client, those of the development team, those of the management, etc. as they relate to the project requirements and objectives. A good project manager should be able to detect the "larger" expectation gaps and address them even before the contract is in place. As the delivery takes place, the "smaller" expectation gaps should be addressed before issues or actual conflicts arise. If expectations are adequately managed throughout the project, then the project is very likely to unfold well, and be a success for all involved. Competence of the delivery team does matter a lot, but the methodology might not be that important if correct expectations are set and maintained with all stakeholders throughout the execution.

PM Hut said...

I'm not sure that you can say that Agile has failed. I don't think it has. I don't think it's a good methodology (in fact, I think it's bad, it's leading to Cargo Cult Project Management - see: http://www.pmhut.com/cargo-cult-project-management ), but I don't think it's failed.

Failure means something isn't selling, and Agile is selling like hotcakes. Every single company nowadays wants to be Agile.

I think Agile has won considerably, considering it's just a set of project management practices that were passed as a methodology that nobody has actually been able to manage a real project with.

Tero Väänänen said...

Many of the concerns raised here sound true to me. I went through this same exercise a couple of years ago.

You get lost in the backlogs, points, and measurements and other prescriptives. You almost get a licence to suck by doing some Scrum dance. As long as the points come out right, we're good!

Harald M. said...

I appreciate your concerns. But you don't seem to have a clue about humans and humanity. It seems you are a "real programmer" - as opposed to someone who is concerned for all people.

(For the record, I have been a programmer AND a manager in the last 20 years; I know "both sides" - but they are NOT "sides", as you want to make us believe).

Harald M.

Gabriel Batista said...

I disagree the fact that a manager have to be a developper to be a good manager.
I think, that like other businesses the manager have to discuss with the specialists to have the good indicators.

I agree that few developpers know how to measure technical debt, and that the main matter.
We have to promulgate the methods and tools which measure technical debt.

Tommy Bryntse said...

Good one, especially the first part! I don't agree that you can't lead a software development project without technical background tho, as long as you make sure to trust the team on the technical decisions and adhere to any feedback they provide to the plan.

I wrote an article about a year ago called "What is agile?" that discuss just this that agile is now only about processes and tools even tho the first amendment in the manifesto is "Individuals and interactions over processes and tools".

You can read it here if you'd like.

Anonymous said...

I have to call BS on the "No hard deadlines" requirement. Everyone has deadlines. When business needs a product in time to satisfy a contract or other need, the deadline is real. Are there a lot of deadlines imposed that are arbitrary? Likely, but coddling programmers as coding divas, with no responsibility for the success of the business, leads to the programmers having to find new jobs and blaming the business for it.

Sai said...

@Mike - "Software does not scale." - what did you mean by this line ? (I am really asking, not pointing out anything).

Mike Hadlow said...

Sai,

"Software does not scale." I agree it's quite a broad comment, and patently untrue because there are plenty of large scale successful software systems out there. This is a blog post in itself, and one I've been meaning to write for a long time, but I guess what I'm saying is that as a software system grows, you can't just keep writing more and more code in the same way and expect it to scale linearly.

There are all kinds of scale effects that manifest themselves, and beyond a certain size, software becomes very hard to understand and maintain. You need to adopt architectural strategies to divide the software up into manageable components with clear boundaries and contracts. Software teams do not scale well either. There's a very important human aspect to software architecture; it needs to be structured in such a way that teams can be small. The worst thing an organization can do is throw more developers at a large monolithic system.

Any successful leader of a software team needs to address these issues and because they are technical, it simply reinforces my belief that non-technical managers are a waste of space.

Anonymous said...

Well, the truth is always in the middle. Managers still do destroy projects by micro-management. And developers still destroy projects by very bad technical practices.
The idea behind Agile is the self-organizing team : a team that learns even from each other and improves continuously that way. Every advancement a team member achieves is automatically communicated and duplicated to the others. And yes, this team defines with its Product Owner its own deadline and budget in negotiation with the client. They deliver estimates, provide information about what high priority features can be delivered in the requested remaining time with the offered amount of money. They deliver the solution nit by bit, get feedback and correct their course, where necessary. This all works automatically without the need of any project manager. A PM is truly just an obstacle and an unnecessary overhead for such a working team.

BUT - is every team able to become such a self-organizing unit ? NO !!! No, no, and again no !
And that is the problem.
Many people (including SW engineers) are just happy to be told what to do, they do their 9-5 job, and are in essence more interested in other things. And many managers cannot become servant leaders.

In my experience it all boils down to : trust !
If the team can show that they can deliver what they promised, it will be easier for the managers to be able to "retreat" to a function of pure servant leader. The team earns the managers' trust.
And if a manager gives the team some credits at the beginning and is not micro-managing for a while, the team begins to trust that they can actually be self-responsible and self-organized, that they are in fact truly responsible for their own fate. That provides a totally different level of motivation.

But again - it will never work with everybody. It is the combination of the right personalities in the right environment that makes it successful. Not the practices themselves (most of them should be used in traditional/waterfall environments too - the practices are good, but they do make a team or an organization "Agile").

Filippo De Luca said...

I love this post, it summarize my toughs that I don't have the "balls" to say.

Jonathan Harley said...

Don't confuse "Agile" with "Scrum"; Scrum is a type of Agile, and in the original versions did not specify any technical practices (ala XP) but now apparently does.

The "Agile" that Uncle Bob refers to in his book is XP, not Scrum (especially not the original Scrum).

It makes no sense to declare Agile dead since so much of what was intended (more craftsmanship, smaller increments of delviery, focus on testing, etc.) are pretty much de rigeur in the software development industry now (though maybe not as much in Enterprise SD).

Anonymous said...

Well this qualifies as a rant for sure. I have to agree with many of the points. Many articles like this conflate the issue of the method with the issue of the implementation.
That is a bit of an issue here as well.
Altogether it's very amusing reading the posts of the non-technical defending non-technical management.
Well we all have mortgages I guess.
....JW

Paul Moores said...

Great post, more people need to feel empowered enough to stand up and speak their true feelings surrounding this thorny issue rather than being frowned upon as anti progression/trouble makers/stick in the muds/etc...

I have for a couple of large companies recently who call themselves agile, use extreme programming techniques etc... and just don't deliver any product.

This whole agile drive appears to have been hijacked by new hipster-esque developers out to prove themselves as super developers. An excuse for little or no documentation; too much time focusing on purism and writing tests that are often far more complex than the application they are testing and harder to maintain, although no extra time is allowed for the creation of such tests, like they appear by magic??????

Cannot wait for the tried and tested to come back around where what matters most is delivering what the customer has requested.....

Ignatius said...

I would just like to caveat your warning about not using 'non technical managers' to manage software projects. A good non-technical manager will recgnise that where the expertise in the team lies and will trust them to make the right technical decisions. S/he will not use the agile framework to beat the team but will make sure the framework supports what they are trying to achieve.
Your concern appears to be more around the 'command and control' behaviours of certain managers (technical or non-technical) which is the real problem with agile implementation in the real world. Without addressing the cultural/behavioural shift required in an organisation, there will be a complete mismatch between management and development that cannot possibly be fixed.

Shubhangi said...

Agile has not failed, its just the way you use agile is flawed. Actually no software methodology in complete, they all need bits of tweaks and adjustment.

Anonymous said...

Yep, it's self-pretentious crap, people who think that Dilbert defines how they should think. Agile has become an excuse and a refuge for the prima donna developer who doesn't want to commit to a deadline, refuses to estimate when they'll be done (until they're done), and sneers at doing anything else besides heads-down code, no matter what the need. Fire those zealots; they just drag down an organization far more than the much-sneered-at "non-technical manager".

Anonymous said...

Yes Agile has been subverted. Too many wrong people involved who subjugate programmers and programmers just accept this not standing up.

bryan.c.oconnell said...

Mike, thanks for your thoughts. I've collected some similar articles and shared some comments here: http://goo.gl/Gxu5qs

Anonymous said...

So I think a better "world peace" analogy to large corporations doing agile is Putin spending billions of dollars on the Olympics and annexing Crimea the next week. And then blaming peace activists for not showing him how to deliver on the promise of the Olympics.

I think the real problem with the community are the unaccountable companies behaving as petulant children, trying to blame the community for their software boondoggles. Especially the large 1950's corporations with their 1950's culture that refuse to acknowledge that they have to start acting like real software companies.

Jordan said...

Alright, cracks fingers, lets clear some of the air here.
This is clearly written by a developer who had a bad day. I get it, I even agree with some of it, but by no means is "Why Agile Has Failed" a valid blanket statement for the entire industry.
Here is a metaphor for you. If I buy a Honda Civic with my own money, out of my own pocket. I am going to be more prone to saying good things to my friends about Hondas. "Hondas are the best, most reliable cares for the money." Something to that affect, because after all, I made the decision to spend money on that particular car, and I'm a pretty smart guy, right? Therefore it is a logical decision everyone else should consider.
Now, lets substitute this argument into a similar path of thinking. "I have had bad experiences with Agile on my projects as a developer being managed by non-technical managers". I'm an industry professional, and I'm pretty smart, right? Therefore my experience should / will be the same for everyone else in the same career as me, and so they should take my statement of "Agile has failed" as the truth.
It is bullshit, and this is coming from a technical product/project manager and former BA with a background in development / network engineering.
I always get frustrated with these kinds of arguments. What purpose does it serve if you have no alternative? In what world has any of us been involved with a project that has a process that EVERYONE, including the devs, agree on? I'll wait for someone to raise their hand. No one? Ok.
I have never, repeat, never, been at a client's organization when they have implemented agile in the way it was intended. The fault doesn't lie with any one person, or role, as this article makes it out to be. In fact, it lies on a ton of different factors, from software developers building features that are unnecessary in an order not aligned with the business goals to non-technical managers dictating from Mt. Olympus, to improper requirements being gathered and communicated.
All of this is not specific to Agile, it is specific to humans. Humans, in the structure of an organization are fallible. They make mistakes, they change their minds, their ego dictates their actions. All of this cannot be solved with the removal of roles, or the implementation of any framework.
Further, if Agile was failing, and wasn't considered valuable by organizations, than how come Agile consultants have a roof over their heads? How come they get paid real money? Why do we continue to see article after article over how one framework is dying over another, year after year? Why is it that the majority of them are coming from frustrated developers and not the people who sign the checks?
Just look at stack overflow for the example. Zealotry at it's finest. This language over that one, this framework over the other, if you learn x you're not a real programmer, if you learn Y first you're an idiot, etc... Every GOOD developer I work with writes all of this kind of shit off completely. Every BAD developer I've worked with (keyword worked) relies on these kinds of thoughts to get through their day. Maybe they write an article about it.
At the end of the day, there are always going to be people who think what you do is useless; yes, even you developers (you're viewed as a commodity). Everyone thinks they can the chief in the tribe, and everyone thinks their past experiences will follow them into the present and then the future. But the facts really are, there are people who are good at what they do, and their are people who are bad at what they do; in all roles, both technical and non-technical, Agile consultants, developers, managers, etc...
So can we please, please, stop beating this dead horse by firing the people who suck, hire/retain the people who don't, and get them all to agree on the best way to get things done and stick to it until things need to evolve?

Anonymous said...

http://zenexmachina.files.wordpress.com/2012/07/standish-chaos-report-waterfall-v-agile-success-2010.png?w=479

Agile, clearly a failure.

Matthew Thompson said...

some of us non technical coaches still actually get agile. the rituals are only the means to an end. it is the empowerment of the technical teams that drive product improvement. agile works well when done well.

Anonymous said...

Where are those 10x and 100x improvements eh? Snake oil at its finest...

Charles Ferguson said...

I think you raise some valuable ideas but some of them I question:

* non-technical managers: my 20 yrs as a dev doesn't gel with this. The best manager I ever had was non technical. But she was very adept at honing in on exactly what she needed to know about the domain (but not more). Excellent people skills, encyclopedic domain knowledge, never scared to go toe to toe with management on behalf of her team or to push back against bs requirements. And a number of my weakest managers have been highly technical.
* I will back the output of a solid team with a mix of high-to-medium-ability members against a team entirely of genius assholes any day.
* James Shore's article says nothing about Agile failing, just about the spread of cargo-cult Agile. Just saying.

Anonymous said...

I have seen just about every crazy thing you can imagine happen under the guise of "Agile" process. Like one week sprints followed by the weekly production release. Like two week sprints that encompassed major pieces of functionality that were time-boxed then forgotten once the next sprint started. No matter the status of said functionality. In that project, I once worked 24 hours straight fixing 18 defects the weekend before a release.

I also once saw a manager fire an "agile coach" who was getting paid $125 an hour, and the project took off like a rocket after that.

I don't have a problem with non-technical managers - I've had some good ones. What made them good? They took the heat, and stayed out of our way. Because they knew we were good, and that we delivered.

However - there are people out there with no background AT ALL in software development who are getting some sort of "SCRUM certification" and schmoozing their way onto development teams. That is never going to work in my opinion. I once worked with a guy that fits that description, who didn't understand basic, fundamental concepts about IT in general. It is a real head-scratcher how that could ever happen, but it does.

There is no magic bullet. You have to have good people.

Anonymous said...

The first sentence says it all "They identified that the programmer is the central actor in the creation of software, and that the best software grows and evolves organically in contact with its users."

The customer is the central "actor" (another agile term i hate) in the entire process. The programmer is just a tool to achieve the end result. Wake up and get some common business sense. Also, software isn't grown organically like a plant. Software needs to be precise and written preferable once based on precise rules (aka specifications), smart coding and bullet proof testing. It's all about precision not iteration.

Anonymous said...

Consider me for a moment.

Years of experience in developing software solutions, on a variety of platforms - working on small teams and large teams from time to time.

Whether it was waterfall or something else, I always had the curiosity and motivation to approach "the business" whenever I didn't understand a requirement, or if it was incomplete or simply if my intuition told me it wasn't right. I had good managers, technical AND non-technical, who recognized a good thing and let it ride. They saw results, they trusted me, and they got out of the way. Things got done, and got done well.

From 2010 to the present - I am increasingly exposed to Agile - specifically the Scrum variety. In some cases, mere lip service and we go about our business and deliver results. However, the hype reaches a fever pitch as Agile adoption starts to crest.

Finally I land in a project that is completely sold out to the Scrum methodology. Never have I felt more restricted and hemmed-in by a process. Never before have I experienced the crushing overhead of daily meetings, status reports, poker-planning sessions that didn't really "plan" anything. Never before had I been asked to plan, design, develop, and deliver complex functionality in the span of a few days. That by the way, was intended to be production quality.

My exposure to any business people or product owners was completely cut off. We had project managers, analysts, "UX architects", Scrum masters, etc. who interacted with the product owners. Not one single developer.

Documented requirements for an epic or a story were typically a sentence or two, sometimes stretched to a paragraph if we were lucky. "Planning" sessions consisted of a half-day to maybe a day of rambling discussions where no decisions were made or any direction given. The developers on the "team" were asked for input from time to time, which was summarily dismissed or immediately trumped by a contrary opinion from a non-developer.

Anyway, my good friend of 15 years, who I consider to be a better developer than myself, quit in frustration. He is now the proud owner of a Chick-Fil-A franchise in Alabama. I held on in that situation for another six months and then I left as well. I took an Architect job at a different company, who has cautiously adopted Agile methodologies on a FEW projects as a pilot. I just don't think I can function as a developer in that kind of environment. Anyone who is "selling" Agile, is going to sell management on the things they want to hear. Be warned.