Friday, June 06, 2014

Heisenberg Developers

TL:DR You can not observe a developer without altering their behavior.


First a story.

Several years ago I worked on a largish project as one of a team of developers. We were building an internal system to support an existing business process. Initially things went very well. The user requirements were reasonably well defined and we worked effectively iterating on the backlog. We were mostly left to our own devices. We had a non-technical business owner and a number of potential users who gave us broad objectives, and who tested features as they became available. When we felt that piece needed refactoring, we spent the time to do it. When a pain point appeared in the software we changed the design to remove it. We didn’t have to ask permission to do any of things, so long features appeared at reasonable intervals, everyone was happy.

Then came that requirement. The one where you try to replace an expert user’s years of experience and intuition with software. What started out as a vague and wooly requirement, soon became a monster as we started to dig into it. We tried to push back against it, or at least get it scheduled for a later version of the software to be delivered at some unspecified time in future. But no, the business was insistent, they wanted it in the next version. A very clever colleague thought the problem could be solved with a custom DSL that would allow the users themselves to encode their business rules and he and another guy set to work building it. Several months later, he was still working on it. The business was frustrated by the lack of progress and the vaguely hoped for project delivery dates began to slip. It was all a bit of a mess.

The boss looked at this and decided that we were loose cannons and the ship needed tightening up. He hired a project manager with an excellent CV and a reputation for getting wayward software projects under control. He introduced us to ‘Jira’, a word that strikes fear into the soul of a developer. Now, rather than taking a high level requirement and simply delivering it at some point in the future, we would break the feature into finely grained tasks, estimate each of the tasks, then break the tasks into finer grained tasks if the estimate was more than a day’s work. Every two weeks we would have a day long planning meeting where these tasks were defined. We then spent the next 8 days working on the tasks and updating Jira with how long each one took. Our project manager would be displeased when tasks took longer than the estimate and would immediately assign one of the other team members to work with the original developer to hurry it along. We soon learned to add plenty of contingency to our estimates. We were delivery focused. Any request to refactor the software was met with disapproval, and our time was too finely managed to allow us refactor ‘under the radar’.

Then a strange thing started to happen. Everything slowed.

Of course we had no way to prove it because there was no data from ‘pre-PM’ to compare to ‘post-PM’, but there was a noticeable downward notch in the speed at which features were delivered. With his calculations showing that the project’s delivery date was slipping, our PM did the obvious thing and started hiring more developers, I think they were mostly people he’d worked with before. We, the existing team had very little say in who was hired, and it did seem that there was something of a cultural gap between us and the new guys. Whenever there was any debate about refactoring the code, or backing out of a problematic feature, the new guys would argue against it, saying it was ‘ivory tower’, and not delivering features. The PM would veto the work and side with the new guys.

We became somewhat de-motivated. After loosing an argument about how things should be done more than a few times, you start to have a pretty clear choice: knuckle down, don’t argue and get paid, or leave. Our best developer, the DSL guy, did leave, and the ones of us arguing for good design lost one of our main champions. I learnt to inflate my estimates, do what I was told to do, and to keep my imagination and creativity for my evening and weekend projects. I found it odd that few of my new colleagues seemed to actually enjoy software development, the talk in our office was now more about cars than programming languages. They actually seemed to like the finely grained management. As one explained to me, “you take the next item off the list, do the work, check it in, and you don’t have to worry about it.” It relieved them of the responsibility to make difficult decisions, or take a strategic view.

The project was not a happy one. Features took longer and longer to be delivered. There always seemed to be a mounting number of bugs, few of which seemed to get fixed, even as the team grew. The business spent more and more money for fewer and fewer benefits.

Why did it all go so wrong?

Finely grained management of software developers is compelling to a business. Any organization craves control. We want to know what we are getting in return for those expensive developer salaries. We want to be able to accurately estimate the time taken to deliver a system in order to do an effective cost-benefit analysis and to give the business an accurate forecast of delivery. There’s also the hope that by building an accurate database of estimates verses actual effort, we can fine tune our estimation, and by analysis find efficiencies in the software development process.

The problem with this approach is that it fundamentally misunderstands the nature of software development. That it is a creative and experimental process. Software development is a complex system of multiple poorly understood feedback loops and interactions. It is an organic process of trial and error, false starts, experiments and monumental cock-ups. Numerous studies have shown that effective creative work is best done by motivated autonomous experts. As developers we need to be free to try things out, see how they evolve, back away from bad decisions, maybe try several different things before we find one that works. We don’t have hard numbers for why we want to try this or that, or why we want to stop in the middle of this task and throw away everything we’ve done. We can’t really justify all our decisions, many them are hunches, many of them are wrong.

If you ask me how long a feature is going to take, my honest answer is that I really have no idea. I may have a ball-park idea, but there’s a long-tail of lower-probability possibilities, that mean that I could easily be out by a factor of 10. What about the feature itself? Is it really such a good idea? I’m not just the implementer of this software, I’m a stake holder too. What if there’s a better way to address this business requirement? What if we discover a better way half way through the estimated time? What if I suddenly stumble on a technology or a technique that could make a big difference to the business? What if it’s not on the road map?

As soon as you ask a developer to tell you exactly what he’s going to do over the next 8 days (or worse weeks or months), you kill much of the creativity and serendipity. You may say that he is free to change the estimates or the tasks at any time, but he will still feel that he has to at least justify the changes. The more finely grained the tasks, the more you kill autonomy and creativity. No matter how much you say it doesn’t matter if he doesn’t meet his estimates, he’ll still feel bad about it. His response to being asked for estimates is twofold: first, he will learn to put in large contingencies, just in case one of those rabbit-holes crosses his path; second, he will look for the quick fix, the hack that just gets the job done. Damn technical debt, that’s for the next poor soul to deal with, I must meet my estimate. Good developers are used to doing necessary, but hard to justify work ‘under the radar’, they effectively lie to management about what they are really doing, but finely grained management makes it hard to steal the time in which to do it.

To be clear, I’m not speaking for everyone here. Not all developers dislike micromanagement. Some are more attracted to the paycheck than the art. For them, micromanagement can be very attractive. So long as you know how to work the system you can happily submit inflated estimates, just do what you’re told, and check in the feature. If users are unhappy and the system is buggy and late, you are not to blame, you just did what you were told.

Finely grained management is a recipe for ‘talent evaporation’. The people who live and breathe software will leave – they usually have few problems getting jobs elsewhere. The people who don’t like to take decisions and need an excuse, will stay. You will find yourself with a compliant team that meekly carries out your instructions, doesn’t argue about the utility of features, fills in Jira correctly, meets their estimates, and produces very poor quality software.

So how should one manage developers?

Simple: give them autonomy. It seems like a panacea, but finely grained management is poisonous for software development. It’s far better to give high level goals and allow your developers to meet them as they see fit. Sometimes they will fail; you need to build in contingency for this. But don’t react to failure by putting in more process and control. Work on building a great team that you can trust and that can contribute to success rather than employing rooms full of passive code monkeys.


Nick said...

Wow - I recently wrote a post about building teams based on all the experiences and observations you list in this post:

I just couldn't agree more with anything you said.

Corey Perkins said...

Agreed, 1000 times over!

dustractor said...

True on so many levels. Reminds me of the anecdote about programmers as people whose lives depend on getting eight consecutive hours of sleep, yet managers tend to sit on the edge of the bed asking "are you asleep?" every hour on the hour.

Patrick Q said...

Absolutely brilliant post! This has to be one of the most insightful articles I've ever read on the developer mindset.

psjp said...

"If you ask me how long a feature is going to take..."
That para is pure gold.

However, I'd like to point out one important positive that Jira brings - PM tools like Jira, i feel, are used to ensure that a Minimum work output is there from developers. Over-reliance on them can cause all the problems you mentioned in your post.

Anonymous said...

I agree completely and have experienced exactly this, multiple times.
But, I have a question. How did "That Requiremeng" get resolved? Did it get built? The autonomous developers seemed to be struggling with it. Did the micromanaged developers get anywhere with it?

Anonymous said...

I think I've never read anything as spot on about this particular topic.

Very well described, and I agree 100%.

Thanks for putting it into words, and describing it so exactly and brilliantly.

[^MiSaTo^] said...

I mostly agree with your arguments but I would like to add something about it. Most of the problems is that PM don't understand how software development works.
In my company we use agile metodology and I find it useful ONLY if PM and clients understand how we develop software. It's useful for us (developers) to plan and slice in smaller "problems" and focus in them.Although we use 2-3 week sprints and we have a meeting every morning to see if there were problems or we need to adjust or whatever. A lot of times we underestimated user stories. Some others we overestimated. We usually estimate on difficulty more than in time.
Also our PM DO understand the need of refactoring, the need of making good code so we really focus in the quality of the product and not only on delivering stuff.

I'm happy where I work because of that. Before being in this company, I was in others just like you describe. At the end you get demotivated developers or other people that are not really creative but only want to get something done, without much thinking, deliver it and go on. I think you can't do good quality software like that but unfortunately that's what a LOT of software companies are. And PMs obsessed with points, delivery in time, and such are mostly on those companies.

Ollie Riches said...

I've worked in both environments several times over, and yes when you're experienced and motivated you can churn out of the work at a fantastic rate.

But, when you work for someone else you sing their tune and if you don't like it you can leave.

Sorry but developers expect to much...

Eric Parshall said...

Sure Ollie, that is how it works. If employers want passive code monkeys they'll get exactly that, and more bugs and features that take longer to create, etc. His point is that employers would be wise to let go of their need to measure everything and get more motivated emplyees that do a better job. If employers don't like that they can ignore the advice and carry on business-as-usual.

Ollie Riches said...

'Sure Ollie, that is how it works'

If employers want passive code monkeys they'll get exactly that, and more bugs and features that take longer to create, etc.

Employers don't care about developers, they care about making money from software, if they are making money from 10,000 lines of spaghetti code they won't care about anything else. If they aren't making money they'll probably go under, at best they might try again with better people.

Developers don't seem to realize you are just a commodity, you're hired for your experience & intellect and employers are very well aware at what level they are recruiting.

There's more to life than writing code...

Steve said...

Great article, but...

Autonomy isn't a panacea either. In my time I've seen it allow some really bad examples of cathedral-building and CV-driven-development.

The scenario you described was certainly dysfunctional and a classic example of how to destroy a culture, but it sounds more like the root of it all was a failure of the organisation to understand the real needs of their real end users, and that this (management) mistake getting compounded into the destruction of a high-functioning dev team.

I am 100% for autonomy, but not if the team is autonomously fixing the wrong problems...

Jacob said...

I'm on Ollies side here. It's not that I think employers intentionally set out to make lives miserable for the autonomous developers - they just have different interests. They need to run a business and sell their product. So when they ask developers: "How long is it going to take", what they are really asking is: "How long do I need to go on paying you before I see some return on my investment that I can get customers to pay for?".

If you want to be allowed to work autonomously you need to provide some transparency for the management. One way is to be specific about difficulties, fx. explain that features are not well understood or that requirements are too vague. Be specific about what is not understood or what is too vague. Eric Evans calls this creating the ubiquitous language between business and developers.

At the end of the day people are paid to ship, not to polish.

James Morcom said...

This seems wonderful and makes me envious of those that get to work on internally funded projects! At least in that situation you have the potential to demonstrate to your employer that it works.

I dare say you wouldn't win much work though turning up to pitch to potential customers and declaring that you'll definitely deliver something, but you don't know how long it will take or how much it will cost.

Maybe we should be trying to convince the customer that this way works, and that choosing a developer should be more like commissioning a painting than building an extension to your house.

I guess this would be done by demonstrating a portfolio of previous projects that weren't done to a fixed cost but were successful.

I suppose one advantage we have over the artist is we can offer something like "Pay us some fraction of the budget and make your decision on whether to continue with us based on what we deliver in that time". Although this would be unlikely to work when there's a fixed timescale.

Anonymous said...

Guess you dont care about delivering quality software then...

Anonymous said...

"I've worked in both environments several times over"

In other words, you didn't know how to deliver software as a developer, so you became a PM...

Anonymous said...

Harsh, but fair.

Ollie Riches said...

In other words, you didn't know how to deliver software as a developer, so you became a PM...

Is Anonymous your first or last name?

contract developer for over 13 years and counting, how about you?

David Arno said...

An interesting article ... mainly it's an interesting illustration of how to blame the tool, rather than the wielder.

These days,I've moved on from writing code, to effectively being the project manager for three separate projects. Having been on both sides of the fence (on more than one occasion), here are my thoughts on this article:

1. Jira is a really good piece of software. Just as someone can write shit code with C#, so someone can do shit things with Jira. Used well, it's an incredible tool.
2. The PM you describe is not a good one. A good PM of a software project will know what technical debt is and will know that time must be spent writing tests, refactoring etc. He will know that some tasks will massively overrun. He will know how to balance the need to meet deadlines versus the need to create quality code. In shot, a good PM will empathise with developers.
3. Like many things, it's all about balance. Autonomy is good. So is accountability and transparency. Let developers plan a sprint and estimate tasks, then leave them to it until the next planning session (save for daily standups to let them report problems and successes).
4. Find a balance between leaving a developer to work on one task for months and micro-managing her day. Both extremes are crappy. The middle ground works well.
5. If you find yourself increasing your estimates, due to the previous ones being too low, how can this not be a good thing? Far too many developers (me included, all too often) under-estimate tasks.

One successfully manages developers like one manages everyone else: by seeking the best balance between autonomy and process. The exact best balance will vary between teams. What's certain though is that too much autonomy and too much process are both very bad things.

Dean Chalk said...

I think this article describes a very stereotypical set of scenarios, whith PM's and developers being portrayed as caricatures of the real thing.

Ive been a freelance enterprise developer for 15 years, and there are pro's and con's to micro-managed teams as well as unmanaged teams.

Ive worked in just about every possible team configuration and there is no sure-fire recipe for success. If you not able to be ingenious, imaginative and capable - your arent a great developer. If you cant floursh in a micro-managed environment - you are not a good developer.

A good developer understands that team-based software development is very complex, with many different convoluted paths to success.

Anonymous said...

I dare say you wouldn't win much work though turning up to pitch to potential customers and declaring that you'll definitely deliver something, but you don't know how long it will take or how much it will cost.

Hahaha great comment

Doggie Sensei said...

A sad but typical story. Unfortunately the process management types think they provide consistency because the collect data about developer behavior, and through the use of statistics, they can normalize that behavior and bring "control".

The key element that they miss is that these missions that then send developers on are NEVER THE SAME, so it's like asking a football team to describe how the plays will proceed. They can never tell you, because the ball bounces weird, someone twists an ankle. etc. Software development utilizes methods, but not process in the same way manufacturing does.

Anonymous said...

"it's like asking a football team to describe how the plays will proceed."

That is exactly what software development is like. The difference between good developers and bad developers is that a good developers will have an offensive/defensive formation and player responsibility given out. Bad developers are like sandlot teams that just line up and all do dumb things.

Good developers will give you a fighting chance while the bad developers may get a bounce here and there, but eventually it will come to and end.

...number88 said...

I chose to go down the Scrum route because it positively encourages developer autonomy, but with an eye on what the business thinks is the next high priority item. Devs choose the solution, Business chooses the next item. As Scrum Master I must ensure that devs are given the best environment in which to flourish. If estimates are killing creativity, then my job is to remove that burden.
So.. project management per se is not going to deflate devs; but bad project management will.

Sean Kearon said...

Awesome post!

Drea said...

That post rings very true. The one thing I'd say is that Jira and other project management tools are very helpful of keeping track of things you want to do in the future (otherwise, those things get lost). The problem comes when someone wants "predictable execution". That is where the downward spiral of punishment and padding begins. Predictable execution is a recipe for mediocrity and a predictably glacial pace of development. My personal rule, "no points!". Whenever I've been required to give things points... just give them all the same number. On average, it works out.

Justin Kendall said...

The company that I work for recently went through a restructure. All aspects of Dev., whether they be production or in-house support systems are now managed like they were Sales or some other department where "how long is this going to take?" is an easy and concrete question to answer.
That's one of the many reasons I'm the only Dev. left and also why I'm leaving my job shortly for a better opportunity.
The moments I was left alone and was able to wow management based on their lack of expectations and when I came up with solutions to problems before they were problems were the best of times.
Being micromanaged and forced to kick out shit software on an unreasonable time schedule is not only career killing, but soul stealing. I measure my career worth in inverse proportion to bug fixes.
Thanks for sharing.

Anonymous said...

I get the impression that the only people who replied to this, and loved it, are developers. While I understand the desire and even the need for autonomy in order to harness creativity, please keep in mind that the owners of the business have a responsibility to keep the business in the black. It's childish to assume that you can just let everyone do what they want on their own schedule. In time, it will fall apart. Finding the right balance is critical and something all good owners/managers will do.

Ted Martin said...

Good article. It and the subsequent dissenting comments just show that there is not a single approach to software development but a range of options based on the circumstances.

The author made it clear that their approach to interpreting the user's requirements worked very well and the users (clients/customers/business) was very happy until they hit a requirement that was too vague and really needed a subject matter expert to better define. In my mind, this is where the fine-grained approach for this specific requirement could have helped.

So, I think it is better to be open to different development approaches depending on the situation. I do feel that the software-engineering/fine-grained approach should be the exception and not the rule as it is often overkill.

Ian said...

IMHO a good PM should "manage" client expectations, the quality & depth of requirements statements, scope creep etc, but really, shouldn't be aiming for any level of developer management or control - the best PM's are those who _enable_ project developers - ensuring they remove blocks and enabling the developer to concentrate and build the best code they can that fulfills the requirements. End of.

Victor Scott said...

Great article. It explains the problems caused by micromanagement perfectly. I would add that not only do companies lose good developers but also cause good developers not to apply for their job openings. I refuse to consider any job that requires Agile, Scrum, daily standup meetings, pair programming, Xtreme Programming, and so on. It is all micromanagement!

A good software developer or team leader knows what to do and when to do it (when a meeting is required, clarification is needed, communication is required, and so on). Also, estimates are part of the business. I don't have a problem with estimates. However, a developer should be given a little time to deliver a meaningful estimate (which may require some experimentation and research).

Anonymous said...

Hiring an experienced architect would be my first recommendation. Break teams into smaller groups of max 5-6 devs., architect defines coding standard, creates architecture and performs code reviews. PM takes resource allocation directives from the architect. Architect isoloates areas of volatility, and deals with changes. Senior (and team leader) devs consult with architect regularly. PM works with architect on schedule, integration path etc. Being a good architect is not just knowing the architecture, it is about "driving the machine". Hiring devs in the middle of the project to speed up the progress is suicidal as is giving PM anything else besides a Gantt amount of SCRUM or Agile will fix broken architecture integrity. SCRUM is stupid and does not work. Agile is fine but has to be implemented correctly. Anything that starts with re-, is bad; refactoring, redoing, rebuilding...and then re-testing. Thank you for the story...classic example of "software crisis" see Wikipedia on that. Cheers!!!

Anonymous said...

brilliant post!I mostly agree with your arguments but I would like to add something about it. Most of the problems is that PM don't understand how software development works.If employers want passive code monkeys they'll get exactly that, and more bugs and features that take longer to create, etc.

Guy said...

1. Don't blame JIRA

2. Don't blame Agile/SCRUM practices

3. By all means, blame estimations measured in hours. For software, the only estimation worth anything is relative estimation, and the intention of that is primarily to help with planning/backlog refinement.

4. Blame the untrained PM who is trying to implement a practice no-one understands.

Solution: If you use SCRUM, train the whole team in all of the roles. That way everyone knows the rules and why the roles are important. If management says no, get a new job. Future projects are doomed due to scatter gun mismanagement of a bit bucket of issues using process for the sake of process.

Josip Almasi said...

Speaking of technical debt, there's a wonderful tool called sonarqube - it shows technical debt in terms bosses can understand, like number of working hours, it also paints criticals and majors in red colors etc etc, in a word, wonderful!
So anyone in need of explaining code quality to bosses, hook up your code to sonarqube, makes it waaaay easier.

Oh by the way, it's also handy for monitoring code quality.

Josip Almasi said...

My SCRUM project manager stood up for developers.
And it's supposed to be like that. According to methodology, project owner is supposed to press issues from customer's point of view, while project manager advocates developers view.

So our good PM was first to leave the team.