Wherever I come, whomever I meet, in each and every organization estimates seem to be a problem (and mine is in no way better). There are various sources of problems with estimates. The most common symptom I see is teams taking too long to estimate because they take it too serious. This leads to the typical complaint that method X bears too much overhead, the meetings are too inefficient, boring and cumbersome, and finally the whole of method X is under question.
A typical situation in this context is a longish discussion if feature US12 has 5 or 8, maybe even 13 feature points. Gosh! You can read books on this issue. The even tell you what to do. This situation is supposed to be a clear indication of uncertainty in requirements, which you can resolve in several ways, most of them involving even more overhead. Heavens no, the enemy entered the stage: Variability! The PO must really be a prick! A more modest assumption may be that the team is unsure of the implementation approach, architecture, design, task breakdown, yadayadayada.
First of all: I don't blame the teams in this scenario. The environment in which they live must have signalled to them - in whatever subtle way - that their estimations matter. To say the least. Hell no - the estimates don't only matter, they are part of your commitment and before you say "f$%^ no" the team's commitment is a commitment to a timeline, depending on no less than a tight sprint plan, release plan, which again depends on the correctness of all estimations of the other projects, on the limited variability of all the features any talented PO can think of etc. If you leave this blog for only 5 minutes and try to brainstorm on the dependencies of our current project you will go blind and depressive at the same time. If all those dependencies were really embedded in your 'planning' and thus in your estimations and your commitment you might as well stay home and never touch your keyboard again. But anyway, your rent needs to be paid, so you come back to your mind after the 5 minute break and another cup of tea, destined to go back to work tomorrow, no matter what. At least you now know that a) your estimate can not contain all uncertainties and is only a best guess snapshot of your current knowledge and b) does not matter that much in that context.
Another symptom is that teams can not decide on the estimation process (planning poker based on Fibonacci, planning poker based on any natural numbers, split up stories above a limit of 21, estimate in days and numbers (no stop, I'm back to just hours), classical function point analysis!, ... ) I'm getting nauseas over all the decisions you could feel forced to make. And people can get religious over it. They do.
What I will tell you now may let you relax a little over all those micro decisions and estimates. It may well be that what I think and say only matters for some areas, maybe 40% of software development. I personally think it matters for around 80% of software development (or all of it once one got the point). Anyway, here is what I think.
Don't care (too much) about your estimates. They don't matter too much for several reasons. One not so sublime reason is that while one might think it makes sense to get the variability out of the estimations by science and sound reasoning, I do think that most of the variability simply levels out statistically. One time you estimate too high, the other time you estimate too low. If you have a systematic error in your estimations, that will be learned by the team over time and level out this way.
The more sublime reason which will be much debated when you bring up this topic is that estimations where brought into the world for the wrong reason and support the wrong reason and they don’t even do it good. Estimates support the wrong approach. Estimates are good for chasing time lines, but they are no good at all for judging value created, value to be created, complexity of a certain topic etc. To explain that, let me change the perspective: Think of my job where I am working on a standing organization honing and fine tuning a standing Internet product (for years). So, I am talking about product development right now, just to make the point more clear. If you are an (even a small one) amazon, ebay, google, whatever revenue generating product with a margin in the internet you are only talking about product development. By that I mean the constant refinement and change of our product. And my statement is that timelines do not matter in product development in general. There may be some very few exceptions, e.g. the infamous press conference to announce your product change with your product going bang into another direction at the same day, 10 am. If your Corp Comm Dept. is clever they know how to rip this event off and make it a success without going into the details as much as generating an issue when a certain percentage of the product is not ready. (Yes, I’m saying that even in this case the exact timeline makes no sense). What matters is the right spirit, (fast) user feedback, ideally a constant flow of product changes to try out new angles and if you're really good a good approach for handling the white space out there to keep innovative and not getting 'MySpace'd' by another facebook.
Timelines are only good for blaming people. Blaming people who have been stupid enough to commit for timelines they could not hold any responsibility for and then couldn’t finish 110% of the features. The world is too complex to maintain timelines and have people getting shot at.
In general, going for the timeline is the same mistake as going for utilization. Reaching any or both targets does not proof a thing. You may keep a timeline and keep utilization high and still have a crap product. It's the same as you should not commit to a Scrum sprint regarding a set o features or user stories but rather on a goal. And the goal would be something like, publish product part X – and in the sprint have a lively productive discussion how this is feasible, which features are necessary, which can be simplified, descoped, which ones need to be more elaborate etc. If you’re lucky you can get closer to value generation by having users opinions as a main influence in these discussions. And guess what - for this approach you don’t need exact estimations. I go as far as to say you don’t even need clear requirements. What you need is much harder to achieve but harder to describe: You need a clear goal and a deep and thorough common understanding of what it means to reach that goal.
In this scenario, no doubt, good estimations can help coming up with good real options on the way of reaching the common goal. But being a few points off does not matter at all. But for sure, an organization with that level of understanding would never burn an individual or a team for not keeping a timeline. (And the teams themselves would be the first ones to ask themselves what went wrong an punishing themselves for being off too far.)
So, my point was: Set clear goals on a quite abstract level, estimate to catch a glimpse of your current understanding of the complexity of your current goals (or your next releases etc.). Reach those goals by an intelligent negotiating process in the team coming up with a set of good options and choosing the most pragmatic and realistic ones. In this setting estimates support you on a very high level but need not be exact. As a rule of thumb, discussing an estimation of a User Story for more than one minute seems to be overhead to me. Really.
Now, why do I think the same holds for any project rather than product development also? Simply because product development is done in … projects. So, a project is simply an organization form and in itself means nothing. In the scenario of a software shop delivering a project to a client, the project will simply be a part of some other product development process. And absurdly, the one project (yes yours) can not be so important with regard to the timeline as to treat it differently than product development should be treated. I know, I know – your client doesn’t see it that way. But that doesn’t mean you have to see it that way (and you can still do an excellent job, delivering all you have to deliver). To get a grip on the overall risk and variability of your current project just listen to some aspects an examples. I once committed to building a major international job portal based on three lines of requirements in Excel with a team of six developers. I knew what I had to do. I had to meet the client each and every week to exactly find out what he needs, where we need to be strict, where we have Leeway, finally to make clear to him that if I fail the problem is bigger for him than for me etc. What role would exact estimations play in this context? Or: do you think that you’re much better off in your current project, just because you have 157 User Stories in a DEEP product backlog? How deep is the level of common understanding of the details of those User Stories throughout the team? What if you overlooked that line that means you do not only have to integrate that SAP HR systems but adapt it also? Or on a smaller level – what if your release cycle is 6 weeks and miss the spot by a day? (Yes, you now lost six weeks based on a three day User Story). What use are your fine grained estimations now? I guarantee you that the little variability you have in your estimations is negligible compared to the rest of the uncertainty in your project and that there is a) no way to drastically get rid of it and b) no cheap way to do decrease that level of uncertainty to a very small level. Further it is just the wrong lever you’re pulling.
The bad news is that what you have to do is much harder and on a different level but much less technical an less process etc. You will have to be clear about what you want and you have to be good about what you’re doing as a profession. This means you have to have open discussions in your team. Where estimates help you is to get a good common sense of your basic speed.
If you ever get the chance to play a round of getKanban, you will have a lot of counterintuitive learning experiences. One of them is that the variability of the User Stories pulled across the board plays a minor role. Lead time is king here, and the learning that waiting times dominate actual work times lets lead time dominate ever more. That means the worse and less lean a system is, the less important god estimations are. In fact, the worse the system is, the less probable are good estimations. The other observation is that (nearly) no one blames the variability of the dices (1200% in getKanban). And finally you will learn that simply watching the lead time of your work pieces together with the visualization of the wok flow, explicit rules and the daily discussion on what to do how will never let you miss a beat. The teams, with all the variability in the game, very seldom miss a fixed date ticket.
