No Waterfall Trap

I was referred to a post about a keynote speech Jeff Paton gave at XP Days London 2007. It’s called “The waterfall trap for Agile projects” by Gojko Adzik. The initial argument of the post is that iterative is not the same as incremental, something I will not argue. Gojko gives some nice Mona Lisa examples to illustrate the difference. Unfortunately, he goes on to develop some ideas and conclusions that I strongly disagree with and thus this article, in which I have developed my own “Van Gogh” counterexamples.

We will discuss three strategies to non-waterfall software development:

• Pure incremental
• Pure iterative
• Iterative and incremental

Missing the big picture: the “pure incremental” approach

In this approach to agile software development, it seems that you literally only look at the first story in your backlog, implement it fully, and then scratch your head and ask yourself what comes next. This would imply no analysis and design up front, not even the slightest hint of it (sounds like no Product Backlog and no Product Owner). This is clearly risky in the sense that nobody - nor the development team nor the customer- has any idea on what they are doing or where they are going. They might be moving fast, but in a (potentially) random direction.

Analysis & Design      Delivery 1             Delivery 2             Delivery 3

Pure Incremental

Honestly I have never heard of any team doing this except for pseudo-Agile teams that think that Agile means no documentation and cowboy coding. In large projects, you normally tend to have  the opposite problem (too much analysis and design up front). In any case, I believe that if this is a serious risk the root cause of the problem lies in the lack of a strong Product Owner. Remember the Product Owner (not the team) is responsible for achieving ROI, so if the team ends up making the sunflowers instead of the self-portrait, it’s the PO’s call. I talk more about this at the end of the article.

Never finished: the “pure iterative” approach

Which brings us to the opposite tactic: to go pure iterative. Paton and Adzik seem to defend this approach as the best way to minimize the risk of not delivering anything, and this is where I strongly disagree. I actually see the risk of not delivering “anything” as even larger than in the first case.

In pure iterative (or “spiral development”), you start with an outline of your application and you iteratively refine it until the customer is satisfied. This would seem to bring the advantage that if the customer is happy with a lower level of detail (a.k.a. “quality”), you would finish your project faster and with full coverage of all the scope. Sounds nice in theory.

Analysis & Design         Delivery 1           Delivery 2         Delivery 3

Pure Iterative

What’s wrong with this? In my experience, two very dangerous things.

First, you are still in a “big bang” mentality. Until you reach the minimum level of quality that the customer needs, you basically have nothing usable. Taking the example above, if the customer would run out of budget after Delivery 2 but this would prove to be not good enough to go live, he is left with nothing.

Second, by taking a breadth-first approach, you are investing precious development effort on medium and low priority features that might otherwise never be implemented. You are basically distributing your budget among high and low ROI work equally. You are potentially wasting the customer’s money, since if he later decides to scope cut the lower half of the picture, he will already have paid for a large part of it.

In large projects, these are two very realistic risks. If you were to underestimate and run out of budget, it is much safer strategy to deliver 50% of the scope 100% done than to deliver 100% of the scope 50% done. The first scenario will probably get your 50% approved and this might buy you more budget. The second situation will probably make you fail your Phase 1 UAT and get your project cancelled.

So, what’s the ideal approach to reduce risk?

Iterative and incremental

I believe the ideal approach much closer to scenario one, but with a catch: some analysis and design up front. This is what is normally referred to as iterative and incremental, a mix of both tactics, but I would heavily bias it towards the incremental.

Analysis & Design     Delivery 1              Delivery 2             Delivery 3

Agile: Iterative & Incremental

By first taking a high level approach to defining both the scope and the architecture of your application you eliminate the risk of not knowing where you are going or how to get there.

With this mixed strategy, you get the full benefits of an incremental approach: allowing the customer to leave stories open until the moment they are about to be implemented, maximum ROI, possibility to scope cut or even stop the project at any moment and still get quality working software. And all without any risks of missing the big picture or not being able to evolve the architecture accordingly as the application grows.

Not converging

The risk for the team of not converging to a working application is zero, because it is not their job. This is the responsibility of the Product Owner. A well maintained backlog and burndown will clearly show the project planning and thus where it converges. Any change in requirements will be immediately reflected in the backlog, and any new stories will push the delivery date further ahead. As long as all this is decided by the customer, this is fine. The original XP Days keynote speaks of clueless customers who continuously change direction for lack of an understanding of what they want or need. If this were to happen, the most you can do is detect it early on and try to help take corrective actions. In any case, this is a business risk and not so much a software development risk. It is the cost the customer pays for not having a good Product Owner.