Lean, Scrum, Agile and Extreme Programming

Two thousand years ago, Epictetus told us “it is impossible for a man to learn what he thinks he already knows”. This principle is still one of the main reasons why it is so difficult to deploy change, when new (development) methods are concerned. I have been advocating for lean and agile methods for many years and the most common pattern I hear is: “I know, I do it already” :)

The goal of this post is to focus on the difference between agile in general, SCRUM in particular, and lean software development. I have been speaking about lean software development for three years and I have found that the (legitimate) confusion between lean and agile does not help. I keep hearing “I already know about agile, scrum … or lean”. I can relate to this. I started as a research scientist in the field of programming languages and software engineering twenty-five years ago, and was there (in the US & the academic community) when agile was born, I knew personally some the founders of the Agile Manifesto. I started practicing agile and extreme programming when I returned to France to create the e-Lab. I brewed my own agile philosophy for a small team of “PhD programmers” under the designation of Fractal Programming. Since then, I have joined larger organizations and tried to promote agile methods. At first, I was promoting what I once knew and tried, but I have quickly discovered that the agile movement is growing constantly, and that we know much more today than what you learned yesterday. Hence the Epitectus quotation.

The topic of mapping the similarities and differences between lean development and agile is quite interesting because the influences and relationships make for an intricate pattern. Agile methods and its various subsequent developments have been implementing “lean principles” for many years. Lean Software Development, on the other hand, inherits from the knowledge built by agile communities. There is already a number of interesting post on this topic. Matthias Marschall’s contribution is a great synthesis which is very similar to what I’ll write in a few minutes. He shows the strong relationships and the differences, and point out at the development process dimension: lean helps to get a larger perspective. His slideshare is also interesting, but is not structured enough from my “conceptual” perspective, which I why I found useful to write my own analysis down. Another great contribution by Cecil Dijoux points out the influences but stresses the benefits of lean as providing a larger perspective, especially the ability to scale up agile methods to enterprise management and transformation. Cecil talks about Kanban, to emphasize that it goes further than the traditional visual management practices of SCRUM. There are so many other examples. For instance, talks about agile methods such as Laurent Sarrazin’s which makes the influence of lean principles obvious. Laurent’s is a great example of a modern talk about agile, which goes much deeper that what I was hearing 15 years ago. In a reciprocate manner, great pieces about lean development, such as Welovelean’s, are still very close to the founding principles of the Agile Manifesto.

The claim of this post is : there is more to lean software development than a great synthesis of the best agile methods. To make my point, I will briefly summarize the key ideas of the successive steps of the evolution path from Agile, XP, SCRUM to Lean Software Development. I need to stress out that the following description is both too brief and incomplete. It is quite succinct because (a) I have already talked about this in previous posts (b) it is easy to find better references for Agile (manifesto), XP, SCRUM or Lean Software. It is incomplete by construction, since to promote clarity, I have only selected three additional ideas/principles when moving from Agile to XP, then to SCRUM, then to LSF.

Let me then characterize the essence of agile (when it started) as the combination of:

1. Small teams : a strong emphasis on strong interactions.
2. Small batches : break a larger project into manageable chunks (sprints), do them one at a time and constantly re-evaluate what the next chunk will be.
3. Time boxing : there is nothing like a time constraint to help you focus on what really matters. Deliver less, but deliver on time.
4. Coevolution of code/design/architecture : the end of the waterfall model, the code and its requirements are built continuously.
5. Role of face-to-face communication : A key insight of agile is that to break the waterfall model, you need to reintroduce face-to-face interaction and banish the email push, which breaks the responsibility and engagement chains.
6. User stories : the only way to capture a need is to give it some flesh and make it into a story. Read Daniel Pink’s “A Whole New Mind” to understand why.

There is a lot of lean in these principles & practices. Obviously once cannot reduce agility to this short list of principles, I should also explain a lot of associated practices (supported with tools) regarding the development process. Let me now turn to Extreme Programming, which I will characterize with three additional ideas (or increased emphasis):

1. Test-driven development: the best way to develop code is to write test cases first.
2. Sustainable pace: the only way to satisfy a customer in the software world is to last, because software is a “living thing” that evolves constantly. Hence, the rhythm must be set to reach long-term goals and respect everyone in the development process.
3. Code is valuable: XP was the first method to significantly raise the status of code, through practices such as code reviews, refactoring or pair programming and principles such as “elegant code”, “do it right the first time”, etc.

Here, as well as in the following section on SCRUM, lean is everywhere J That is, we can see these principles as the application of lean principles to software development, derived from the TPS. I would then characterize SCRUM with the following additional contributions:

1.  Visual Management: give the team a project room, use the walls to share release notes, the backlog, planning information, etc.
2. Practices and Rites: SCRUM is practical method that makes agile easier to learn through its rites, such as standup meetings.
3. Reflection: SCRUM proposes sprint  retrospectives, a great tool to promote continuous improvement. One could say that standing back is part of agile, but SCRUM brings a true emphasis on “doing it for real”.

These twelve sets of practices make a maturity model for SCRUM. To qualify to lean software development, I would add another three steps:

1. Kanban: the application of visual management to the visualization of process flow and work in process (WIP). Kanban is a critical tool to reduce WIP and to implement pull (just in time), two key lean principles.
2. Kaizen :  the heart of the lean approach towards problem solving. Lean makes problem solving a team activity and a learning tool. There are many associated tools and practices such as the “Five whys” or Toyota’s A3.
3. 5S and waste removal : sort, clean-up, organize, focus on simplicity and writing less code. Some of it is part of XP, but lean goes further, with more tools and deeper practices (e.g. value-stream mapping).

This is a simplified view of lean software (although the combined practices of the 15 previous bullet points represents a fair amount of commitment), see the seven principles of Mary and Tom Poppendieck or my own description of Lean Software Factory for more details. A great practical introduction to lean software development is “Lean from the Trenches” by Henrik Kniberg.

Another key evolution of the past 20 years is the scope of agile/lean development method, which has grown from software development to product innovation. In the spirit of keeping this post simple, I will consider the scope with four successively embedded steps (from the smallest to the largest):

• Software development is the original scope of agile methods. It obviously includes testing, as well as the proper code management & configuration tools to make coding as efficient as possible. See my reference to Curt Hibbs in my previous post on software factories.
• The next step is to move to continuous integration. The ability to generate a “fresh product” every night (for instance) that can be fully tested, with as many automated test as possible, yields a completely different culture and level of innovation. This requires a significant amount of tools and training. This is the case where you need to change the tools in order to change the culture.
• The third step is continuous deployment, which is best illustrated by Devops. I have mentioned Devops already, the big step is to include operations in the lean/agile process. Moving from CI to CD requires more tools, more automation … and a big culture change since it require to move from “agile in the lab” to “agile in the field”.
• The last step is continuous product improvement and innovation, which is beautifully explained in The Lean Startup from Eric Ries. In addition to continuous deployment, we need to gather continuous feedback (e.g., with A/B testing) and re-inject this knowledge into the development cycle. A great source to better understand lean startup principles is Ash Maurya’s blog, and these principles do not apply to startups only!

Lean Software Development is the combination of both dimensions: the depth of lean practices and the scope. I could easily derive a maturity index from this: the maturity level would be the product of:  (a) how many of the 15 principles are really in action, times (b) what is the scope of the lean/agile process. I won’t go into this direction because lean software is much more than a development method; it is a work philosophy, in the “Toyota Way” sense. It is mostly a work philosophy that is built on the motivation and satisfaction of the software producers. A key insight from “Implementing Lean Software Development” is that one must move away from a project-oriented culture based on contracts towards a product-oriented culture based on collaboration. A wonderful illustration of the scope of the lean transformation is the great book “Les Géants du Web”, just released by Octo. Its subtitle is “culture / practice / architecture”; most of what I have just described is beautifully explained and illustrated. There is lean and agile everywhere in that book, but this is a book for everyone, not reserved to managers or software developers. The book is about what makes the best “Web companies” successful, from a culture, practice and architecture point of view. However, the similarity with the lean philosophy is striking.