Follow-up on Lean Architecture

This post is a sequel to the previous post regarding the lean architect. It’s main topic is the book review of “Lean Architecture – for Agile Software Development”  from James O. Coplien & Gertrud Bjørnvig  This is really a follow-up in the sense that I have found most of the ideas from my previous post expressed in this book, but they are more thoughtfully presented :).  It also goes further than my previous analysis, which is why I am writing this quick – and incomplete – book review. James Coplien  is both a prolific author and a serious expert on software architecture, with an itinerary that is not so different from mine, especially with respect to object-oriented programming, which use to be my own domain of expertise twenty years ago. Similarly the 10 years which I have spent working on information systems architecture from 1997 to 2007, are well mirrored by Coplien track record in system’s architecture in the 90s and 00s.

The first key idea of “Lean Architecture” is the reconciliation of agile and architecture, because of the increase of scale for projects that agile methods are addressing today. “Extreme Programming (XP) started out in part by consciously trying to do exactly the opposite of what conventional wisdom recommended, and in part by limiting itself to small-scale software development. Over time, we have come in full circle, and many of the old practices are being restored, even in the halls and canon of Agiledom”.  On page 161, one finds a nice graph from Boehm and Turner that shows the effect of size on the need for architecture.  As usual with Boehm, this is a data-derived graph that captures one’s intuition:  once a software project becomes large, anticipation and forethinking is required.

There is no surprise here. On page 15 we read “Ignoring architecture in the long term increases long term costs”, which any one with gray hairs knows firsthand. The insight from this book is that the lean contribution to agile is to bring back the long term and systemic focus into agile, hence the need for architecture.

Coplien and Bjørnvig bring a fresh and simple vision of lean software development, which is summarized in the book by two formulation of the same principle:

• Lean = All hands on deck + long term planning
• Lean = Everybody, All together, Early on

I would not say that “The Toyota Way” could be fit into such an equation, but it captures an important part of what lean management is about, and it’s relevant to emphasize the long term vision which is indeed a key aspect from lean. It also helps the authors to introduce a (creative) tension between lean and agile:

• Agile is oriented towards change and organic complexity. For instance it is good to defer decision (this is not new or “agile per se” since Knuth is quoted to have said “premature optimization is the root of all evil”).
• Lean is focused towards large-scale complexity (including “complication”)  and long –term resilience. Hence the focus on standardization and bringing decisions forward.

This distinction is presented in a full table page 12, and, frankly, I disagree with most of its rows. For instance, team versus individual is not a good characterization of lean versus agile. Nor is the complicated versus complex debate … and the “high throughput (lean)” versus “low latency (agile)” debate is too restrictive.  My views are expressed in a previous post and I think that there is much more in common between lean thinking and agile thinking. Still, I would definitely agree that the management of time, the systemic and holistic view, and the overall long-term perspective is different between lean and agile. Clearly, the goal of lean software development is to combine both, and this is where the extra emphasis on architecture comes from.  It also explains the emphasis on maintainability, as expressed by this quote from Jeffrey Liker about  lean:  “a culture of stopping or slowing down to get quality right the first time to enhance productivity in the long run”. The tension between lean and agile is helpful to explain why one needs both the systemic practice of “Five Whys” in the search of root cause while problem solving, together with the frequent feedback and adaptive cycles. Indeed the complexity and the rate of change of our current environment require both: complexity demands the 5 whys, but the rate of change means that this is not enough, quick feedback loops are required.

An interesting development is this book deals with Conway’s law. This principle states that there is a strong dependency between architecture and organization. That is, the system architecture (module organization) will be strongly influenced by the management organization (teams & departments) that produces the piece of software. Conversely, things work best (from a management perspective) when people organization follows the system architecture. Coplien derives two consequences from this law, which strike me as truly relevant:

• “Manager are über-architects, a responsibility not to be taken lightly” … here goes the myth that one may do a good job at managing software developers without a keen insight about software architecture :)
•  “There is a need for modularity in large-scale system” – this is obviously true … and indeed a consequence of Conway’s law. The goal of modularity is to keep changes local, and since it is unfortunately mandatory on the people side as soon as the scale of the projects tips on the “large side”, modularity in the organization must translate into modularity in software, hence the need for architecture :)

This post will not do justice to this excellent book, which is full of wisdom. I do not have the time to collect all pearls, such as “Remember that architecture is mainly about people” or “Software development is rarely a matter of having enough muscle to get the job done, but rather of having the right skill sets present”.  I refer you to my book for my own two cents of wisdom about information system architecture and management. Still, it worth repeating that architecture is foremost a communication tool to manage change, and not a blueprint for a better word.  One of my favorite quote from “Lean Architecture” is the following: “The customer has a larger stake in the development process than in the service that your software provides”.  This is the crucial idea that is at the heart of the “Lean Software Factory”. In a dynamic and changing world, software is not an object, it is a process. Qualities such as evolvability, modularity and openness come from the people and the development process, much more than they apply to a finished product.

This brings me to my second favorite quote, page 131: “The essence of “Lean” in Lean architecture is to take careful, well-considered analysis and distill it into APIs written in everyday programming language”. Here we see the long-term/resilience thread of lean thinking brought into software development in a very practical manner. This quote expresses two key ideas into one sentence:

• Being “future-proof” is having the right APIs so that the code may be both extended easily as well as reused in different manners from which it was intended in the first place.
• Defining, I would rather say “growing”, the right set of APIs is an art; it requires practice, wisdom and aesthetical judgment. Still it required forethinking and analysis, which is architectural thinking.

This should be enough to reconcile agile programming with architecture, if there ever was a need. Coplien and Bjornvig spend a few delightful pages debunking “Agile Myths” such as:

1.  “You can always refactor your way to a better architecture”. Refactoring is crucial because software is a live object that evolves constantly as its environment changes. Agile methods, like any iterative development processes, are bound to produce “accumulations” that need to be cleaned up.
2. “We should do things at the last moment”.
   Here we find the “anticipation versus as-late-as-possible” debate that was mentioned earlier.  Many authors, including Mary Poppendieck, consider that lean thinking translates into taking design decision as late as possible. This book does a good job at balancing the arguments. There is no single answer, but there is a strong plea for “thinking ahead” and “preparing oneself” through architectural design.
3. Agile = “don’t do documentation”.
   The agile tradition of no comprehensive documentation that gets obsolete before it is used still stands. However, as soon as scale grows, and as soon as the life expectancy of the software piece grows, there is a need for serious documentation. There are many tools to automate part of the documentation task, especially that which is closely linked to the code. A large piece of software requires storytelling, and this has nothing to do with software, it’s a consequence of human nature and what is needed to motivate a large group of people (see Daniel Pink).

User stories are an integral part of Agile/SCRUM development methods. There are a few interesting pages in this book that show the link between user stories and business processes, which are commonly associated to large-scale waterfall development process.  The last part of the book deals with DCI (Data, Context and Interaction), a framework that extends MVC and which is proposed as the proper foundation for designing modern distributed object-oriented systems. This topic is out of the scope of this post, although I find many similarities with the design philosophy of the CLAIRE programming language. Some of the key insights may be summarized as the need to recognize and separate the need for complex algorithm from object classes (“Does this mean that procedural design is completely dead ? Of course not. There are algorithms that are just algorithms”), the reification of roles and business processes and the use of context objects to develop functional polymorphism (hence to share and reuse more code), a practice that reminds me of my early day at Bouygues Telecom when we created the PTEs (Processus Techniques Elementaires) – cf. my book on Urbanization.

I will conclude with two quotes from page 92:

• “Let the human considerations drive the partitioning [architecture], with software engineering concerns secondary”.
•  “A complex system might use several paradigms”.