« The Lean Architect » - Do we need Abstraction and Planning « on the Gemba » ?

Today’s post is a first attempt at defining the role of a software architect in a “Lean Software Factory”, that is, an agile development organization built around lean software principles. There is a natural tension between the two words “lean” and “architect”, so one may think that this title is an oxymoron: “lean management”, in general, tries to stay away from abstraction and too much planning, and promotes practical, concrete team-oriented problem solving “on the gemba”, where the action takes place. Agile development says that conception (requirements and design) evolves continuously while a software product is being developed. This is in contradiction with the architect’s role as defined in the V software development model, which is the product of Taylorism applied to development, where the Architect (capital A) thinks a grand design that is built by others. The team-oriented approach of lean software development relies on democracy and consensus, where each voice is heard, and where no “design architecture” can be accepted as a “non-negotiable” requirement. Similarly, a key practice of lean/agile software development is synchronicity (everyone working on the same “takt time”), which is implemented through stand-up meetings, common project rooms and intense real-time discussion with the product owner, which means that the very concept of “planning” and “design” seems in jeopardy.

The first question I’d like to ask is, therefore, “do we need an architectural process when software development is led in a lean/agile mode ?”. My answer is a resounding “YES”, with the following three arguments:

• Agile or lean software development does not prevent from building software products that are actually complex and require a carefully-crafted form of internal organization. Let us recall that architecture is foremost about communication: its role is to provide organizations and actions with a narrative sense.  In a lean software development process, architecture is a “co-creation”:  architecture and design continuously evolve together with product development; they are not “completed” until the product is.
• In a continuous development and integration process, we need an architectural framework to prevent from diverging (too many additions that move the product in different and opposite directions) and maintain a continuous and efficient refactoring. Software code is a living object, which evolves together with the rythms of sprints and update deliveries. As a living object, constant “gardening” is required, which is guided by a vision (of what the garden should be), which is precisely what an “architecture” should be. As Jean-Claude Ameisen explains in his wonderful book “La Sculpture du Vivant”, the birth of a living organism requires as much pruning and removal as creating and addition; this applies to software : there should be as much thoughts and care put into code pruning and removal as there is in lines-of-code addition.
• A key principle from Toyota/Lean Management is the importance of a systemic vision that is shared by all actors (sharing a systemic understanding is a key role of the Obeya room). Architecture in a lean software development process is not restricted to a few hand-picked specialists; it is everyone’s responsibility, especially developers’. A major feedback from SCRUM practitioners is the importance of “design reviews”. Organizing successful design reviews is better conducted by “experts”, who are people with experience (architecture is an art, much more than a science – one learns from previous mistakes). These are precisely “architects”, people who are well qualified to lead a successful review and to encourage learning/capitalization. Agile does not mean than everything needs to be rediscovered “on the gemba”; 30 years of software architecture have produced both useful principles and tools.

The second question to ask is then « What role should an architect play in this emergent software development process ? ». Part of the answer is common to the previous question and has been brushed in the previous three bullet points. The architect is required to shepherd the continuous refactoring and to lead the regular design reviews. This is quite different from the classical role as defined by the V-model. Let us emphasize the practical and cultural differences through three critical insights:

• An architect needs to be a story teller. Agile processes suggest working on requirements expressed as “story boards”. Story boards make for “units of requirement” which are gathered into “backlogs”. The “architectural vision” also needs to be expressed as a set of “stories”.  Stories are a great form for expressing a requirement because they both carry meaning and they are robust. The heart of the agile process is that volatility, randomness, complexity of environment will force the product developers to adapt continuously during its completion. Strict and formal requirements tend to become obsolete or incompatible with one another too soon! Stories are robust because they can evolve; the combination of their meaning is context-dependent. I would qualify stories as “antifragile” following Nassim Taleb’s insight: the volatility and variations that will necessarily occur will “shake” the story and often produce even more value than the original thought contained. Serendipity is produced by the random shocks between stories from the backlog and continuous feedback from the users.
• An architect works in a “pull mode”, she is pulled by the need for advice, assistance or explanation from developers. This is very different of the “push mode” from the Taylor-style organizations where the architect would hand out a document to the developers: “please implement the following architecture requirement R1 to R112”. This requires some form of change management, but this new way of working is quite fulfilling for an architect. She may feel the pleasure of participating to the actual creation, instead of being frustrated to see her “great principles, well-written in a forward-looking design document” ignored. This is the instantiation of a key lean principle that says that each actor in a process is at the service of the next actor (in the value creation process, from a chronological viewpoint). Kanban, both its industrial and software version, is the tool that puts this principle into practice.
• Architects are stakeholders in the SCRUM backlog management: they ensure that performance tuning and bug fixing (especially “non-functional ones”) are maintained on the top of the stack.  Obviously the product owner is the key stakeholder, and these concerns should be hers/his as well. An architect is not a “chief engineer” in the Toyota sense (it is unclear to me if there is such a role in a lean software development process, although it is often used in the Devops context). As with any complex project, there exists a recursive/fractal structure associated with a product (systems and sub-systems). There are a number of “product owners” (for subsystems) and a master “product owner” who plays a role similar to that of a chief engineer. The architect is a member of the engineering team, a voice that reflects the value of long-term thinking, simplicity and “well-oiled” running. Using lean/agile does not remove the age-old conflict between short-term and long-term goals. Products tend to last (at least, customers wish they would), hence long-term characteristics are essential to a well-designed and well-built product.

To conclude this post, I would like to stand back and ask a more general question « What is the role of the architect from a systemic viewpoint ? ».

• At the side of the product owner, the lean architect carries the critical mission to build the “situation potential” in Sun Tzu’s sense. The concept of “situation potential” is the cornerstone of François Jullien’s book on efficiency. In a complex world, both François Jullien and Nassim Taleb tell us that forecasting (as in “strategic planning”) does not work well any longer and one should become “opportunistic”, in the (positive) Chinese sense (i.e., be prepared to jump on opportunities). Being able to leverage the future, whatever it holds, is the heart of this “Art of War” strategy. It relies on agility: to react quickly as soon as the opportunity arises. It also relies on building the right set of capabilities: the situation potential. A “good architecture” is precisely a “situation potential”. This becomes even more obvious as soon as the necessity of a platform strategy occurs – the core message of “What would Google do ?” by Jeff Jarvis. Most successful software products are platforms (cf. the famous post by Steve Yegge). Designing a platform requires every developer to be eager to contribute to an “open API culture”, to be ready  “to eat your own dogfood” (the best way to implement the feedback circle that is necessary to reach quality). However, designing a powerful platform is precisely an architectural task.
• A key tenet of “lean software”, as explained by Mary & Tom Poppendieck, is to free up creativity and innovation through cooperation. Architecture may be seen as the “grammar” for this cooperation. This is the core of “service-oriented architecture” (SOA). A software architect, irrespectively of the software development method (agile or not), aims to develop sharing and reuse of software components. This brings us back to the critical role of architecture to develop a software platform. This is lean & pretotyping at work: designing a platform is not (only) about thinking right, it is mostly about feedback loops and listening to users/customers. The value of a grammar is judged from the beautiful sentences that other write through it.
• The more evolution we see in our current world, the more we need solid foundations to avoid losing our time building, destroying and re-building over and over. Layered architecture stays a relevant paradigm in the world of “lean software” because layer structure tells us the order in which things should be built and re-built. Experience shows that one cannot rely only of “auto-organization”: too many iterations – that entail massive rework – are necessary until a fixed-point is reached. Also, the practice of 5S (sorting, set in order, systematic cleaning, standardizing, and sustaining), applied to software (as explained by Mary Poppendieck for instance), is both relevant to code development and software architecture. There are many other interesting architectural contribution to a lean software product, such as the DCI architecture, which may be found in the book “Lean Software Architecture”.

This short post does not do justice to the complex topic of “lean software development” mixed with “software architecture”. The title, “Lean Architect”, applies to a long-term goal, which is to reconcile large-scale-systems-engineering with agile development methods. My own conviction is that “lean” – in the “Toyota Way” sense -  is the appropriate framework for such an integration. Lean adds to agile methods the combination of a global/systemic vision, together with a practical focus on concrete issues (“on the gemba”). This is actually the core of Nassim’s Taleb “Antifragile” book: complex systems are learned by doing, practice is superior to thinking – and this obviously applies to architecture. Because it emphasizes practice and respect for people, lean management is geared to address the complex challenges of our environment.