Sunday, January 6, 2013

« The Lean Architect » - Do we need Abstraction and Planning « on the Gemba » ?

Today’s post is a first attempt at defining the role of a software architect in a “Lean Software Factory”, that is, an agile development organization built around lean software principles. There is a natural tension between the two words “lean” and “architect”, so one may think that this title is an oxymoron: “lean management”, in general, tries to stay away from abstraction and too much planning, and promotes practical, concrete team-oriented problem solving “on the gemba”, where the action takes place. Agile development says that conception (requirements and design) evolves continuously while a software product is being developed. This is in contradiction with the architect’s role as defined in the V software development model, which is the product of Taylorism applied to development, where the Architect (capital A) thinks a grand design that is built by others. The team-oriented approach of lean software development relies on democracy and consensus, where each voice is heard, and where no “design architecture” can be accepted as a “non-negotiable” requirement. Similarly, a key practice of lean/agile software development is synchronicity (everyone working on the same “takt time”), which is implemented through stand-up meetings, common project rooms and intense real-time discussion with the product owner, which means that the very concept of “planning” and “design” seems in jeopardy.

The first question I’d like to ask is, therefore, “do we need an architectural process when software development is led in a lean/agile mode ?”. My answer is a resounding “YES”, with the following three arguments:
  • Agile or lean software development does not prevent from building software products that are actually complex and require a carefully-crafted form of internal organization. Let us recall that architecture is foremost about communication: its role is to provide organizations and actions with a narrative sense.  In a lean software development process, architecture is a “co-creation”:  architecture and design continuously evolve together with product development; they are not “completed” until the product is.
  • In a continuous development and integration process, we need an architectural framework to prevent from diverging (too many additions that move the product in different and opposite directions) and maintain a continuous and efficient refactoring. Software code is a living object, which evolves together with the rythms of sprints and update deliveries. As a living object, constant “gardening” is required, which is guided by a vision (of what the garden should be), which is precisely what an “architecture” should be. As Jean-Claude Ameisen explains in his wonderful book “La Sculpture du Vivant”, the birth of a living organism requires as much pruning and removal as creating and addition; this applies to software : there should be as much thoughts and care put into code pruning and removal as there is in lines-of-code addition.
  • A key principle from Toyota/Lean Management is the importance of a systemic vision that is shared by all actors (sharing a systemic understanding is a key role of the Obeya room). Architecture in a lean software development process is not restricted to a few hand-picked specialists; it is everyone’s responsibility, especially developers’. A major feedback from SCRUM practitioners is the importance of “design reviews”. Organizing successful design reviews is better conducted by “experts”, who are people with experience (architecture is an art, much more than a science – one learns from previous mistakes). These are precisely “architects”, people who are well qualified to lead a successful review and to encourage learning/capitalization. Agile does not mean than everything needs to be rediscovered “on the gemba”; 30 years of software architecture have produced both useful principles and tools.

The second question to ask is then « What role should an architect play in this emergent software development process ? ». Part of the answer is common to the previous question and has been brushed in the previous three bullet points. The architect is required to shepherd the continuous refactoring and to lead the regular design reviews. This is quite different from the classical role as defined by the V-model. Let us emphasize the practical and cultural differences through three critical insights:
  • An architect needs to be a story teller. Agile processes suggest working on requirements expressed as “story boards”. Story boards make for “units of requirement” which are gathered into “backlogs”. The “architectural vision” also needs to be expressed as a set of “stories”.  Stories are a great form for expressing a requirement because they both carry meaning and they are robust. The heart of the agile process is that volatility, randomness, complexity of environment will force the product developers to adapt continuously during its completion. Strict and formal requirements tend to become obsolete or incompatible with one another too soon! Stories are robust because they can evolve; the combination of their meaning is context-dependent. I would qualify stories as “antifragile” following Nassim Taleb’s insight: the volatility and variations that will necessarily occur will “shake” the story and often produce even more value than the original thought contained. Serendipity is produced by the random shocks between stories from the backlog and continuous feedback from the users.
  • An architect works in a “pull mode”, she is pulled by the need for advice, assistance or explanation from developers. This is very different of the “push mode” from the Taylor-style organizations where the architect would hand out a document to the developers: “please implement the following architecture requirement R1 to R112”. This requires some form of change management, but this new way of working is quite fulfilling for an architect. She may feel the pleasure of participating to the actual creation, instead of being frustrated to see her “great principles, well-written in a forward-looking design document” ignored. This is the instantiation of a key lean principle that says that each actor in a process is at the service of the next actor (in the value creation process, from a chronological viewpoint). Kanban, both its industrial and software version, is the tool that puts this principle into practice.
  • Architects are stakeholders in the SCRUM backlog management: they ensure that performance tuning and bug fixing (especially “non-functional ones”) are maintained on the top of the stack.  Obviously the product owner is the key stakeholder, and these concerns should be hers/his as well. An architect is not a “chief engineer” in the Toyota sense (it is unclear to me if there is such a role in a lean software development process, although it is often used in the Devops context). As with any complex project, there exists a recursive/fractal structure associated with a product (systems and sub-systems). There are a number of “product owners” (for subsystems) and a master “product owner” who plays a role similar to that of a chief engineer. The architect is a member of the engineering team, a voice that reflects the value of long-term thinking, simplicity and “well-oiled” running. Using lean/agile does not remove the age-old conflict between short-term and long-term goals. Products tend to last (at least, customers wish they would), hence long-term characteristics are essential to a well-designed and well-built product.

To conclude this post, I would like to stand back and ask a more general question « What is the role of the architect from a systemic viewpoint ? ».
  • At the side of the product owner, the lean architect carries the critical mission to build the “situation potential” in Sun Tzu’s sense. The concept of “situation potential” is the cornerstone of François Jullien’s book on efficiency. In a complex world, both François Jullien and Nassim Taleb tell us that forecasting (as in “strategic planning”) does not work well any longer and one should become “opportunistic”, in the (positive) Chinese sense (i.e., be prepared to jump on opportunities). Being able to leverage the future, whatever it holds, is the heart of this “Art of War” strategy. It relies on agility: to react quickly as soon as the opportunity arises. It also relies on building the right set of capabilities: the situation potential. A “good architecture” is precisely a “situation potential”. This becomes even more obvious as soon as the necessity of a platform strategy occurs – the core message of “What would Google do ?” by Jeff Jarvis. Most successful software products are platforms (cf. the famous post by Steve Yegge). Designing a platform requires every developer to be eager to contribute to an “open API culture”, to be ready  “to eat your own dogfood” (the best way to implement the feedback circle that is necessary to reach quality). However, designing a powerful platform is precisely an architectural task.
  • A key tenet of “lean software”, as explained by Mary & Tom Poppendieck, is to free up creativity and innovation through cooperation. Architecture may be seen as the “grammar” for this cooperation. This is the core of “service-oriented architecture” (SOA). A software architect, irrespectively of the software development method (agile or not), aims to develop sharing and reuse of software components. This brings us back to the critical role of architecture to develop a software platform. This is lean & pretotyping at work: designing a platform is not (only) about thinking right, it is mostly about feedback loops and listening to users/customers. The value of a grammar is judged from the beautiful sentences that other write through it.
  • The more evolution we see in our current world, the more we need solid foundations to avoid losing our time building, destroying and re-building over and over. Layered architecture stays a relevant paradigm in the world of “lean software” because layer structure tells us the order in which things should be built and re-built. Experience shows that one cannot rely only of “auto-organization”: too many iterations – that entail massive rework – are necessary until a fixed-point is reached. Also, the practice of 5S (sorting, set in order, systematic cleaning, standardizing, and sustaining), applied to software (as explained by Mary Poppendieck for instance), is both relevant to code development and software architecture. There are many other interesting architectural contribution to a lean software product, such as the DCI architecture, which may be found in the book “Lean Software Architecture”.

This short post does not do justice to the complex topic of “lean software development” mixed with “software architecture”. The title, “Lean Architect”, applies to a long-term goal, which is to reconcile large-scale-systems-engineering with agile development methods. My own conviction is that “lean” – in the “Toyota Way” sense -  is the appropriate framework for such an integration. Lean adds to agile methods the combination of a global/systemic vision, together with a practical focus on concrete issues (“on the gemba”). This is actually the core of Nassim’s Taleb “Antifragile” book: complex systems are learned by doing, practice is superior to thinking – and this obviously applies to architecture. Because it emphasizes practice and respect for people, lean management is geared to address the complex challenges of our environment.


Benoît Déhais said...

Two complementary ideas :
- One of the key elements of the lean culture is to promote standardisation. The lean architect could be the one that generalises those good ideas coming from the "gemba".
- Architecture and standards may help in another goal of the lean approach : "develop flexibility"; especially when key ressources or teams are changing or when transfering to run teams that may have to operate various information systems in parallel.

Yves Caseau said...

Absolutely !
Standardization is a cornerstone of lean ... and architects are well positioned to play a role. This would be worth a development, since "standardization" has two different meanings in the world of software architecture (the promotion of standards) and lean management (capitalization and continuous improvement through a well-documented and shared representation of how the work is done). These two meanings have a strong intersection, which makes Benoit's first point so interesting.
I obviously agree with your second point as well.
One way to summarize your contribution would be:
(1) As always with lean manufacturing, a standard is needed to promote capitlization, continuous improvement, training and flexibility (through faster training)
(2) in the world of lean software, architecture is part of the standard (in the lean sense), and should include as much standards as possible (in the SW sense)

Sasouki said...
This comment has been removed by the author.
Sasouki said...

It's more an understatement 'une litote' or a paraphrase, than oxymore;
Without trying to offend purists, i would replace lean by architecture, in the software environment:

The architect is required before even think about how to manage: he thinks primarily on the feasibility of the system, designs, challenges choices, he plans tasks, selects the profiles and before conception, he wondered how to improve ‘standards’ without affect existing, he defines the load test scenarii , he continues to refine, asks everyone why, even if he knows the answer, he lets others give it (this is perhaps the reason why we like reading the work of architect, but much less frequent : because of his questions and our duty to respond: he's 'soualant') ...same thing as the lean?

As we can't claim to know lean, we should not proclaim to be architect: we become architect, becoming is continuously performed, never reached..
It's surprising to see 7 years duration for training a building architect, but anything similar anywhere for information-IT architect; hence architect has an undefined place in our IT environment (no record exists in the cigref nomenclature about design architect, only the OS profile is mentioned..)
We get however a great pleasure when meeting a good architect: he is often who uses the right sentence to describe the right topic with alternative options, the precondition to achieve a pretty advanced system, but if he's often followed, he's rarely heard..
I believe this is a good question to be analyzed…in order to not agree either with 'providence': even if the architect is employed at this position when disaster occurs (often unfortunately). Such providential savior role is antagonistic with the character of the architect to provide a quality solution and an efficient code design..
About this intervention of savior, the architect can enjoy (his ego is well developed..) but if he is cunning he may enjoy much more since he enhances quality at every step in the set of software life cycle.

The senior manager has to learn to recognize architect, and to promote their fulfillment, by promoting search environment, to diversify 'standards' that may sometimes inhibit the creative designers...
As you said, standards tend to make the world better organized, and because i have to share this view, i propose a couple of architects: who has head in the stars and whoever feet on the ground.
In fact i think these are the only profiles needed to make software development: an abstract / an implementer or business/ technical or logical / pratical... to reduce the whole set of developer, designer, integrator, architect, tester, manager...
Is it ambitious or unnecessary this double profile? i see it close to 'simplify to improve' ,
This is how the 2 attributes -architect and lean- are joined together, in my view.

Yves Caseau said...

I very much agree with most of what you wrote :)
I am currently reading "Lean Architecture" by J. O. Coplien and G. Bjornvig, which is a truly great book on this very topic. I will return to this in a future post ... and I will adress some of your comments.
Coplien writes about the combination of lean (Toyota-style, e.e., Liker's) and agile in a way which is very similar to "my" 'Lean Software Factory" ambition. One especially interesting idea is that lean is more focused on long term than "agile", with a nice complementarity (lean to build a long term solution, agile to adapt continously). I hope, therefore, to be able to revisit my previous writing with a deeper view once I have finished this book :)

Technorati Profile