What does “doing agile” mean ?

 

11. Introduction

Agile methods have clearly won the software development popularity contest and Agile ways of working have become a necessary component of ever modern company’s strategy; Everyone want to “do agile”. Agile is no longer an IT topic, it has become an organization method that applies to the whole enterprise. Most companies claim to “do agile” or to have started an “agile transformation”. This is actually not a surprise since Agile actually works and delivers value in a VUCA world. Each letter of the VUCA acronym (Volatile, Uncertainty, Complexity and Ambiguity) represents a challenge for the traditional (Taylor-influenced, planned, decide-then-execute, specialize activities, …) ways of working in general, and the waterfall software development process in particular. Agile methods are iterative, based on continuous feedback from the environment and the collaboration of multiple viewpoints (design, development, operations,  …) that brings solutions to the VUCA challenges. You may read the BCG article “Why Agile Works” by David Ritter and Linsay Chim to get some additional arguments. They explain that “agile organizations establish an unbroken chain of Why”, between purposes, outcome and work, that makes the outcome much more robust to the VUCA uncertainties: “one of the most powerful benefits of agile is the ability to quickly recognize when things are going off course and to adjust on the basis of learning”.

Agile in a VUCA world has many flavors because it may focus on different goals. First, it may mean to better innovate, to better satisfy the customer, in an ambiguous and uncertain world. What matters then is the iterative nature to co-develop with the customer. It may also mean to achieve a difficult outcome in a complex world (digital transformation is not always about innovation, it also means to do the things that your customer expect right). In that way, what matters is the short cycle time, the “fail fast” capability. Agile may also be about constant adaptation to your volatile environment (digital homeostasis). In that case, agility is not simply about mindset or ways of working, it is also a characteristic of your digital systems. As Arnaud Lemaire points out in his wonderful “Let’s reset Agile”  talk (in French), if adaptability is your goal, what matters is how much time it takes to change a component … which is definitely as much a property of your code (how elegant your code is, how modular your architecture, …) as a property of your process.

Today’s post is a review of “Doing Agile Right”, a book that explore what how companies should adopt the agile approach to innovate and deliver more value to their customers. Learning to “do agile right” takes time, hence this is a book about agile transformation. Agile transformation is a bigger challenge as soon a scale kicks in, when the legacy of existing systems and operations has to be factored in. Agile software development for a small independent team developing a new product has never been a challenge. The challenge starts when a company needs to increase the rate of change to its existing organization (people and systems). This is the topic of this book and, therefore, the topic of this blog post.

 

2.  Doing Agile Right

 

“ Doing Agile Right : Transformation Without Chaos” is a book written by three Bain partners, Darrell Rigby, Sarah Elk and Steve Berez. This is first and foremost a book about Agile transformation. It is based on two key ideas : Agile works and the transformation towards agile ways of working is long and difficult. A good part of the book provides with example that show that agile delivers value and more precisely, agile helps companies to deliver innovation in a VUCA word. For the authors, agile is a business mindset change, much broader than the software development aspect of innovation. Since the book is written by Bain partners with a lot of combined experience assisting large companies, this book focuses on such companies. Hence this book contributes to the testimony that agile can scale (even if it is difficult) and that “agile at scale” works for many companies. However, and this is what gives the title to the book, to deliver value at scale, one must understand “deeply” how agile works and, therefore, “do Agile right”. It is important to “do Agile right” because, according to the authors – and many other observers – many companies are “doing it wrong”, generating frustration and lack of significant and long-lasting results.

This book holds a great review of the history of agile, starting with a reference to the paper that is recognized as the seed in agile development methodology :  “ In 1986, Ikujiro Nonaka and coauthor Hirotaka Takeuchi published an article in Harvard Business Review called “The New New Product Development Game.” Studying manufacturers that were releasing successful innovations far faster than competitors, the authors identified a team-oriented method that had changed the design and development process”. I am grateful to the authors of “Doing Agile Right” to have prompted me to read this seminal article. Reading this 1986 paper in 2021 is fascinating, because so many of the key ideas about Agile are already in the paper, with deep insights about why this “new way of (software) product development” has become necessary in a VUCA world. Teams play a key role; they are described as autonomous, with cross-fertilization (cross-functional) and self-finality. Some of the major traits of the Agile-to-become method are the multi-learning, the overlapping of design and development, and the chaos-friendly “subtle” control (because the goal is to orchestrate a group of teams, not to manage a single one). Some of the key ideas about the role of the product generations as support for transfer learning that I learned in 2012 thanks to my experience with set-top box design are already there, beautifully articulated. Let me last mention a great quote – to fill you with the desire to read for yourself – about the role of products: “Third, management should assign a different mission to new product development. Most companies have treated it primarily as a generator of future revenue streams. But in some companies, new product development also acts as catalyst to bring about change in the organization.”

As usual, the following is not a proper book review, it is a curated selection of the eight topics/ideas that I found most relevant.  I encourage you to put your hands on your own copy of the book, it is an entertaining as well as educating read.

 

2.1  Agile is a method that has a proven track record to help companies to innovate

The first key message of the book should come as no surprise to the readers of this blog : Agile way of working is well suited to our VUCA world and delivers value.  The authors give multiple examples of companies that have switched to agile approaches to better innovate:  “The reasons for agile’s rapid spread are neither obscure nor surprising. Most big companies find it difficult to innovate. They are weighted down by the structures and procedures of bureaucracy”. As the readers of Accelerate know, this is not a matter of opinion, there today enough studies to demonstrate that claim : “These are grand claims, but the data support them. Study after study find conclusively that agile teams are far more successful at innovation than teams that work in traditional fashion”. Companies are adopting the agile way of building new products, services, and software systems and this helps them deliver more value in volatile and uncertain world. In a world that changes constantly, adapting your products, services and systems is a necessity, innovation becomes a rule : “Insufficient focus on innovation leads to a static enterprise that will fail to adapt to changing conditions. Insufficient emphasis on operations creates chaos—poor quality, high costs, and dangerous risks to customers and to the business.” Last, if Agile transformation has statistically proven track record of delivering customer value, it is a deep transformation that takes time : “Organizations embarking on a transformation to an agile enterprise are like triathletes in training. It’s an ambitious project. There is an optimal pace. It’s likely to take years to come to fruition. But successful companies will be able to do things that few others can even contemplate”.

Agile is foremost a mindset, a business attitude towards your context and your environment. To be agile means mostly to recognize that your environment changes constantly in a way that you cannot foresee, so you replace planning with a regular and deep pattern of observation, then you take a corrective action and you iterate. This has nothing to do with technology to start with : “Agile innovation works in situations beyond IT.  As we noted earlier, many people believe that agile began in IT and works only there. They are wrong on both counts”. The authors give multiple accounts of companies who have adopted agility as a business attitude, such as NPR: “It’s common to hear, for example, that agile is great but only for technology-based innovations and the IT departments that generate those innovations. This will come as news to National Public Radio, which used agile methods to create new programs”. Because the agile mindset is focus on adapting, hence observing the environment, the agile approach is “outside-in” and requires team autonomy and the reversal of the traditional top-down chain of command : “Agile teams work differently from chain-of-command bureaucracies. They are best suited to innovation—that is, the profitable application of creativity to improve customer solutions, business processes, and technology”.

 

2.2  Doing Agile Right, as opposed to “faux-agile”

In his famous article « The State of Agile Software », Martin Fowler describes what he calls “faux-agile”, the abuse of the form of agile over substance: “Our challenge at the moment isn't making agile a thing that people want to do, it's dealing with what I call faux-agile: agile that's just the name, but none of the practices and values in place”. Faux-agile is a plague because it fits the need for control that is still common in many organizations: it is based on procedures and tools, with an emphasis on structure and organization to the detriment of team’s autonomy. This creates a static (and heavy) framework – one size fits all – where software development methods should adapt and evolve continuously. Faux-agile downgrades technical skills which is critical to build agile systems (agility is also a property of the systems that are built) and the focus on processes tend to stick companies with a project mindset where most often a “shift from project to product” is required. “Doing Agile Right” proposes a description of “Agile without meaning” which is very similar to “faux agile”. The authors have tracked and observed the persistence of the Taylorist mindset (planning over observation, thinking over execution) that produces “faux agile”: “But his approach caught on, and indeed it has long outlived him—even today, many companies have plenty of managers and executives who are Taylorists at heart. And when Taylorists try to implement agile, bad things happen”. The thirst to be in control, what Frederic Laloux calls “the beautiful illusion of control”, means that every random incident that is the signature of a VUCA world will create the strong desire to return to “the safe ground of command and control” : “It’s a version of Gresham’s law: bad agile drives out good. If that happens too often, agile will be discredited—and the business world will be back where it started, with top-heavy bureaucratic corporations struggling hopelessly to keep up with brash upstarts and rapidly changing markets”.

I strongly recommend this part of the book since the authors propose their insights about what causes the “rise of faux agile” : managers being out of touch with the reality. This is why Lean is so insistent on “Genchi gembutsu” and “gemba walks”, if you do not walk out to see by yourself and if you manage according to PowerPoints, the VUCA nature of the world is guaranteed to escape you. The disconnect from reality can be surprisingly strong : “ At first we thought the executives must be lying, but we soon discovered they are merely out of touch. They are so distant from the agile work that they only know what subordinates tell them, and subordinates tell them only what they want to hear”. The next quote is so close to situations that I have seen in my previous professional lives that I applaud the audacity: “Instead of responding to change, program management offices build complex Gantt charts with bright red dots to flag people who deviate from plans”. I will conclude this section with another quote from Martin Fowler : “A team should not only choose its process, but continue to evolve it”.

 

2.3  Agile where it matters

This book sends a strong message that you need to understand how agile works to succeed with your agile transformation, but also to understand when and where it is needed. Agile is not a panacea that replaces “the errors of waterfall approach” : “Agile, Agile, Everywhere Some agile gurus pitch the approach as a panacea that must replace bureaucracy everywhere—in every company, in every business unit, in every function”. Traditional project management has an amazing track record of success for very large and complex projects of the past century. System engineering, detailed specifications, proven specifications, simulation for validation are still very relevant in this century and many situations ask for the same level of forecasting and analysis, especially when machines and protocols are involved with no human insides. When a domain is stable (as opposed to VUCA), hierarchical decomposition and specialization (the signature of bureaucracies - “A bureaucracy works when its organizational tasks—what to deliver and how to deliver it—are clear, stable, and predictable”), waterfall project management is still appropriate. “The challenge, in short, is not to replace bureaucracy with agile everywhere but to find the right balance between the two. Every company must run its businesses. It must be good at operations. Every company must also change the business, continuously introducing not just new products and services but new operating methods and procedures”… “More agile is not always better agile. There is an optimal range of agility for every business and for every activity within a business”.

 

2.4  Agile for VUCA

As told in section 2.1, Agile is an iterative method that constantly validates and invalidates hypotheses made previously from how the “real world” (the enterprise’s environment) react: “Agile is founded on empiricism and the scientific method. It stresses that hypotheses should be tested against real-world results rather than trusting alluring theories or intuitions”.  This is the heart of the “small steps” principle: “So agile favors small batches, produced in time-limited (less than a month) work cycles called sprints”. From an optimal control viewpoint, we look to reach digital homeostasis, that is adapting the internal rate of change of the enterprise to the environment’s rate of change: “Ideally, an agile business system would operate at the golden mean between change deficiency—leading to a static business system that adapts too slowly to survive—and change excess, creating a chaotic business system that constantly risks spinning out of control”.

This is not a static or stable problem; it requires a continuous and adaptive transformation. This is why “faux agile” is such a bad service to companies: “Agile practitioners also know that solutions, processes, and technology must continually adapt as customer needs change. They believe that agile teams are the tools best suited to develop innovative solutions when what to deliver, how to deliver it, or both, are vague and unpredictable—the typical situation when addressing customer needs”. Although this book focuses mostly on innovation, it is worth underlying the deep match between agile as a complex system optimal control process and the nature of the VUCA world:

• Agile matches Volatile: iteration and small steps are here to constantly adapt to volatile (frequent) changes. This is precisely the point made in the first paragraph about homeostasis.
• Agile matches Uncertainty: Not only changes are frequent but they are hard to forecast. The heart of agility is frequent observation of your environment. High frequency sampling replaces planning.
• Agile matches Complexity:  complexity produces “unforeseen consequences”. Small steps and frequent adjustments are a great tool to master the pains of complexity. The principle of continuous integration (to rebuild as frequently as possible, so that if you have made a mistake, you find out before you may cause too much damage) is a perfect illustration.
•  Agile matches Ambiguity: agile replaces the specification of the results (how to achieve the desired goal) by user-centric user stories that describe outcomes.

 

2.5  Agile can scale

“Doing Agile Right” is a book about the agile transformation for large companies. It addresses the issue of “agility at scale”,  that is, how does one adapt agile methods with a large number of teams that needs to collaborate on a common goal (product, service, etc.). My own experience matches what the authors report: agility at scale is necessary, it can be achieved – even though it is clearly a much more difficult task because collaboration and synchronization require communication – and many companies have found ways to make this work : “Large-scale agile teams of teams also improve results.  Despite concerns that agile was designed for individual teams and can’t scale effectively, research shows otherwise”. The book contains a discussion about scaling framework that is not so different from my own review (of SAFe, LeSS and Disciplined Agile): “ The latest entrants include the Spotify Model, Disciplined Agile Delivery (DAD), Large Scale Scrum (LeSS), Enterprise Scrum, Lean Management, Agile Portfolio Management (APM), Nexus, and Recipes for Agile Governance in the Enterprise (RAGE). … About 30 percent of companies scaling agile say they use the SAFe framework. It is by far the most detailed and prescriptive approach”.  If you understand that now framework will substitute from a deep understanding of how agile works, and even more importantly if you understand that any framework places your company at risk to converting to faux-agile, then either SAFe or Scrum@Scale (Scrum of Scrums) are interesting choices that bring value : “SAFe’s strengths include the depth and breadth of its prescriptions, its training programs, its big-picture view of performance beyond the team level, its appeal to control-oriented executives, and its ability to coordinate interdependencies among teams”; “ Scrum@Scale’s strengths include its ambition to improve the agility of the entire organization; the complete consistency of the framework with successful Scrum values, principles, and practices”.

2.6  Agile Teams … and Squads

Teams are the foundation unit of the agile approach. Teams must be autonomous, small, cross-functional and empowered: “To tackle an opportunity, the organization forms and empowers a small team, usually three to nine people, most of whom are assigned full time. The team is multidisciplinary”. Empowerment is critical, the book gives many examples about companies that have undergone this transformation and tells how to recognize when this is the case: “I want to walk into an auditorium and ask, ‘Who owns the member’s change-of-address experience?’ And I want a clear and confident response from a team that owns that experience, whether a member is calling us, logging into our website on a laptop, or using our mobile app”. Empowerment means that the role of manager changes, as is now abundantly clear: “In an agile environment, leaders take a different approach. They may tell a team what to focus on, but never how to do it. Figuring out the how is up to team members themselves”.  There is a dual synergy between agile teams and agile way of working, you need teams to deliver agile value, but agile ways of working also produces a better work environment for team: “Compared with traditional management approaches, agile offers a number of major benefits, all of which have been studied and documented. It increases team productivity and employee satisfaction. It minimizes the waste inherent in redundant meetings, repetitive planning, excessive documentation, quality defects, and low-value product features”.

The team is multi-disciplinary (cross-functional) with no boundary between “business” and “technology” (for digital products and services, the boundary is very blurry, anyhow): “Having the business and technology people work together as one team is essential to success.” The authors have collected a number of testimonies about the necessity of merging the business and technology viewpoints: “Having these groups work separately, even with the best efforts at alignment, would not have given us anything close to the speed and product quality we require”. Talent matters very much in agile teams. The following is one of the critical feedbacks from companies that have embarked on their agile transformation: “I wish our company had started work on talent earlier.” We can scarcely count the number of times we have heard that regret from executives whose companies are launching agile”.  “Doing Agile Right” is a very practical book: each chapter ends with a clear summary and a few questions that you should ask yourself as a maturity assessment of your own practices. For instance, here are two of the questions that you should ask your agile teams: “How can people collect more feedback from customers?” and “How can employees minimize work in process?”

The term “Squad” has been introduced by Spotify and I have regularly used it as a short-cut: a squad is a team that fits the agile ways of working: size, cross-function, autonomous and empowered. This book makes a few references to the “Spotify model”: “The Spotify model is intuitive and easy to understand; it works well in Spotify’s engineering department, though it is not a significant factor in areas such as strategic planning or finance”. I will conclude this section with a few personal remarks about this agile model. Like many people I liked the “Spotify model” when it came up 10 years ago because I found value in:

•  The use of “squad” as shorthand for cross-functional and autonomous.
•  The squads / tribe (teams of teams) model which was a good fit for my own challenges.
• The concept of chapters and guild to add a matrix dimension for skill capitalization.

Since then, there have been many warnings saying that taken literally, the “Spotify model” does not work. For instance, you may read the strong critics of Jeremiah Lee in “Failed Squad Goals: Spotify does not use the spotify model and neither should you”. I actually still appreciate the value that I found in the Spotify papers in 2011, and I have applied successfully some of its ideas. Furthermore, I dislike this kind of after-the-fact criticism (sure, we understand scaling agile better in 2020 than we did in 2010 and nobody is forced to take ideas illustrated in a position paper as rules), but I would definitely agree with the three following warnings from Jeremiah Lee:

• Team autonomy is good but there is such a thing as “too much autonomy”. Teams must collaborate to build “one system” and orchestration of squads is a tough (architecture, to start with) topic.
• Scaling requires more structure, management and tools than what was outlined in the first Spotify papers. Tools such as PI planning are probably SAFe most useful contribution.
• The “Tech lead” role is critical. Software products need a “chief” (in the Toyota sense) who embody the technology vision and mindset, as much as they need a product manager.

 You may think that this debate about taking a paper or a book literally is unsignificant, but it is not. In a VUCA world, for the very same reasons that make managing throw PowerPoint impossible (the map is not the territory), you should not take a document (this blog post, the book that I am commenting, the SAFe framework or the Spotify papers) as rigorous guidelines to be followed.

 

2.7  Customer focus

An agile approach should always start with the customer: “ So how should agile teams go about innovating processes? In some ways, process innovation is much like any other agile innovation. You start with the customer and work backward to solve their needs in an incremental, iterative way … As a philosophy, agile focuses intently on customers”. This why user stories a critical role in the agile approach. The book gives interesting examples, such as Dell, about how agile companies gather extensive customer input and let the customer be the best judge on what they want. This is not so easy, as is illustrated by a number of anecdotes such as this one: “He realized that simply putting together cross-functional teams and giving them a mission wasn’t enough. He had been hearing for some time about agile innovation from colleagues inside and outside the bank, and realized that a broader set of agile practices could make the teams more effective and better sustain their success. After learning more, he decided to embark on a customer-focused agile transformation, beginning with the home mortgage business”.  Once the customer voice is onboarded, the importance of design, from user experience design to interface design, becomes critical: ”Led by the designers, people on the team conducted customer research to gather feedback. Team members with operations and customer service backgrounds then designed new digital and people-based processes to create the experience that customers wanted, and the team’s software engineers wrote the code to enable these new processes, assisted by data engineers and data analysts who ensured the availability and maintenance of accurate data”.

 

2.8 Lean & Agile, Flows and Metrics

This book touches many times the topic of Lean & Agile relationships and the importance of flow, as Mik Kersten did in his great book “Project to Product”. Flow efficiency is a critical concept for large organization, especially in the context of a large transformation. Without flow efficiency, local progress may be totally hidden and the agile transformation may stop : “But the most difficult problem—and one that is likely to be counterintuitive—is this: even though agile teams may develop innovations better and faster than ever before, leaders are likely to find that the company’s overall innovation velocity is not improving. When they investigate that problem, they uncover the concept of flow efficiency”.

Another important part of the book discusses metrics, which is a hot topic for agile methods, and strikingly difficult when one starts an agile transformation in a controlled-obsessed company (which is both an oxymoron and a statistically significant occurrence). The authors propose an in-depth analysis based on five kinds of metrics: inputs, activities, outputs, outcomes and purposes. The two salient ideas are that, first, companies need to understand the difference between the five (as the Chinese proverb says: one does get a plant to grow faster by pulling the stem). Second, each team should be left responsible with the way it organizes input, activity and output metrics to be organized towards continuous improvement, whereas outcome and purposes metrics are meant to be shared: “Agile teams set clear goals for themselves. They try to understand what goes well and what doesn’t go well as they pursue those goals”. This is where reading the seminal paper “The New New Product Development Game”, referred to in the introduction, is enlightening. Some metrics should be used as KPI because of their capacity for alignment and because they define the share strategy. Some metrics should be used to better understand performance and efficiency and should be left to the teams since turning them into goals is counterproductive.

I would end with a word of caution about one of the best practices proposed by the authors: “ They Tie Funding to Outcomes”. Although there is a systemic elegance in the idea to replace budget funding with a adaptative flow that is linked to the outcome (give more money to agile products when they have achieved the desired outcome), it is very important to discuss the “timescale” for this piece of advice. On a long timescale, this is obvious: one should not keep funding a team who does not deliver. On a shorter timescale, it becomes much more subtle: digital companies do make bet and invest funding for a while before they see outcome.  The funding should definitely follow an agile iterative process of its own, but the frequency should be different from the system delivery cycles and the logic reflects the principle of “affordable loss”.

 

3.  Conclusion

The most important conclusion that one may draw from reading this book is that agility matters. Agility matters since it helps companies to deliver more value – better innovations – in a VUCA world. Agility is a tool for constant adaptation to the environment, when enterprises are seen as living organisms. Agility as an active pillar of digital homeostasis – constant adaptation to the fast rate of change of the digital world – is a key principle of my own book, “The Lean Approach to Digital Transformation”. “Doing Agile Right” is a great contribution to the better understanding of how agile works in large compagnies. There is now a broad consensus. I have started with a BCG paper in the introduction, I will refer to a McKinsey article in this conclusion, “The five trademarks of agile organizations”. A quick summary of these five traits should come as no surprise to you, at this point in your reading : (1) network of empowered teams, (2) autonomy but a shared “North star” goal – a common finality that is embedded and embodied (we are still humans) across the organization, (3) rapid decisions and learning cycles, (4) Passion and mindset – the focus of “Doing Agile Right”, (5) Next-gen enabling tech.  The last item is important, because in a digital world, technology matters.

“Doing Agile Right” is not a book about IT, nor is it a book about software or software architecture. IT is mentioned a couple of times, to point out that IT departments have to be a part of the solution rather than being part of the problem: “Sad to say, IT departments and the software they develop are notorious for creating such difficulties. A common issue for some companies is spending millions on customized software when standard off-the-shelf solutions would meet their needs”. A few pages talk about the importance of modularity and service-oriented architectures – with the expected reference to microservices: “Design Operations as Modular Capabilities Today’s software systems are typically built as microservices—small, modular units of functionality with clearly defined interfaces”; “A modular arrangement like this allows an agile team to improve the functioning of the capability without worrying about interfering with other parts of the organization”. Although there is nothing wrong with these two statements, they represented a simplified and somehow naïve view of information systems architecture. I refer you to the excellent “Designed for Digital” book – in addition to my own - to get deeper insights about this. If you are familiar with the Kano diagrams, you may easily understand the paradoxical importance of IT/software/technology in the success of digital transformation. Success, as defined by customer satisfaction and value creation, is much more a matter of mindset and culture than it is a matter of technology. However, if the technology capabilities are not there, failure is the most probable outcome. In terms of the Kano model, IT and software capabilities are a “basic need” of digital transformation.