Lean IT, Devops and Cloud Programming

I have become more and more interested with lean IT over the years. It started with the book "The Art of Lean Software Development" by Curt Hibbs. 

I enjoyed this book because its introduction of lean is 100% compatible of what I learned from Jeffrey Liker and other great authors about TPS. This simple book helped me draw the lines between good software development methods such as extreme programming or agile programming, and lean management. For those of you who are not familiar with lean or extreme programming, here is a very crude summary of some of the most salient points of lean software development :

• No delays: avoid as much as possible work that is sitting between two steps of the development process (what is called WIP – work in process – in the lean jargon). This is true at the process level (the goal is to design a streamlines/ “single piece flow” development organization) and the developer level. A practical goal is to avoid switching tasks as much as possible: focus on one thing and do it right!
• Quality in (right first time) :  this lean principle translates into testing as early as possible (a tenet from agile programming) but also to use all techniques that improve the quality of the code, even at the expense of source code productivity since we all know that it is cheaper not to produce a bug than to remove it later. Here comes, for instance, the practice of pair programming and code reviews, but also “good practices” such as programming guidelines and standards
• Fast delivery: the lean principle is to reduce the “lead time” of the software development process, which requires working on all stages. Removing in-between delays (cf. earlier) is necessary but not enough. Continuous integration is a core technique to achieve this goal, such as fast deployment techniques.
• Short deliveries: It is more efficient to produce small pieces of software at a high rate than bigger pieces at a lower rate. This is another key principle from lean (“small batches”), which is doubly true for software development: not only are l smaller batches easier to build (a well-known law of software engineering), but the continuous evolution of customers needs and environment makes it easier to adapt with a small batch approach.
• Less code: this is the application of the KISS principle !  Lean software development tries to stay away from complexity (see later in this post).  Unnecessary code may pop up from many sources, lean applies a technique called VSM (Value-Stream Mapping) and a posture of “muda removal”.  Muda (waste) removal means to go through the process with the “eyes of the customer” and remove all that does not produce value from her perspective. VSM is a tool that tracks the value creation and assigns it to each step of the process. Lean software development aims at producing the right product, without unnecessary features. It is also an architecture principle (stay away from complexity) designed at simpler and faster maintenance over the years.
• Customer participation: the best way to produce only what is necessary from the user’s perspective is to ask her frequently ! This is why end-user/customer participation is a tenet of agile programming. When the customer is not available, the principle mixes with “small batches” to become : deliver fast, fail faster to succeed sooner.
• Regular work effort : the leveling of the effort is a key principle from extreme programming, the equivalent of “heijunka” from lean programming. A few years ago, when I was still a CIO, I started thinking about “extreme IT” (applying extreme programming at the information system level), and the sustainability of the effort is a crucial point. Regularity has a counterpart, which is discipline. Using methods and tools (such as configuration management, source code versioning, automated testing) is crucial. One should take a look at the wonderful talk from Mark Striebeck on “Creating a testing culture” at Google.
• Minimize handovers: complex tasks, such as writing software, are better accomplished when the responsibility chain is not truncated into too many segments. Working as a team is the only alternative to deliver complex projects (this is the topic of my third book). This is another insight from the Agile Manifesto, this is why today’s best practice it to assemble multi-disciplinary teams including software developers, marketers and designers.

There are a number of interesting books on this topic. The most classical ones are those from Mary Poppendieck, such as “Implementing Lean Software Development: from Concept to Cash”.  I had the pleasure of presenting my last book at the Lean IT Summit a month ago and to meet with Mary Poppendieck (a number of great talks, including hers, Michael Ballé’s and Georges Striebeck’s, are now available online).

There are also books which are not about software development per se but very closely related. I am especially fond of two books which I have reviewed in my French blog:

• The Lean Startup, from Eric Riess,
• The principles of Product Development Flow, from Donald G. Reinertsen

Thanks to Guillaume Fortaine, I have come to learn about Devops. Quoting from Wikipedia, “DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals”.  The more I read about Devops (Devops.com is a treasure trove of great articles), the more I find that Devops is the missing link between Agile/Extreme Programming and lean management (Toyota-style, hence Lean-Startup-style).  Although Devops claims to “help finishing what Agile Development started”, there are different flavors in what one may read on the web: bridging the development /admin & operations gap, making lean IT happen, making collaboration happen (and also),  bringing agility to operations,  which in turns makes it easier to leverage the benefits of new computing resources such as cloud programming. My first obvious interest with Devops has been the practical deployment of lean IT, as a followup of what I just explained earlier. For instance, Devops promote a number of interesting tools related to deployment automation, such  as GLU. It turns out that the “Cloud and Devops intersection” is also quite promising. Indeed, leveraging the strengths of cloud programming requires a shift in development/architecture culture that requires a “Devops-like” approach.

It happens that I gave a talk last week at the Model-Driven Day about « Complexity, Modularity and Abstraction” (the talk is available in the “My Shared Document” box on the left). The talk is about, among other things:

• Complexity (why it is important, how to measure it and to tame complexity – since avoiding complexity altogether is not an option at the IS level)
•  Sustainability: how to transform enterprise architecture into a regular practice. This is related to the concept of “extreme IT” : avoid the “heroic struggle” to move towards the “continuous transformation of the information system”. This is a major reason why I have been advocating for SOA as a company-wide enterprise architecture practice for many years.
• “Architecture-oriented services” : I made this pun to emphasize the difficulty to produce the “right” services through SOA. “Architecture-oriented” means services that have the right level of abstraction, that are modular and “easy-to-compose”. To my knowledge, there is no easy recipe for this, but the wisdom and folklore of 40 years of software architecture design apply.
•  Cloud computing: I have added a small addition about “cloud-ready architecture” in the 4th edition of my first book. I strongly believe that Information Systems will change in a spectacular manner when we learn how to exploit massively parallel architecture (using tools/approaches such as Hadoop/MapReduce). Using Cloud Computing to provide with a few Saas-based front-office services is nice and relevant, but the big change comes when cloud-computing is applied to back-office services (provisioning, billing, data mining). This requires an architecture change, but mostly requires a culture change.

The thought that prompted me to write this post is as follows. Devops is the missing link between the themes of my MDD talk: managing complexity, delivering agility/modularity and moving to the new century of massively parallel computing (including, but not restricted to, cloud computing). I speak of "Cloud Programming" in the title because I agree with a comment made by Georges Reese: the Cloud is a computing resources defined by its API, i.e., a resource that is managed through programming. The exact quote is:

Cloud is, for the purposes of this discussion, is sort of the pinnacle of SOA in that it makes everything controllable through an API. If it has no API it is not Cloud. If you buy into this then you agree that everything that is Cloud is ultimately programmable.

 Not only the cloud is more agile (up/down scalability), it may be controled by the piece of software which is using it. From a systemic perspective, it's a "whole new game" :)