What Today’s AI can and cannot do ? (Part 1)

1. Introduction

This last post of 2018 returns on the topic of the year : Artificial Intelligence and Machine Learning. Many new books have been published in the second part of the year, including Kai-Fu Lee’s Book “AI Superpowers – China, Silicon Valley and the new world order”. This thought-provoking book has achieved the New York Times bestseller status, rightly so in my opinion because it brings the multi-cultural insights of someone who knows very well the US AI ecosystem – having worked, for instance, both at Apple and Google – and the new vibrant China AI ecosystem. Besides, Kai-Fu Lee has made a number of provocative press conferences and interviews where he is very forceful about China and US superiority versus the rest of the world, especially  Europe, where he sees “no chances for the AI sector”.

The core argument of Kai-Fu Lee’s Book is that Artificial Intelligence has moved from Science to Engineering. This is a very strong statement, with two parts: (a) the incredible advances of the past 10 years are ready for massive deployment and will have a huge impact on the work, (b) what matters is there already and what will come next is too speculative to be a foundation for the next ten years to come – hence the implementation race has started. The first part is no surprise, the same message is at the core of our NATF report on AI and ML, or in the two previous great books that I have quoted many times : “The Mathematical Corporation - Where Machine Intelligence and Human Ingenuity Achieve the Impossible” and “Human + Machine – Reimagining work in the age of AI”. The second part is much more controversial.

Indeed, many voices are saying the opposite of the second part of the statement: AI today is still in its infancy and we have many hurdles to address before we can scale to truly significant achievement. Among those, the great paper from Melanie Mitchell, “Artificial Intelligence Hits the Barrier of Meaning” has received a lot of well-deserved attention. One could see a contradiction in the Kai-Fu Lee’s versus Melanie Mitchell’s “state of AI” viewpoints. I believe that they are both right (hence a little bit wrong as well), as is beautifully summarized by this great quote from Pedro Domingos – the author of the “The Master Algorithm” bestseller :  “People worry that computers will get too smart and take over the world, but the real problem is that they’re too stupid and they’ve already taken over the world”. My goal with this two-part blog post is to deep dive into this apparent contradiction.

Fortunately, another great book was published recently by Martin Ford, “Architects of Intelligence: The Truth about AI from the People Building it”, which is clearly the best book that I have ever read on this topic. This book is a series of 24 interviews from some of the best minds in the world – a great selection of the best-known scientists who have made AI advances in the past 30 years. This book, in my opinion, gives all the necessary clues to see all that is relevant in Kai-Fu Lee’s great book – including the call to action and the dire warnings – but also to see that this is only a small part of the story : the future of today’s AI. The other part, the future of tomorrow’s AI, is by construction much harder to speculate about but it does not entail that what we not know will not exist J The first part of this post will address the future of today’s AI following the steps of Kai-Fu Lee. The next part – in an upcoming blog post – will look at the other half of the question following the footsteps of the “Architects of AI”.

2. AI Superpowers – The upcoming revolution of today’s AI

This very synthetic book review should include a more detailed profile on the author, Kai-Fu Lee, that you may find here. It is important to understand that Kai-Fu Lee has started his career as a scientist in the field of AI  - “As a young Ph.D. student researching artificial intelligence at Carnegie Mellon University, I studied under pioneering AI researcher Raj Reddy. In 1986, I created the first software program to defeat a member of the world championship team for the game Othello”. He has worked in many prestigious Silicon Valley companies and made significant achievements in the field of natural language processing. He then returned to China as the head of “Google China” before quitting to start his own VC activity in China. The book is well written, deeply personal (I will not talk here about the last part, which you should read by yourself), and I will simply here point out some key ideas without any pretense of completeness.

The book starts with a recollection of AlphaGo victory over Ke Jie. After the spectacular victory over Lee Seedol that stunned the world, AlphaGo beat in May 2017 the Chinese champion considered the best world player at that time. Kai-Fu Lee emphasizes the importance of Go in Chinese culture and how it had been associated for a long time with human intelligence. Hence the “machine victory” came as a huge surprise and a very strong waking call for Chinese government and entrepreneurs: “To people here, AlphaGo’s victories were both a challenge and an inspiration. They turned into China’s “Sputnik Moment” for artificial intelligence”. Kai-Fu Lee considers that this victory over the Go champion played a critical role for the Chinese government to make a strategic investment into AI: “ And less than two months after Ke Jie resigned his last game to AlphaGo, the Chinese central government issued an ambitious plan to build artificial intelligence capabilities”. This is the starting point for the rest of the book, which describes how China has become the biggest playfield for Artificial Intelligence development in the world: “But over the past three years China has caught AI fever, experiencing a surge of excitement about the field that dwarfs even what we see in the rest of the world”.

A foundation idea of this book is that AI requires a lot of data, and the more data you have the better your AI will be: “Given much more data, an algorithm designed by a handful of mid-level AI engineers usually outperforms one designed by a world-class deep-learning researcher. Having a monopoly on the best and the brightest just isn’t what it used to be”. We see the same pattern here, the combination of a well-admitted idea “you need lots of data to develop world-class AI capabilities” – there is not one book or report that does not explain this – and a more debatable point that says that if you have more data, you will necessary win, independently of the amount of talents. Kai-Fu Lee uses the remarkable story of machine vision, and the role of the ImageNet data set, as an illustration of his vision: “Doing this requires massive amounts of relevant data, a strong algorithm, a narrow domain, and a concrete goal. If you’re short any one of these, things fall apart”.  The importance of massive data collection is the first argument about China’s future superiority. China is collecting more data because of its size, because of the advanced status of digital life in China – from WeChat being the ubiquitous digital platform to the massive deployment of connected objects and mass-market IoT, and because regulation is much more “friendly” towards leveraging value through AI : “the rich real-world interactions in China’s alternate internet universe are creating the massive data that will power its AI revolution”. There is a very strong argument here, that we shall see in Martin Ford’s book as well, that AI development thrives in a fully digitalized society. The result is the first competitive advantage of China according to Kai-Fu Lee: “China has already vaulted far ahead of the United States as the world’s largest producer of digital data, a gap that is widening by the day”.

A key conviction of Kai-Fu Lee is that, after a decade of deep learning scientific innovation, we are entering a decade of implementation, where critical masses of data, talents and energy will tell who wins the race. He sees a paradigm shift, where the question is no longer to develop new tools but to find as quickly as possible – because this is a race – where value may be found using our existing tools: “ That global shift is the product of two transitions: from the age of discovery to the age of implementation, and from the age of expertise to the age of data”. Following the two books that I quoted in the introduction, Kai-Fu Lee issues a strong call to action: “But making that distinction between discovery and implementation is core to understanding how AI will shape our lives and what—or which country—will primarily drive that progress”.  The paradigm shift towards implementation means that software and engineering skills play a critical role, which I could not agree more since this was the key message from our NATF report. There are many examples in the book where Kai-Fu Lee insists on the specificities of software architecture and solutions that were developed by Chinese entrepreneurs to adapt to their markets: “Chinese companies have never truly embraced enterprise software or standardized data storage, instead keeping their books according to their own idiosyncratic systems”.  I will quote his argument for “growing AI from local usage specificity” because it is both very true and somehow contradictory with the concept of AI worldwide power-houses: “AI has a much higher localization quotient than earlier internet services. Self-driving cars in India need to learn the way pedestrians navigate the streets of Bangalore, and micro-lending apps in Brazil need to absorb the spending habits of millennials in Rio de Janeiro.”

As previously said, this paradigm shift comes with the – debatable – conviction that AI today’s toolbox is now complete or good enough: “Much of the difficult but abstract work of AI research has been done, and it’s now time for entrepreneurs to roll up their sleeves and get down to the dirty work of turning algorithms into sustainable businesses”. This viewpoint is derived from the conviction that the time from the initial idea to market-readiness of software technology is long, may decades: “Deep learning marked the largest leap forward in the past fifty years, and advances on this scale rarely come more than once every few decades. Even if such a breakthrough does occur, it’s more likely to emerge out of the open environment of academia”. It is clear that this is where many would disagree, especially Rodney Brooks, whom we shall meet in the second part of this blog, and his famous paper “The Seven Deadly Sins of AI Prediction”.

The core consequence of the previous analysis is that China will inevitably rise as one of world AI “superpowers” (hence the title) because it combines precisely the four following strengths. Kaifu-Lee gives his list of the four components that are necessary for a successful AI ecosystem: “ Harnessing the power of AI today—the “electricity” of the twenty-first century—requires four analogous inputs: abundant data, hungry entrepreneurs, AI scientists, and an AI-friendly policy environment”.  We have already talked about the “data collection advantage”, another fascinating part of the book is the description of the internet entrepreneur ecosystem in China. As a venture capitalist, Kai-Fu Lee is in a very good position to describe and explain the strength and the differences – versus Silicon Valley and Europe startup ecosystems. One should keep in mind though that, as an investor, this not an unbiased viewpoint – Kai-Fu Lee has definitely “much skin in the game”. To him, China’s ecosystem is simply “the most cutthroat competitive environment on the planet. They live in a world where speed is essential, copying is an accepted practice, and competitors will stop at nothing to win a new market”.  I will let you discover the – sometimes caricatural – description of the extraordinary energy and willingness to work long hours in the Chinese startups. As a VC, Kai-Fu Lee points out the rapid adoption of “Lean Startup concepts” (MVP, pivoting, etc.) in the Chinese ecosystem: “Core to its philosophy – The Lean Startup Book - is the idea that founders don’t know what product the market needs—the market knows what product the market needs”. The race that motivates Chinese entrepreneurs is the race of disruption through AI, they do not see themselves as technology partners to larger incumbent companies, but as challengers: “Instead of helping those companies access AI, these startups want to disrupt them using AI. They aim to build AI-first companies from the ground up, creating a new roster of industry champions for the AI age”. As mentioned earlier, Kai-Fu Lee sees that China is not simply a country with more data – because of its size and a government that is indeed more AI-friendly in its data privacy regulation – but also a country with better data : “But China’s data advantage extends from quantity into quality. The country’s massive number of internet users—greater than the United States and all of Europe combined—gives it the quantity of data, but it’s then what those users do online that gives it the quality. The nature of China’s alternate universe of apps means that the data collected will also be far more useful in building AI-driven companies … that data is tailor-made for building profitable AI companies”.

What makes the “AI race” exciting is that the goal is nothing short than a profound revolution and the opportunity to acquire new leader positions in the future economy. Because of the digital nature of most those AI-related opportunities, there is indeed a race: “The AI world order will combine winner-take-all economics with an unprecedented concentration of wealth in the hands of a few companies in China and the United States … The positive-feedback loop generated by increasing amounts of data means that AI-driven industries naturally tend toward monopoly, simultaneously driving down prices and eliminating competition among firms”. This is nothing new, but AI will amplify the opportunities: “But as remarkable as these accomplishments have been, these changes will pale in comparison to what these entrepreneurs will do with the power of artificial intelligence”.  Kai-Fu Lee gives numerous examples of Chinese startups that he sees as future world leaders of their fields. For instance, Face++ is described as “a world leader in face- and image-recognition technology”. He quotes another example from his own field of expertise – but one should keep in mind that, as a VC, he is necessarily biaised : “Nearly twenty years and dozens of AI competition awards later, iFlyTek has far surpassed Nuance in capabilities and market cap, becoming the most valuable AI speech company in the world”.  There is   an interesting discussion about the availability of skills and talents in China, and Kai-Fu Lee gives convincing arguments that China is ready to develop the successive waves of AI : “The complete AI revolution will take a little time and will ultimately wash over us in a series of four waves: internet AI, business AI, perception AI, and autonomous AI”. One could object that this enthusiasm, about Chinese investments and government strong push, reminds us  of the Japanese “Fifth Generation” program in the 80s. Kai-Fu Lee is aware of this : “While all of this may sound exciting and innovative to the Chinese landscape, the hard truth is that no amount of government support can guarantee that China will lead in autonomous AI”. His topic of interest is not the next generation of AI or how government funding may steer that race, but the current generation of AI and how a well-structured ecosystem will deliver faster growth.

The relative strong positioning of China in AI is not simply a matter of size, it is the consequence of a deliberate plan to develop and strengthen an ecosystem. I strongly urge any regulator to read carefully the story of Guo Hong, a startup founder who became a key player for the Chinese government and played a critical role in setting up the Chinese AI ecosystem. Guo Hong worked hard to understand “what really made Silicon Valley tick”, his first mission “to build the Avenue of the Entrepreneurs was just the first trickle of what in 2014 turned into a tidal wave of official policies pushing technology entrepreneurship”. A lot of thinking about emergent ecosystems and the role of data went into play “In the age of AI implementation, the impact of these divergent data ecosystems will be far more profound. It will shape what industries AI startups will disrupt in each country and what intractable problems they will solve”.   In comparison, Kai-Fu Lee is not impressed with the approach that Europe in general, and France in particular, is taking to succeed this AI race because they focus too much on research and not enough on usage ecosystems: “Canada, the United Kingdom, France, and a few other countries play host to top-notch talent and research labs, but they often lack the other ingredients needed to become true AI superpowers: a large base of users and a vibrant entrepreneurial and venture-capital ecosystem”. Our proclivity toward issuing reports is gently mocked, whether there are European or American reports: “But the report—issued by the most powerful political office in the United States—had about the same impact as a wonkish policy paper from an academic think tank”. Very logically, Kai-Fu Lee is not impressed with GDPR and the European desire to control rather than to develop AI-based usage of data: “While Europe has opted for a more heavy-handed approach (fining Google, for example, for antitrust and trying to wrest control over data away from the technology companies), China and the United States have given these companies greater leeway, letting technology and markets develop before intervening on the margins”.

Although Kai-Fu Lee is bullish on the future impact of AI on society, he is also quite aware of the complexities and difficulties of the journey ahead. He is especially sensitive to the social risk that AI automation is likely to cause:  “Does that mean I see nothing but steady material progress and glorious human flourishing in our AI future? Not at all. Instead, I believe that civilization will soon face a different kind of AI-induced crisis”. His vision of the possible impact on job loss is not very different from the Frey-Osborne study : “Based on the current trends in technology advancement and adoption, I predict that within fifteen years, artificial intelligence will technically be able to replace around 40 to 50 percent of jobs in the United States...  Actual job losses may end up lagging those technical capabilities by an additional decade, but I forecast that the disruption to job markets will be very real, very large, and coming soon”.  He proposes his own analysis in his book, based on a number of job characteristics, such as requiring or not dexterity, using a structured versus unstructured context and environment, or the level of social interaction that is required. With a two-axis / four quadrant approach, he makes his own predictions about which jobs will be impacted the most by AI-automation: “While the simplest and most routine factory jobs—quality control and simple assembly-line tasks—will likely be automated in the coming years, the remainder of these manual labor tasks will be tougher for robots to take over”.  I read where something that I first head two years ago at the Singularity University : “contrary to popular assumptions, it is relatively easy for AI to mimic the high-level intellectual or computational abilities of an adult, but it’s far harder to give a robot the perception and sensorimotor skills of a toddler”. There is an interesting discussion about the need for solidarity, inclusion and some form of redistribution – Kai-Fu Lee is not a proponent for Universal Basic Income, but what he calls for may not be that different : “I want to create a system that provides for all members of society, but one that also uses the wealth generated by AI to build a society that is more compassionate, loving, and ultimately human”.  This vision requires a massive investment in education: “To be clear, I do believe that education is the best long-term solution to the AI-related employment problems we will face … But the scale and speed of the coming changes from AI will not give us the luxury of simply relying on educational improvements to help us keep pace with the changing demands of our own inventions”. Eventually, many of the future jobs will turn to be “interaction jobs”, but we need to find a way to redistribute some of the AI-created wealth, since those jobs have no “market invisible hand” that would push their values forward: “Private companies already create plenty of human-centered service jobs—they just don’t pay well.”. As is proposed by most futurists, Kai-Ful Lee quotes “health and personal care” as a great example of such a domain: “The U.S. Bureau of Labor Statistics has found that home health aides and personal care aides are the two fastest growing professions in the country, with an expected growth of 1.2 million jobs by 2026”.

3. Temporary Conclusion  

To summarize, I believe that there are some excellent points in this book that make it a must-read, especially for anyone who is planning to use AI in their own business:

• The importance of data collection is well spelled-out, not only the quantity but also the quality and the virtuous circle that links usage data, data collection, data enrichment and value creation.

• The description of the Chinese digital and AI entrepreneur ecosystem is fascinating. Kai-Fu holistic vision of what it takes to succeed should be required reading for anyone in a public position planning to “foster” AI development and competitiveness.

• The book is full of great examples of applying today’s AI to real-life situations. This book echoes the NATF recommendation to companies: what matters to succeed with AI is the willingness to try and the speed at which you execute.

On the other hand, there are a few debatable points. They do not change my positive opinion about the book, but they mean that this book does “not cover the whole story”. More precisely:

• The book puts a very high emphasis on data and describes “a new world where expertise may be harvested from data”.  Actually, in most cases, data-driven knowledge benefits from prior expertise and does not replace existing knowledge but augment it. There is more to AI than Deep Learning, and there is more to creating value with deep learning than brute-force application independently of the expertise put in labelling.

• There is a strong claim that “this era will reward the quantity of solid AI engineers over the quality of elite researchers”. Here also, this statement applies to the “present of AI” but not to what will happen in the future. As is pointed out by Melanie Mitchell, there is so much that we do not know how to do that we clearly need “elite researchers” to push out new boundaries. Let us illustrate this with a thought experiment: If we are in 2028 and look at AI usage, what do you think the ratio will be, of the value created by today’s methods versus the new methods that will be invented in the decade to come: 10, 1, or 10% ? I have no crystal ball and thus no answer. But I would disagree with anyone to claims to know.

• As I stated earlier, some of the strong claims made by Kai-Fu Lee about AI newcomers, such as “the widespread adoption of deep learning in 2013 turbocharged these capabilities and gave birth to new competitors, such as Element AI in Canada and 4th Paradigm in China. These startups sell their services to traditional companies or organizations, offering to let their algorithms loose on existing databases in search of optimizations”, will have to be checked a few years from now.

I have been following the comments about the book on Twitter and found that I agreed with both sides. I am a big fan of this book, especially because of the wonderful insights about the China ecosystem. I have had the privilege to work with Chinese teams and partners in my past three jobs and to visit the Shanghai startup ecosystem. Reading the book has helped me to better understand what I saw (in the same way that reading “Start-up nation” is a must to go and work with Tel-Aviv entrepreneurs). I have also agreed with the negative comments that said approximately : “wait … there so much more about AI than what is said here” and those opportunities could be explored everywhere in the world, including France and Europe.

As stated in the introduction, this paradox can be easily explained with the reading of Martin Ford’s “Architects of Artificial Intelligence”. This will be the topic of “Part 2”.

From Digital Transformation to Service Architecture and Reliability Engineering

1. Introduction

This post talks about Digital Transformation from an operational excellence perspective. I propose a simplified book review of “the textbook” on reliability engineering as a path to hammer some key principles that make digital systems efficient. There should be no surprises with this approach, since I have been advocating for a while in my blogs that:

• Digital transformation is about digital capability
  
• Customer satisfaction is first about operational excellence
• The playground has changed - the GAFA have set new standards for operational excellence, as explained by the Web Giants book.

Systems’ quality of service (QoS) is a topic that is very close to my heart. I do not consider myself a true expert since I have not spent tens of years running systems full time daily (what it takes to be a true Ops expert) but I have a long experience thinking about and observing firsthand what makes systems reliable :

• I have been privileged to start my exposure to availability lectures as a teenager from my departed father Paul Caseau on topics such as queuing theory, Jackson networks and Markov processes. I was told about MTBF and MTTR from practical examples such as shoes when still very young.
• I started my career as an Operations Research scientist with many years spent on scheduling, planning and routing, with the later addition of stochastic optimization.
• I then got my first exposure to real-life QoS issues as Bouygues Telecom CIO 15 years ago. This resulted in my attempt to formalize what I saw in books, a new piece of simulation and research about SLA called “Optimization of Application Integration” and later the opening of this blog. The paper “Self-adaptive Middleware: Supporting Business Processes and Service Level Agreements” is very similar to what this post will describe.
• I had the remarkable opportunity of a few in-depth conversations with key SRE people at Google including world-class stars that are quoted in the book that I will present today.
• Last, my experience as a lecturer at Ecole Polytechnique has a helped me to formalize what I have learned from the trenches.

I plan to share a detailed summary of the “Google SRE” book because most of what I have learned and know from experience may be found in this book. On the one hand, this is a book about large-scale distributed systems and how to design and operate them in a reliable manner. This is a complicated topic and there is a wealth of knowledge to learn and share. On the other hand, the technology “lego box” has changed : things that were very hard to design 10 years ago are much easier today thanks to open source technologies such as cloud, containers, distributed orchestration, distributed storage platforms, etc. High-availability is no longer a “high end” feature and many small companies including startups build high-availability architectures with QoS performance that would have been hard to achieve for a telco 20 years ago. Many of the open source pieces that created this “reliability revolution” come from Google components that are described in the SRE book. Readers of this blog who know about “Autonomic Computing” will notice that Google has done nothing less than delivering an ambition proposed by IBM over 15 years ago. Self-monitoring, Self-optimization, Self-provisioning, Self-healing are characteristics of modern reliable systems

This is not a linear book review, because the material is very deep (the book is 475 pages long). I selected nine key topics that are closely related to digital transformation and to my experience as a CIO; then I tried to summarize the key insights from the book. Managers are not the main audience for this book, which is foremost written for practitioners. For technical readers this book is a treasure trove of insights, examples, and practical advice. However, I strongly suggest anyone to open this book and read at least the first hundred pages.

2. Google Site Reliability Engineering

The book “Site Reliability Engineering - How Google runs production systems”, edited by Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy, is a collective book from Google SRE members: “This book is a series of essays written by members and alumni of Google’s Site Reliability Engineering organization. It’s much more like conference proceedings than it is like a standard book by an author or a small number of authors”. This heavy book is filled with case stories and technical details about the different tools that are used by Google engineers. Although the 33 chapters are mostly focused on practical issues and problems, this is also a principled book about computing:  “We apply the principles of computer science and engineering to the design and development of computing systems: generally, large distributed ones”. True to the principles that I will present very soon, the book is rich with failure analysis and “anatomy of unmanaged incident” examples.

The relationship with the theme of this blog, “biology of distributed information systems” will become self-evident when you read this post. As mentioned earlier, the ambition of “autonomic computing” is also deeply embedded: “The global computer … must be self-repairing to operate once it grows past a certain size, due to the essentially statistically guaranteed large number of failures taking place every second”. 

2.1. Information System Reliability is a Strategic Imperative

The book starts with the observation that the real world of information systems is chaotic, both because of the size and the complexity, but also because of the astounding number of changes that affect the systems daily. The first chapter is written by Ben Treynor Sloss, Google’s VP for 24/7 Operations, originator of the term SRE, who claims that reliability is the most fundamental feature of any product. Ben Treynor is the father of SRE: “SRE is what happens when you ask a software engineer to design an operations team. When I joined Google in 2003 and was tasked with running a “Production Team” of seven engineers, my entire life up to that point had been software engineering”. SRE is the story of a quest towards reliability through automation and simplification. He quotes C.A.R. Hoare in his Turing Award lecture: “The price of reliability is the pursuit of the utmost simplicity”.

SRE is the brainchild of a scientist and engineer who looks at failure and reliability with cold unemotional eyes and tries to maximize the outcome while reducing the effort. By construction it is a “DevOps” approach where the skills of software engineering are applied to improving operations. Obviously, this approach is draws from the scaling and size issues experienced at Google over the years: “Ensuring that the cost of maintenance scales sublinearly with the size of the service is key to making monitoring (and all sustaining operations work) maintainable”.

SREs (Site Reliability Engineers) work with the complete product team, from product owners and development teams to operations specialists and partner. Following engineering practices, business and users goals are translated into explicit and measurable goals that can be “engineered”. With no surprise to readers of this blog, key metrics that are used throughout the book are availability, latency and throughput :  “User-facing serving systems … generally care about availability, latency, and throughput”. As an engineer, one knows that a price must be paid for any complex requirement, including performance and reliability. Failures and mistakes are managed in a cold engineering approach, without exaggerating the requirements. In many places, the book warns us against looking for “over-safe, over-expensive” approaches, but to keep looking for a balance. 

2.2. Distributed Systems

As mentioned in the introduction, this is a book about distributed systems: “As SREs, we work with large-scale, complex, distributed systems.” Distributed systems engineering is a wonderfully exciting discipline, albeit a difficult one. It takes time, energy and dedication to become an expert in such matter. It also takes humility and curiosity.  “Systems are complex. It’s quite likely that there are multiple factors, each of which individually is not the cause, but which taken jointly are causes. Real systems are also often path-dependent, so that they must be in a specific state before a failure occurs.”

The emphasis on “distributed” here means that one must learn to step back and look at the whole system, not the platform or the component that is actively being built. One of my key action as a manager is to promote the “One System” culture, which is the understanding that we (as an organization) are building ONE large, complex and distributed system. This requires observing and sharing : “There are many ways to simplify and speed troubleshooting. Perhaps the most fundamental are: Building observability — with both white-box metrics and structured logs — into each component from the ground up. Designing systems with well-understood and observable interfaces between components”.

The emphasis about logging and configuration management is obvious throughout the book. There are far too many practical recommendations, such as the use of different verbosity levels, to be reproduced here in this summary. Some of them I learned many years ago at Bellcore, when working with true distributed system experts such as my friend Quoc-Bao Nguyen. Being able to debug, re-parameter, re-configure without shutting the system down has been a good practice for decades. On the other hand, distributed issues that used be very hard to solve, such as distributed locking, are now easier because of the wealth of scalable and robust open-source solutions. Still, one must understand the complex nature of distributed system and learn about tested protocols: “Whenever you see leader election, critical shared state, or distributed locking, we recommend using distributed consensus systems that have been formally proven and tested thoroughly. Informal approaches to solving this problem can lead to outages, and more insidiously, to subtle and hard-to-fix data consistency problems that may prolong outages in your system unnecessarily”.

The same remark can be made about the complexity of distributed large-scale storage. One must understand and accept the CAP theorem and learn to live with eventual consistency (or learn to live with right-time versus real-time) : “A growing number of distributed datastore technologies provide a different set of semantics known as BASE (Basically Available, Soft state, and Eventual consistency). Datastores that support BASE semantics have useful applications for certain kinds of data and can handle large volumes of data and transactions that would be much more costly, and perhaps altogether infeasible, with datastores that support ACID semantics”.

As mentioned earlier, the book is full of examples of self-adaptive mechanisms to make distributed systems more reliable. Techniques that are heavily used in network management such as exponential decay have obviously their place at the application level. Throttling is a good example of self-adaptation: “Client-side throttling addresses this problem. When a client detects that a significant portion of its recent requests have been rejected due to “out of quota” errors, it starts self-regulating and caps the amount of outgoing traffic it generates. Requests above the cap fail locally without even reaching the network”

2.3 Distributed Systems Reliability at Google

Google has developed a wealth of know-how on distributed systems reliability. The goal of this book is to share some of that knowledge because it is widely applicable in a context larger than Google or the web community. Reliability is not a set of simple recipes; it is a discipline of redundant protection through multiple layers:  “Given the many ways data can be lost (as described previously), there is no silver bullet that guards against the many combinations of failure modes. Instead, you need defense in depth. Defense in depth comprises multiple layers, with each successive layer of defense conferring protection from progressively less common data loss scenarios”. Some of the techniques, such as idempotent scripts, are now well understood in the DevOps community, but this book offers a comprehensive survey which should be useful to most of us who are not true geeks.

The model of the SRE team is to build a small team of highly trained ops specialists with enough software engineering expertise to understand and manage the systems: “Ultimately, SRE’s goal is to follow a similar course. An SRE team should be as compact as possible and operate at a high level of abstraction, relying upon lots of backup systems as failsafes and thoughtful APIs to communicate with the systems” … “In order to work at scale, teams must be self-sufficient. Release engineering has developed best practices and tools that allow our product development teams to control and run their own release processes”. This dual DevOps capability is critical to perform “ProdTests” (tests on the production environment). A SRE team should be autonomous in its decision process, but it works together with the rest of the organization (SRE teams supplements the engineering organization, it does not replace it).

How does one develop these highly trained specialists? By doing and mostly by learning from previous failures: “There is no better way to learn than to document what has broken in the past. History is about learning from everyone’s mistakes. Be thorough, be honest, but most of all, ask hard questions”.  The customer focus is critical and visible throughout the book. SRE must understand the user perspective and take extra care of customer-facing systems : “The frontend infrastructure consists of reverse proxy and load balancing systems running close to the edge of our network. These are the systems that, among other things, serve as one endpoint of the connections from end users (e.g., terminate TCP from the user’s browser). Given their critical role, we engineer these systems to deliver an extremely high level of reliability.”

One key responsibility of SREs is to assist release engineering, that is, how to test and release a new version of a software component or service. There is a lot of emphasis on gradual releases, which goes hand in hand with continuous release: “Almost all updates to Google’s services proceed gradually, according to a defined process, with appropriate verification steps interspersed. A new server might be installed on a few machines in one datacenter and observed for a defined period of time. If all looks well, the server is installed on all machines in one datacenter, observed again, and then installed on all machines globally.”

2.4 Automation and Monitoring

As the book points out, the simplest and most powerful principles behind reliability engineering are automation and monitoring. Automation is the natural solution to the efficiency/sublinear requirement made earlier, and the best way to remove a large part of human errors which are still the roots of most system failures. “There’s an additional benefit for systems where automation is used to resolve common faults in a system (a frequent situation for SRE-created automation). If automation runs regularly and successfully enough, the result is a reduced mean time to repair (MTTR) for those common faults.” The history of SRE, as told by Ben Treynor, comes from the automatization of operations by software engineers. Automation is a continuous task, not a once-for-all milestone. “Automation code, like unit test code, dies when the maintaining team isn’t obsessive about keeping the code in sync with the codebase it covers.”

Operations need to be automated and monitored. Monitoring is, as told earlier when we talked about the necessity of observation, the heart of reliable operations. “Whether it is at Google or elsewhere, monitoring is an absolutely essential component of doing the right thing in production. If you can’t monitor a service, you don’t know what’s happening, and if you’re blind to what’s happening, you can’t be reliable.” Monitoring and automation go hand in hand. Alerts produced by monitoring should trigger actions which are as automated as possible – back to the autonomic computing ambition – human intervention should be seen as the last resort option. Automation should come with heavy instrumentation to feed the monitoring and ensure proper execution (Automation is no guarantee against human errors, at first – only continuous improvement produced fail-proof scripts). “Running a service with a team that relies on manual intervention for both change management and event handling becomes expensive as the service and/or traffic to the service grows, because the size of the team necessarily scales with the load generated by the system.”

The regular practice of logging and monitoring provides massive opportunities of using Artificial Intelligence and Machine Learning for operations. Using algorithm ranging from classical time series data mining (e.g., with Splunk) to more advanced machine learning techniques to detect correlations and patterns, one may generate advanced alerts when the system requires tuning to avoid entering a hazard zone in the future. Automation and monitoring act on the two components of reliability: reducing MTBF (less error with automation, early detection and prevention with monitoring) and reducing MTTR (better diagnosis with monitoring and faster reparation through automation).

2.5. The Most Common Cause of Loss of Service Availability is Change

The SRE team at Google has found that roughly 70% of outages are due to changes in a live system: “Most outages are caused by some kind of change — a new configuration, a new feature launch, or a new type of user traffic — the two teams’ goals are fundamentally in tension”. This tension between the need for change (and for continuous delivery) and the need for reliable operations is wonderfully covered by the great book “Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations”, by Nicole Foresgreen, Jez Humble and Gene Kim. The only path forward is to automate change and treat it as seriously as possible at the same time. Companies that succeed to deliver high-availability reliable quality of service are those who perform frequent changes.

API management is a core dimension of modern system engineering, including from an operations perspective as is illustrated many times in this book. API modularity is critical for keeping systems simple. API change management is equally critical and helps to understand while reliability engineering and software engineering are mutually dependent: “While the modularity that APIs offer may seem straightforward, it is not so apparent that the notion of modularity also extends to how changes to APIs are introduced. Just a single change to an API can force developers to rebuild their entire system and run the risk of introducing new bugs.”

A key component of change management is Capacity planning.  This topic is covered in depth in this book, with a systemic vision (not only one platform as a time, but with a global perspective). “Good capacity planning can reduce the probability that a cascading failure will occur. Capacity planning should be coupled with performance testing to determine the load at which the service will fail.” System capacity planning is a complex modeling and engineering task that may leverage techniques from operations research (cf. the Auxon solver that formulates a giant mixed-integer or linear program based upon the optimization request received from the Configuration Language Engine: “At Google, many teams have moved to an approach we call Intent-based Capacity Planning. The basic premise of this approach is to programmatically encode the dependencies and parameters (intent) of a service’s needs, and use that encoding to autogenerate an allocation plan that details which resources go to which service, in which cluster.”

Capacity planning needs to encapsulate reliability engineering and propose designs that can operate under the simultaneous occurrence of planned and unplanned outage. This type of thinking makes “N+2” configurations a classical pattern to follow. Capacity planning should ensure that an acceptable service may still be delivered when the two largest instances are unavailable. Similarly, capacity planning needs to be mixed with load balancing policies. The book covers load balancing policies (such as Weighted Round Robin) in depth because they are a critical part of QoS : “Avoiding overload is a goal of load balancing policies. But no matter how efficient your load balancing policy, eventually some part of your system will become overloaded. Gracefully handling overload conditions is fundamental to running a reliable serving system.”  Queues management policies (FIFO, LIFO, CoDel, …) have similarly a key impact on QoS, which is precisely the topic of the OAI simulations that I ran 10 years ago (cf. “Self-adaptive and self-healing message passing strategies for process-oriented integration infrastructures”)

2.6 The Practice of the “Error Budget”

The concept of the “error budget” is a key contribution of Google SRE practice, where the operations organization finds a “QoS homeostasis” by balancing its operations requirement with the current state of its services. The aversion to change and the rigorous check-up of all necessary tests is not cast in stone but adapts to the current quality of service as seen by the user. The SRE team “balances reliability and the pace of innovation with error budgets (see “Motivation for Error Budgets”), which define the acceptable level of failure for a service, over some period. …As long as the service hasn’t spent its error budget for the month through the background rate of errors plus any downtime, the development team is free (within reason) to launch new features, updates, and so on … If the error budget is spent, the service freezes changes (except urgent security and bug fixes addressing any cause of the increased errors) until either the service has earned back room in the budget, or the month resets.”

The error budget must be managed from a customer-centric perspective. The “unavailability budget” is computed as the outage time that is seen by the end user: “Measuring error rates and latency at the Gmail client, rather than at the server, resulted in a substantial reduction in our assessment of Gmail availability, and prompted changes to both Gmail client and server code. The result was that Gmail went from about 99.0% available to over 99.9% available in a few years.”

There is an implicit but crucial point here: the outage/failure/error is no longer a “bad thing” that must be avoided at all costs, it is an “expected part of the process of innovation”, something that must be managed rationally.  Launches are necessary, but the SRE team has multiple ways of controlling the launch process : “Google defines a launch as any new code that introduces an externally visible change to an application. Depending on a launch’s characteristics — the combination of attributes, the timing, the number of steps involved, and the complexity — the launch process can vary greatly. According to this definition, Google sometimes performs up to 70 launches per week.”

Among the variables that can be acted, the SRE team must schedule the planned downtimes and the disaster recovery tests. Google spends a lot of energy on simulations of crashes and live drills. As we shall see later, the only way to develop and certify recovery capabilities is to test them regularly. Living with a failure is a part of what it means to run operations. As pointed out in the book, most people reaction when facing a failure is to start troubleshooting instantly and spend all their energy to find a root cause as quickly as possible. The proper set of action (as with a human accident) is to make first the system work as well as possible in a degraded mode. This is the type of behavior that you can only get with practice. Once again, failures are a natural occurrence of large scale complex systems. Minimizing impact of these failures is the role of reliability engineering.

2.7 Technical Debt and Recovery Capabilities

Complex systems that evolve constantly are poised to produce unnecessary weight and complexity that should be assessed and removed periodically. This is the principle of “taking constant care of one’s garden” (removing weeds, pruning, digging, …), that is, removing “technical debt” in the words of information systems : “Without constant engineering, operations load increases and teams will need more people just to keep pace with the workload.”  This is also why the authors warn us against “over-engineering” reliability :  “that would waste opportunities to add features to the system, clean up technical debt, or reduce its operational costs.”

When dealing with large-scale systems, data engineering becomes a critical skill. In the past 20 years I have lived through a number of very significant outages. Throughout theses (tough) experiences, I have noticed that the time to recover, to move or to install new data sets is always much longer than planned. In most cases, the recovery plan is executed (hence the disaster is averted) but the total recovery time is longer than what was planned initially, because unit operations are longer. Google report similar stories in this book: “Processes and practices applied to volumes of data measured in T (terabytes) don’t scale well to data measured in E (exabytes). Validating, copying, and performing round-trip tests on a few gigabytes of structured data is an interesting problem.”

A fair amount of pages deals with backup and recovery. Obviously, what matters is recovery, backups are just a tool. As a subcomponent of disaster recovery, data recovery must be tested regularly. Designing data recovery is a hard problem. More generally, designing reliable distributed systems is hard. As mentioned earlier, the book points out the necessity of deep system skills and expertise: “Today, we hear a brazen culture of “just show me the code.” A culture of “ask no questions” has grown up around open source, where community rather than expertise is championed. Google is a company that dared to think about the problems from first principles, and to employ top talent with a high proportion of PhDs.”

The proposed approach is to first create SRE teams with the proper mix of expertise: “To this end, Google always strives to staff its SRE teams with a mix of engineers with traditional software development experience and engineers with systems engineering experience.”  Then for these teams to succeed, it is critical that the senior management recognizes reliability and quality of service as a strategic imperative for the company. In all companies that I know, there are many “operations anonymous heroes” that keep the systems running as well as possible. What distinguished the digital champions such as Google is the recognition that these individuals and teams play a major role in the value creation of the company and deserve a just recognition: “Another way to get started on the path to improving reliability for your organization is to formally recognize that work, or to find these people and foster what they do — reward it”.

2.8  SRE : a Team with End-to-End Responsibility About Operations

SREs are hybrid production teams, which combine ops and software engineering capabilities (they aim at making Google systems run themselves) with a dual ambition of customer satisfaction and continuous innovation. “SRE is concerned with several aspects of a service, which are collectively referred to as production. These aspects include the following: System architecture and interservice dependencies; Instrumentation, metrics, and monitoring; Emergency response, Capacity planning, Change management; Performance: availability, latency, and efficiency”.

SRE teams follow many of the DevOps principles (software engineering and operations skills working as a team, involvement of IT function in each phase of a system’s design and development, heavy reliance on automation, …). SRE teams are associated to a service (or a service domain) : they own the responsibility of the run for this service: “In general, an SRE team is responsible for the availability, latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning of their service(s).” 

Somewhere in the book, the authors notice that most concepts, principles and technique reported here are not special but rather part of a well-accepted state-of-the-art practice. However they also notice that a surprising number of ops team do not take these practices seriously, such as capacity planning, resulting in unnecessary failures.

As stated earlier, SREs make a heavy use of blameless postmortems as a tool for training and evangelization. “Google’s Postmortem Philosophy: The primary goals of writing a postmortem are to ensure that the incident is documented, that all contributing root cause(s) are well understood, and, especially, that effective preventive actions are put in place to reduce the likelihood and/or impact of recurrence.” Blameless here means that the root cause analysis focus “on identifying the contributing causes of the incident without indicting any individual or team for bad or inappropriate behavior”.  Postmortems are not meant to be kept for the SRE team but to be shared as extensively as possible in the company.

Another key ritual of the SRE are the production meetings, where SRE team members orchestrate the necessary knowledge sharing between all concerned stakeholders (including product owners and software architects) : “In general, these meetings are service-oriented; they are not directly about the status updates of individuals. The goal is for everyone to leave the meeting with an idea of what’s going on — the same idea. The other major goal of production meetings is to improve our services by bringing the wisdom of production to bear on our services”. A key role in the SRE team in the Launch Coordination Engineer, which requires “strong communication and leadership skills” to bring everyone together. In a true DevOps spirit, when the operational load is too heavy, product development teams should contribute so that SRE keep a balance between incident management and continuous improvement. System architects should attend these production meetings regularly to collect up-to-date availability, latency and throughput which are absolutely necessary for reliable system engineering. Architecture schemas that are purely functional lead to disastrous failures in operations.

2.9 The Lean and TQM roots of Google SREs

Throughout the book, the influence of Total Quality Management (e.g., Deming) and Lean Management is quite visible. There is a constant effort to organize work so as to avoid overloads and work with a regular activity flow. Edward Deming famous quote “cherish your mistakes” finds many echoes here: “Google operates under a blame-free postmortem culture, with the goal of exposing faults and applying engineering to fix these faults, rather than avoiding or minimizing them.”

Running complex operations, especially disaster recovery and incident management, is more efficient with a playbook (lean “standard”) : “When humans are necessary, we have found that thinking through and recording the best practices ahead of time in a “playbook” produces roughly a 3x improvement in MTTR as compared to the strategy of “winging it.” Another lean influence is the importance of “system understanding” and making sure that indicators are picked scarcely and wisely. What matters here is the ability for the team members to evaluate and reason properly about a system’s health: “Ineffective troubleshooting sessions are plagued by problems at the Triage, Examine, and Diagnose steps, often because of a lack of deep system understanding.”

The system engineering culture has many aspects in common with Lean SixSigma, namely the heavy use of statistics including the use of distributions rather than mean values: “We generally prefer to work with percentiles rather than the mean (arithmetic average) of a set of values.  Doing so makes it possible to consider the long tail of data points, which often have significantly different (and more interesting) characteristics than the average”. Performance and reliability engineering require taking more than the average case into account.

Another lean trait is the search for constant elimination of dead weight (muda in the lean sense) which we mentioned earlier : “The term “software bloat” was coined to describe the tendency of software to become slower and bigger over time as a result of a constant stream of additional features.” “Software bloating” or “feature creep” is a common plague of software systems that is a clear instance of the Tragedy of the Commons. Adding more features always comes from a local view, where the benefit of the new feature is amplified and the systemic cost of adding to the whole system is reduced if not ignored. To encourage the team to adopt a “one system” mindset forces to re-evaluate with less emphasis on the new feature and more on the global resource consumption. Many resources such as available time (in an ops planning), bandwidth, or memory capacity lead to a “tragedy of the commons” situation where the sum of “small local incremental usages” translates into a global over-consumption.

3. Conclusion

Although this book review is actually quite shallow compared to the depth of the book, it is already substantial compared to what people expects when discussing digital transformation. My intent here is to underline the importance of execution with any digital transformation. As stated in the introduction and wonderfully explained in “The Web Giants”, quality of service is a crucial part of the customer experience. A company cannot be a digital leader without mastering the skills and the practices that are described in this book. The illustration on the right is taken from the Twitter launch of the NATF report on Artificial Intelligence and Machine Learning. What it says is that the Google SRE book should be part of the skill set of any companies that aim to develop a leading service based on data and AI. Engineering matters.

  

Site Reliability Engineering is foremost an ambition about people and culture. Here is a very short summary about an ideal “End to End” operations organization:

• It is made of small autonomous teams that orchestrates, for instance through production meetings as well as postmortems, all stakeholders involved with the delivery of one (end-to-end) service.
• The team owns the responsibility of this service: availability (reliability), latency & throughput , capacity planning, change management, monitoring,  incident management and recovery.
•  This should be a customer-centric organization with is both measure-driven (analytics) and adaptive (flexible). Its operations guideline should fluctuate using an “error budget” to deliver stable quality of service and “change throughput” at the same time.
• This team is necessarily “progress-centric” and operates under continuous improvement using postmortems and root causes analysis. The team has the diverse skill set to own and understand the complexity of the systems that it operates.
• The organization strives towards as much automation as possible. This requires both to enroll software engineering skill and to let them operate on production systems.

This book tells mostly about reliability engineering but, as shown in the previous section, it is also a textbook about distributed systems engineering. Here is a short set of principles for building reliable distributed systems that I have extracted from this reading and which I will comment in a future post (the reference to “Biology of Distributed Information Systems” – the title chosen for this blog 10 yeas ago – is quite deep) :

1. Leverage APIs to develop a multi-modal cell structure with variable rate of change, when change flows from the outside towards the inside of the whole system (like a cell)
2. Organize complexity in layers where advanced systems are backed up by simpler “life-support” systems, the lower the complexity the higher the availability (biomimicry)
3. Develop an event-driven flow architecture to design reactive systems that are scalable, reliable and open
4. Use self-monitoring and “digital twin” to provide self-adaptation, self-optimization and self-healing
5. Journey towards abstraction to move to “server less” systems (necessary to combine SaaS, cloud and on-premice operations)
6. Leverage lean thinking to deliver robustness and flexibility through more available capacity
7. Don’t fight CAP (theorem) and develop a reliable, high-availability, eventually consistent data architecture (think of data consistency as a movie rather than a snapshot).