Systemic Simulation of Smart Grids : First Results

The month of August provides me with the opportunity to get back to my computer and to resume my programming projects. I have been able to complete the first step of S3G (Systemic Simulation of Smart Grids) which I started last year. I had the opportunity to attend the EU-US Frontiers of Engineering Symposium, where I have presented my project and received encouraging and interesting feedback.

The objective of S3G is to simulate the production and consumption of electricity throughout a long period of time (15 years). The S3G model, which is illustrated by the following figure, may be summarized with five parts:

• Energy demand: for each city, energy demand is generated from an hour-by-hour and day-by-day template, adding some random variation (the extent of which is a model parameter) together with a city-specific variation. This number is then reduced by the amount of “negaWatts”, computed from the total amount invested by the city. The model uses a ratio obtained from a concave-increasing function of the investment.
• Dynamic Pricing: both suppliers and operators use a simple affine pricing model, with a constant price when the demand is less than a “base power”, and a linear formula when the demand is higher.
• Production: suppliers use nuclear power according to planned schedule and adjust to resulting demand with fossil plants. Operators always use their green power (store it in the “buffer” or resell it when there is too much of it). They adjust to the city demand with their own fossil plant and wholesale electricity from suppliers, at the lowest marginal cost.
•  Consumption: The actual electricity consumption for each city is the demand, minus “shaving”, which is obtained by applying an S-curve to the sale price.
• Market-share: for each city, the market balance between the national supplier and the local operator is determined yearly using another S-curve.

This S3G model is both simple and complex. It is simple because it is based on a handful of equations, resulting in a simulation code that is 500 lines long. On the other hand, it is a complex model for two reasons. On the one hand, there are multiple feedback and interaction loops that make it difficult to analyze how the system will react to perturbations. On the other hand, there are many unknown parameters in this model (such as market sensitivity, demand-response behavior, negaWatt capabilities, etc.). 

The following figure shows, on the right part, a rough summary of the simulation loop that is run for each time period (3 hours, hence 8 times per day). On the left part, it shows the three main GTES generic procedures.

I have already spent some time on computational experiments that will be presented at CSDM in December. A GTES simulation run returns the average and standard deviation of a few key business parameters, as well as some indication of the Nash convergence. Giving averages and a few deviations is a poor restitution of the rich data gathered during the computational experiment, but the goal is simply to “get a feeling for what is happening”, as opposed to producing a forecast. The current set of experiments is designed to understand the main issues that were exposed in the previous post. I have defined a fictional country somehow similar to France decomposed into 10 regions/cities. I have run 8 experiments that may be defined as follows:

1. The “default” is a reference point, from which “what-if” sensitivity analysis is made. The economic parameters are set in such a way that alternate operators start with a 20% market-share and should be able to increase it if they demonstrate a better management of variability.
2. The second experience raises the variability of energy consumption (globally), while the third experience raises the local variability (each city is more different from each other).
3. The fourth experiment doubles the fossil energy price (gas and coal). In the default scenario, it is randomly drawn between 20€ and 40€/MWh.
4. The fifth experiment imposes a 5% reduction of the nuclear assets for the supplier during the first 5 years.
5. The sixth experiment sets a carbon tax at 100€/t, the proceeds of which is used by the “regulator” to subsidize green energy investment.
6. The seventh experiment explores the impact of overall demand variation in the next 15 years. The model assumes a constant growth/decline of electricity demand which is expressed as a percentage. This experiment plays with different values to see if demand impacts the profitability of smart grids.
7. The last experiment is a small variation of the first one, where wholesale prices are more rigidly constrained. During this set of experiment, I considered that the constraints that regulate wholesale prices for the supplier are fixed. An interesting next step will be to make them a strategic parameter for the regulator (which is a better representation of reality and makes for a more interesting game).

I still need to do some additional tuning before presenting my results at CSDM. I will then make my slides available on this blog. For once, I need to run longer simulations (with larger samples) to ensure that the Monte-Carlo sampling is stable. I also want to introduce better local optimization meta-heuristics, such as Tabu search. One of the key issues for smart grid is the pricing model, both for the operator and the supplier. The more complex the pricing model, the better the actor may play with the demand-response concept. However, a more complex princing model requires better optimization techniques to ensure that the automatic computation from GTES (where each actor “learns” its “best response”) is relevant.

This being said, here are a few findings that may be drawn from the hundreds of runs made with the S3G model, and that are worth sharing:

•  There is a systemic benefit of distribution and autonomy to cope with variation. This is shown in our second and third line. It is a “subtle” variation (small effects), which means that the economy of the local operator is dominated by its capacity to operate at much lower customer management costs than the supplier.
• CO2 tax increases play a very small role, and one that is difficult to anticipate since it both favors the local operator (support green subsidies) and the supplier (raises the difference between fossil and nuclear).
• “De-nuclearization” is a favorable scenario for smart grid operators, as are most regulations that are adverse to the supplier. The obvious limitation is the resulting price increase that reduces the total economy output (and the country’s competitiveness).
• The “community advantage” (that is, the ability for a local operator to better manage the demand-response loop because it is “closer” to its end customer) is marginal, and it is quite unclear if the payback from demand-response management is enough to sustain the operator’s business model.
• Investing in local storage is never an interesting option (at current prices). We needed to slash the price by over an order of magnitude to see a viable payback in less than 10 years.
• There is a clear competition between local operators and suppliers. The learning component of GTES makes for “agile” players who react closely to each other signals. The pricing structure plays an important role (we have only explored a simple variable pricing scheme). A logical consequence is the importance of regulation.
• The results are sensitive to the strategies of the player. A next step for S3G is to build a “strategy matrix”, which is a tabular “what-if” sensitivity analysis where we see what happens if the goals of the players are changed.

Lean Software Factory

This post is a follow-up to an invited talk that I delivered at CESAMES. That presentation, which is available on the box.net widget on the left, talked about complexity, architecture and lean software development. It started from the question – its title – “How to design information systems?” and ended with an introduction to lean software development. Today I’ll go further and explore the relationship between the (software) product and the (software) factory. The key idea is derived from the Toyota Way: you need a great factory to manufacture a great product. Not surprisingly this will lead to the concept of “Lean Software Factory”. This post is organized as follows. The first section talks about what makes a good software product. I will focus mostly here on software development, but most of what will follow is equally relevant to information systems. The second section deals with the concept of “software factory”. Looking at the “software factory” means taking into account the software manufacturing process into account, and not only the final product. It is also a metaphor that carries the need for automation, engineering methods and industrial discipline. The last section is about “lean software factories”, which add agile development practices and wisdom to the strengths of software factories, together with a continuous improvement and culture geared towards excellence that is inherited from lean management.

1. What makes a good SW product?

There are many expected qualities and assessing a piece of software is a multi-dimensional tasks. Here I will only focus on three of them, because they are both very popular (all books and papers related to software engineering discuss those three) and very important to the value that is produced through a software product:

• Modularity – “the degree to which a system’s components may be separated and recombined” (Wikipedia) … or the capacity of the architecture to maximize in its decomposition the independence of subcomponents. Modular systems are easier to maintain and to evolve. In the world of software, modularity translates into the minimization of “impacts” when something must be changed.
• Versability / evolvability – “the property of having many abilities”, that is software that can serve many purposes, together with “the capacity of adaptative evolution” - software may evolve rapidly to support new functions. This combination is often coined as “agility” in the software world.
•  Openness : for a piece of software, being open may mean many things. It may means that it exposes API (application programming interfaces) so that other software components may interact with it. It may also mean that the code is open to all for scrutiny and collaborative improvement, as in “open source”. It may mean that the piece of software is built as a platform, which is designed to be enriched by subcomponents written and proposed by other developers. Writing software as a platform is at the heart of successful development strategies, such as Google’s or Facebook’s.

These three qualities form a deeply interrelated network (each influences the other). They are also notoriously difficult to achieve, especially in a rapidly changing environment. These qualities are resulting properties of both:

• Software architecture, that is, design properties which explain how the software product is organized. This part is well understood, albeit difficult: this the art of software architecture. If I follow François Jullien’s distinction between Greek and Chinese strategy, architecture is clearly the Greek, top-down, left brain, goal-oriented part of the strategy for producing modular, agile and open software.
• The way the product is made (the software development culture). This translates into “genes” which are embedded into the software product. According to the same distinction, software development culture is the Chinese, bottom-up, right brain, emergent strategy for producing modular, agile and open software. This is less understood and even less formalized, it is a form of wisdom transmitted in an oral tradition. It is also controversial since many still believe that once the specification is completed, it doesn’t matter who writes the code …

I am certainly not one of them J. On the contrary, I have found over the years that great software products most often come from companies with great software cultures. It is not a new thought either, even for me. In my first book “Urbanisation, SOA & BPM”, I noticed that modularity and agility are properties that describe the whole lifecycle and not only the development part. Agility in code writing does not do much for you if you cannot run your tests in an agile mode (Chapter 6). Similarly, modularity must translate into modular testing (running fewer tests), modular deployment (deploying fewer components) and modular maintenance (fixing problems faster because the area of investigation is limited to a smaller area).

The remainder of this post is devoted to « the way software products are built », for lack of time but mostly because my previous writings and books have been about software and information systems architecture. You may read the CESAMES slides to get an overview on achieving modularity, agility and openness through architecture. For those of you who don’t read French, here is a “101” summary:

1.  Model-driven Architecture : architecture should be based on models and part of the development should be derived with semi-automated methods. Models are another form of “genes” for a software product: careful design at the model stage goes a long way to embed “future-proofness” (using scenario/whatif methods)
2. Data models are critical: from semantics, conceptual analys, ontology to object lifecycle management. Data architecture is the most useful tool to prevent data distribution difficulties which plague most large-scale, heterogeneous information systems.
3. Layered Architecture : defines abstraction levels and reduces the management complexity, a time-proven “good practice” of software architecture.
4. Service-Oriented Architecture: defines the service structure so that reuse and sharing is made easier. SOA is also the proper tool to align the different stakeholders so that software fits the enterprise’s strategy. A service architecture is a reification of functional architecture (a service may be seen as the combination of a function – what it does- an interface – how to invoke it – and a contract – so-called “non-functional” properties –), which embodies abstraction and encapsulation.
5. Process Architecture : defines the “composition grammar” through a recursive/fractal analysis of interactions. Process architecture is to “Business Process Engineering” what SOA is to “writing down a list of exposed services”: there is much more than listing and formalizing, it is about identifying patterns and sub-patterns, and extracting those patterns which will prove stable through time. It is more an art than a science.
6. Event-Oriented Architecture: identifies “events” as pivots for interaction (which is a subpart of process architecture and a foundation for SOA). Publish/Subscribe is another time-proven pattern for building modular designs, which is built on top of a shared/structured event catalog.

2. Software Factories

Focusing on the way software products are build leads logically to the concept of “Software factory”. The following illustration shows three way of looking at a software system (or an information system):

• The first step (left-most) is to look at the technical system (leading to what we call IT : Information Technology). This view is concerned with practical components such as databases, software code and computing resources. Processes are defined in the “computer science sense”.
• Then we step back and add the human into the picture, mostly the user but also other stakeholders (management, administration, strategy,…), leading to Information Systems. IT/IS distinction is well established in information systems literature. This view adds – rightly so – usage as a key dimension (following the very profound statement that a software application that is not used is of little value), and shared enterprise artifacts such as models and business processes.
• The topic of this post is to step back another time and look at the “production process” (the “factory”) that continuously builds and update the information system. On the right-most part of the illustration, we see both the information systems and the organization (usually the ISD – Information System Division in a large company) that builds and runs it. I have separated the part that builds (dev: software development) and the part that runs (op: operation) as a forward reference to devops.

Because a software factory handles ideas and digital information, it could be argued that it is an autopoietic system (this is not a car factory, there is endless supply of ideas and disk space to write new code). Autopoiesis represents the capacity of a living system to continuously develop itself and to transform the process that creates the system and which is a part of the system itself. Autopoiesis generates emergence (of properties) and (what may be seen as) auto-organization. This thread of thought is quite interesting, because it helps understanding the emergence of our three desired properties, however I will leave it to another day, to avoid adding confusion to conceptual complexity.

To return to more practical grounds, the goal of a software factory is to deliver “good software products”, rapidly and efficiently. Because we now have the advantage of having read “The Lean Startup” (if you have not, I would urge you to do so, it’s the best way to understand what is truly expected of a great software factory), we know that there is another dimension to SW factory performance: the ability to listen carefully to the product’s users and react rapidly to their expectation. This is why Eric Ries advocates for MVP (Minimum Viable Products): the SF (software factory) must be able to crank up simple products very rapidly, which are exposed to customers, and modified according to their reaction (this loop is shown on the illustration).

A lot of things are known about how to build a “good” SF. Sharing a reference with the previous post, I picked the following recommendations in Curt Hibb’s book “The Art of Lean Software Development”:

•  “Practice 0” : Source Code Management and Scripted Built. This is where the word “factory” becomes self-evident: one must use tools (such as source code management) and industrial discipline to manufacture software. There is a “Maslow pyramid” of skills necessary to reach continuous deployment, scripted built is the first step.
•  “Practice 1” : Automated Testing, because testing is crucial for quality, as in any industrial process (automated testing means more testing), but also because we want to run tests as early as possible, following the principles of extreme programming.
•  “Practice 2” : Continuous Integration, which means that the product is built continuously in an iterative manner, instead of the traditional waterfall model. Continuous integration to perform integration testing as early as possible; it is also a way to “see the product” and ensure that the factory is united on a common goal.
•  “Practice 5” : Customer Participation, which is what makes the continuous improvement loop that is pictured in the previous figure possible. Customer participation is at the heart of agile methods.

There is much more in Hibb’s book, as we shall later see, but this is a set of sound, time-proven practices. Another great source of inspiration for designing a software factory is the previously-mentioned DevOps “method”. Many excellent software companies, such as Facebook, are making references to DevOps, which is not a coincidence. Following Octo’s lead, I would summarize DevOps’ contributions with two principles: “Infrastructure as code” and “Continuous deployment”. Infrastructure as code is the ability to address all computing resources (either cloud computing or a regular data center) through API and software. It follows the principles of “Practice 0” (tools & automation) and give true flexibility and agility to the software factory. It also changes the “ops culture” and prepares for the next step. Continuous delivery is the ultimate step of continuous development/testing/integration/deployment. It helps shortening the continuous improvement loop from Eric Ries and is necessary to move towards a “lean software factory” as we shall see later. Taking a look at Facebook development and operations’ culture is worth the effort. Facebook has developed large-scale continuous deployment into an art. I recommend watching Chuck Rossi’s tech talk on “Pushing millions of lines of code five days a week”.

3. Lean Software Factory

The idea of a “lean software factory” comes from mixing “lean software development” (cf. previous post) with “software factory”. That is, adding to what we just saw a few other principles which are taken from Agile methods and from the Toyota Production System (cf. Hibb’s book). This does not mean that all projects from a lean software factory should follow a SCRUM method (take as an illustrative example of agile method). Although this is debatable, I believe that some projects are still better run with a classical V-cycle model (I am not not the only one). There is no magical formula to define the “SCRUM/ agile boundary”, but V-cycle still makes a lot of sense for “back-office projects” with stable and complex specifications (such as a billing system). On the other side, agile methods are better suited to front-office projects (when the human in the loop plays a bigger role), where detailed requirements are difficult to produce because the need is ill-defined or changing rapidly.

To keep this post short, I will summarize the concept of “Lean Software Factory” with four “principles”. Each of them would require a full post, which I may do later on – Each “principle” is in fact a set of practices since “lean is something you do, not something you talk about”.  These principles add to the previously developed concept of “Software Factory”. The first principle captures the essence of “agile methods” and, therefore, only applies to agile projects, whereas the rest applies to both forms of software development (that is, including V-cycle development).

1. The first principle is to apply an agile method such as SCRUM (Backlog + small batch + test as early as possible + continuous build) whenever the end-user has an important stake with the software product. Agile methods fit nicely with the principles of a SF as described in the previous section. They emphasize customer participation and continuous integration. The lean “touch” adds the concept of small batches (called sprints in SCRUM) and the principle of testing as early as possible, with the goal of producing the right code on the first time. Building a product one small step at a time with regular opportunity to adapt to environment feedback is a key idea which is well developed in “The Lean Startup”. Another key inspiration from lean is to reduce “lead time”, which translates into short sprints but also into time-constrained steps, including design.  This last technique also works within V-cycle: using time-constrained workshops to jointly achieve the requirements and design phases. Those familiar with Google’s software development values will recognize the “lean influence” (develop quickly, only what is necessary, remove waste).
2. The second principle is to emphasize autonomous teamwork. Teamwork is critical for all kinds of projects and for most activities. A key teamwork moment from the lean culture is Kaizen, which is used to solve problems and search for continuous improvements (two sides of the same coin). Following “the Toyota Way”, all ideas are welcome and everyone’s contribution is useful in a LSF.  There is a lot to be said about kaizen (which I won’t do today) : it goes further than the continuous improvement culture that one may find everywhere (such as the feedback loop in the SCRUM method). “Lean culture” brings a systemic approach to problem solving, including the search for root causes with “5 Whys” and the construction/follow-up action plans, such as the famous “A3” from Toyota. Autonomy is a systemic condition for kaizen in particular, but more generally empowerment is necessary to achieve performance.  One of the best book to understand kaizen and autonomous teams is “The Lean Manager” by F. & M. Ballé.
3. The third principle is to leverage visual management, which often takes the form of a project/product room. Visual Management has multiple benefits, the first of them is synchronicity, that is working with the same time. In the world of an industrial factory, the lean concept that embodies synchronicity is “takt time”. Although takt time also plays a role in a software factory, synchronicity is mostly about working at the same pace, without waiting. Dynamic wall-planning is the best tool to ensure synchronicity, together with the frequent stand-up meetings in front of the said planning, where the whole team keeps track of progress and difficulties. Visual management is also a tool for systemic training. Toyota encourages its workers to use as many schematics and illustrations as needed to share the detailed functioning of technical processes or pieces of equipment. This translates easily into the world of software. Using the walls to share architectural insights with the team is an efficient way to contribute to continuous learning as well as increase efficiency and prevent errors. Visual management is also used to visually control “work in process” (WIP). Reducing WIP is another key tenet from lean management that applies very nicely to software development. The practice of visualizing WIP with small cards (post-it) that are pasted on the wall according to their status in the development process is called Kanban in the lean IT world, in reference to the Totota pull system. Kanban / JIT (just-in-time) in the world of software is not as strictly enforced as in the world of industrial manufacturing. Still, the “pull” chain of control, where every step in a process is “serving” the following step (that is, providing what is needed at the right time, according to the availability of the receiving step), is a great tool to increase efficiency and avoid “overload” of tasks that are waiting to get completed. The “kanban” practice has two benefits: making each member of the team aware of the availability of others who will use their work later on, and reducing overloads and piles of unfinished work through an implicit capacity limit to what is accepted into the development process. The emphasis is on “no waiting” to both improve agility and quality (asynchrony is the best way to lose “context” when the control is handled from one player to another and to produce misunderstandings – this is explained in my last book, and is especially true for software development, which is a complex task, and where one quickly looses focus once your work  is completed.
4. The last principle tells that a Lean Software Factory is a place with “love for the product”. This supposes two things: to know, understand and “love” the customer, on one hand, and to love software code on the other hand. The love of code is another multi-dimensional mix of culture and practice.  The practice part draws on extreme programming (code reviews, peer writing) and software factory (cf. previous section) with a “lean touch”. Mary Poppendieck explains how the 5S of lean may be applied to the practice of source code management. Lean practice also translates in writing less code (see previously mentioned Curt Hibb’s book). The culture part is also a mix of aesthetic (writing elegant, minimal code that one is pleased to show to the other members of the team) and discipline (following coding styles and guidelines, appropriate density of comments, etc.). This is very close to the values of open source software, one could say that a lean software factory should write its code as if all of it was going to be open source. We are getting back to the opening argument in the introduction: designing open software is a state of mind, hence it should be supported by the culture. There are two reasons why “the love of code” is emphasized as a key value. The first one is that application of what is learned about Toyota: to build a great company, the employees need to love the product that they are making. For Toyota it is about cars, in a software factory, it is about code. Lean management is all about professional pride, technical skills and craft. A lean software factory is not about undistinguishable “man-days” that may be outsourced to a lower-cost country. The second reason is that code is a “live object”, with a renewal rate which is getting higher as the world becomes more complex (Petra Cross from Google says that 50% of their code changes every month). Code that grows iteratively needs to be re-factored regularly (iteration yields accumulation); like a garden, it needs constant care and attention.

Lean IT, Devops and Cloud Programming

I have become more and more interested with lean IT over the years. It started with the book "The Art of Lean Software Development" by Curt Hibbs. 

I enjoyed this book because its introduction of lean is 100% compatible of what I learned from Jeffrey Liker and other great authors about TPS. This simple book helped me draw the lines between good software development methods such as extreme programming or agile programming, and lean management. For those of you who are not familiar with lean or extreme programming, here is a very crude summary of some of the most salient points of lean software development :

• No delays: avoid as much as possible work that is sitting between two steps of the development process (what is called WIP – work in process – in the lean jargon). This is true at the process level (the goal is to design a streamlines/ “single piece flow” development organization) and the developer level. A practical goal is to avoid switching tasks as much as possible: focus on one thing and do it right!
• Quality in (right first time) :  this lean principle translates into testing as early as possible (a tenet from agile programming) but also to use all techniques that improve the quality of the code, even at the expense of source code productivity since we all know that it is cheaper not to produce a bug than to remove it later. Here comes, for instance, the practice of pair programming and code reviews, but also “good practices” such as programming guidelines and standards
• Fast delivery: the lean principle is to reduce the “lead time” of the software development process, which requires working on all stages. Removing in-between delays (cf. earlier) is necessary but not enough. Continuous integration is a core technique to achieve this goal, such as fast deployment techniques.
• Short deliveries: It is more efficient to produce small pieces of software at a high rate than bigger pieces at a lower rate. This is another key principle from lean (“small batches”), which is doubly true for software development: not only are l smaller batches easier to build (a well-known law of software engineering), but the continuous evolution of customers needs and environment makes it easier to adapt with a small batch approach.
• Less code: this is the application of the KISS principle !  Lean software development tries to stay away from complexity (see later in this post).  Unnecessary code may pop up from many sources, lean applies a technique called VSM (Value-Stream Mapping) and a posture of “muda removal”.  Muda (waste) removal means to go through the process with the “eyes of the customer” and remove all that does not produce value from her perspective. VSM is a tool that tracks the value creation and assigns it to each step of the process. Lean software development aims at producing the right product, without unnecessary features. It is also an architecture principle (stay away from complexity) designed at simpler and faster maintenance over the years.
• Customer participation: the best way to produce only what is necessary from the user’s perspective is to ask her frequently ! This is why end-user/customer participation is a tenet of agile programming. When the customer is not available, the principle mixes with “small batches” to become : deliver fast, fail faster to succeed sooner.
• Regular work effort : the leveling of the effort is a key principle from extreme programming, the equivalent of “heijunka” from lean programming. A few years ago, when I was still a CIO, I started thinking about “extreme IT” (applying extreme programming at the information system level), and the sustainability of the effort is a crucial point. Regularity has a counterpart, which is discipline. Using methods and tools (such as configuration management, source code versioning, automated testing) is crucial. One should take a look at the wonderful talk from Mark Striebeck on “Creating a testing culture” at Google.
• Minimize handovers: complex tasks, such as writing software, are better accomplished when the responsibility chain is not truncated into too many segments. Working as a team is the only alternative to deliver complex projects (this is the topic of my third book). This is another insight from the Agile Manifesto, this is why today’s best practice it to assemble multi-disciplinary teams including software developers, marketers and designers.

There are a number of interesting books on this topic. The most classical ones are those from Mary Poppendieck, such as “Implementing Lean Software Development: from Concept to Cash”.  I had the pleasure of presenting my last book at the Lean IT Summit a month ago and to meet with Mary Poppendieck (a number of great talks, including hers, Michael Ballé’s and Georges Striebeck’s, are now available online).

There are also books which are not about software development per se but very closely related. I am especially fond of two books which I have reviewed in my French blog:

• The Lean Startup, from Eric Riess,
• The principles of Product Development Flow, from Donald G. Reinertsen

Thanks to Guillaume Fortaine, I have come to learn about Devops. Quoting from Wikipedia, “DevOps" is an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals”.  The more I read about Devops (Devops.com is a treasure trove of great articles), the more I find that Devops is the missing link between Agile/Extreme Programming and lean management (Toyota-style, hence Lean-Startup-style).  Although Devops claims to “help finishing what Agile Development started”, there are different flavors in what one may read on the web: bridging the development /admin & operations gap, making lean IT happen, making collaboration happen (and also),  bringing agility to operations,  which in turns makes it easier to leverage the benefits of new computing resources such as cloud programming. My first obvious interest with Devops has been the practical deployment of lean IT, as a followup of what I just explained earlier. For instance, Devops promote a number of interesting tools related to deployment automation, such  as GLU. It turns out that the “Cloud and Devops intersection” is also quite promising. Indeed, leveraging the strengths of cloud programming requires a shift in development/architecture culture that requires a “Devops-like” approach.

It happens that I gave a talk last week at the Model-Driven Day about « Complexity, Modularity and Abstraction” (the talk is available in the “My Shared Document” box on the left). The talk is about, among other things:

• Complexity (why it is important, how to measure it and to tame complexity – since avoiding complexity altogether is not an option at the IS level)
•  Sustainability: how to transform enterprise architecture into a regular practice. This is related to the concept of “extreme IT” : avoid the “heroic struggle” to move towards the “continuous transformation of the information system”. This is a major reason why I have been advocating for SOA as a company-wide enterprise architecture practice for many years.
• “Architecture-oriented services” : I made this pun to emphasize the difficulty to produce the “right” services through SOA. “Architecture-oriented” means services that have the right level of abstraction, that are modular and “easy-to-compose”. To my knowledge, there is no easy recipe for this, but the wisdom and folklore of 40 years of software architecture design apply.
•  Cloud computing: I have added a small addition about “cloud-ready architecture” in the 4th edition of my first book. I strongly believe that Information Systems will change in a spectacular manner when we learn how to exploit massively parallel architecture (using tools/approaches such as Hadoop/MapReduce). Using Cloud Computing to provide with a few Saas-based front-office services is nice and relevant, but the big change comes when cloud-computing is applied to back-office services (provisioning, billing, data mining). This requires an architecture change, but mostly requires a culture change.

The thought that prompted me to write this post is as follows. Devops is the missing link between the themes of my MDD talk: managing complexity, delivering agility/modularity and moving to the new century of massively parallel computing (including, but not restricted to, cloud computing). I speak of "Cloud Programming" in the title because I agree with a comment made by Georges Reese: the Cloud is a computing resources defined by its API, i.e., a resource that is managed through programming. The exact quote is:

Cloud is, for the purposes of this discussion, is sort of the pinnacle of SOA in that it makes everything controllable through an API. If it has no API it is not Cloud. If you buy into this then you agree that everything that is Cloud is ultimately programmable.

 Not only the cloud is more agile (up/down scalability), it may be controled by the piece of software which is using it. From a systemic perspective, it's a "whole new game" :)

Systemic Simulation of Smart Grids

This blog has been extremely quiet for two years, mostly because I was busy with other topics, including Lean and Enterprise 2.0. In my previous post, I mentioned that I was looking at “Smart Distributed Things” through two instances, Smart Grids and Smart Home Networks. I will not talk about the second in this blog, since it is work-related (we’ve come up with the acronym BAHN : Bouygues Autonomous Home Network).

Smart Grids make an interesting instance of the “smart distributed network” because of the extraordinary amount of interest/excitement/hype that exists around this topic. There is a range of conflicting opinions, ranging from “this is a straightforward and marginal improvement of existing networks which are already quite smart” to “smart grids are the backbone of a new sustainable society, based on communities, subsidiarity (organic, multi-scale, resilient organization) and self-aware optimal management of resources”.

I am not an expert with any of these topics: electricity, power grids, energy, storage … but like any citizen, I would like to build my own opinion about the future of our country and our planet, as far as energy and global warming are concerned. Hence the idea of a small systemic simulation has emerged during my vacation month (last month). I tried to assemble a crude model of all the different aspects of the “smart grid ecosystem” as we may understand it, without too much detail, just the broad principles. Each aspect is actually quite simple to explain, if taken in isolation (e.g., anyone may understand why it is smart to couple a solar panel with local storage). It is the combination of all viewpoints, together with the huge amount of uncertainty (about the economy, the speed at which technology will get cheaper, the speed at which behaviors will change, etc.), which makes this a hard topic.

This is where I got this insight: I actually have a method for complex embedded models where half is unknown and the other half is unclear: GTES (Game-Theoretical Evolutionary Simulation). GTES is the perfect platform to assemble conflicting views of what a smartgrid should be, conflicting views about how the actors should behave, and try to generate some sense. In a word, I am planning to build a “serious game” to have a closer look at smart grids.

Let me first clarify what I said about the range of opinions and introduce three views of the smart grid, which I could label: the “Utility view”, the “Google view” and the “Japanese view”:

1. The “Utility view” defines a smart grid as adapting the power network to local sources (as opposed to a one-way distribution network that goes from few large GW production units towards millions of consumers), adapting to intermittent production sources (though storage and favoring flexible production units that can adapt to the power surges of intermittent sources like solar or wind) and using price incentives to “shave” demand peaks.  This is a “no-brainer” program (“what to do” is clear), where the major issue is price: most techniques show a cost/benefit ratio that is worse than current practice. To believe in this approach, you must believe that new technology prices will go down (e.g., solar, storage),   or that electricity prices will go “through the roof” in the next 20 years, or that global warming fears will drive a significant price for CO2.
2. The “Google view” defines a smart grid as a change from a tree structure to a network structure (centralized to de-centralized), the use of market forces to create a dynamic and more efficient equilibrium between supply and demand, and the use of IT to provide information to all actors, including end consumers. The importance of signals (pricing and power grid control) is so important in this view that it is often said that telecom, IT and energy network will merge (something which I don’t believe for a second – but it shows the spirit of the importance of “smart” in “smart network”). Calling this the “Google view” is a friendly reference to “What Would Google Do” J  The core of this approach is the principle that a significant amount of efficiency would be obtained with a “hyper fluid market”, made possible through IT technology.
3. The “Japanese view” is human-centered instead of being techno-centered. The goal is to change human behavior to adapt to new challenges (lack of resources, global warming, …). Smart grids are the backbone of a multi-scale architecture (smart home, neighborhood, city, region, country) where each level has its own resources and autonomy, resulting in a system that is more resilient and with more engagement as a consequence of more responsibilities. Smart grids support the necessary behavioral transformation through communities and constant feedback. I call this a “Japanese vision” because I have heard it beautifully explained in Tokyo, but the systemic approach is common to Asia as a whole. Because active communities are “engaging”, Smart Grids help get rid of waste (muda in the lean sense) such as useless transportation, un-necessary usage, etc.

These views do not necessarily conflict; it is possible to envision the union of all these ideas. However, as soon as one tries to imagine a convincing deployment scenario, there are a number of questions that pop to mind. Here are a few examples:

• What part does local storage play? Can there be a smart grid without distributed storage? What price hypotheses are necessary to make this realistic, considering that current energy storage price are too high to justify large-scale deployment? Even if solar or wind becomes free, having to store the energy with today’s techniques make this approach uncompetitive (with today’s parameters, obviously)
• What CO2 price would change significantly the cost/benefits analysis? Actually, this extends more generally to the pair energy prices + CO2 price, but CO2 price is especially significant. Because of the abundance of coal, and because of the increasing availability of new forms of gaz, CO2 price is, from a naïve point of view, the key factor that could change the economic analysis of introducing alternative intermittent sources of energy.
• What is the systemic benefit of local management, that is, giving autonomy to local community to handle a part of their energy decisions, at different scales? This is a “system dynamic” issue, related to the handling of peaks, shortages and crisis. In a regular mode, there are obvious benefits to the centralized approach, from economies of scale to averaging pseudo-independent demand. The “pseudo” is a tiny word but full of consequences: the absence of independence is what produces complex systems and disasters, from the financial crisis to industrial accidents. It is the study of bursts and dynamic scenarios, with feedback loops, that may show the benefit of a “smart system” with faster counter-measures and learning abilities.
• What could be the large-scale effect of dynamic pricing on self-optimization of customer demand?  The “marginal story” of “peak demand shaving” is beautiful: to remove the few hours of peak-production with gaz/oil turbines which produce CO2 at a high marginal cost, by returning some of the value to the consumer, to convince her/him to postpone parts of her usage. The most common example being heating (house or water) since inertia makes postponing a viable alternative. However the story does not necessarily scale so well, nor does it necessarily address the “resilience issue”. If additional capacity is needed no matter what to cope with some kinds of peaks, its marginal cost for dealing with the “other types” become quite small. We are back to a “system dynamic issue” that cannot be resolved with a “back-of-the-envelope” ROI computation, but requires a large-scale simulation.

My goal is to get a first simple set of answers to these questions. Obviously, using GTES is not going to give me a price or a definite answer, but rather a “sense of how things are interrelated and react as a whole (system)”. Here is a simplified description of the model that I have built last month:

      This is a “game theory model” with four actors:

(1)    The regulator (government) which controls the price of CO2 and may both favor renewable energy (investment incentive through tax breaks) or regulate them (impose joint storage with new intermittent sources). The long term goal (what we call a strategy in GTES) of the regulator is to reduce CO2 while maintaining the economic output of the country.

(2)    The utility (national energy supplier) which runs its production assets, distribute electricity and sets its price dynamically according to demand and primary energy (oil / gaz) price variations. One will notice that I implement an ideal world where price can be set up freely and change constantly, which is very far from the truth, but I want to address question #4. The long term goal of the utility is to make money, deliver a proper return on investment for its new acquisition (if any) and ensure resilience (the ability to serve the necessary amount of energy in the future).

(3)    The local operator who operates a smart grid associated to a city or a county. The operator manages all local production and storage capacity that is linked to the grid (wind turbines, solar panels, storage, etc.). It also operates a fossil fuel small-scale plant to provide additional electricity when required, although it may also buy it at wholesale prices from the utility. The long term goal of the operator is simply to make money :)

(4)    The end consumer who tries to reduce her electricity bill (both its average value and its expected maximum value) while preserving her comfort. Optionally, this may mean to reduce her CO2 emissions :)

-        

    The smart grid architecture is pretty naïve. I simulate a country that could be France or another European country, with one electricity utility, one thousand smart cities/ smart communities (with their own local operator), and twenty millions households. The national supplier produces most of the electricity when the game starts, using a mix of nuclear and fossil plants. The electricity is either sold to the local operators, or sold directly to the consumers (hence each operator has a given market share, that is the percentage of households that get their energy from this alternative supplier).

-          The heart of the model is the demand generation. Running the model once demand is established is not difficult, although the operational mode of the operator requires a careful description, since many choices need to be made (local production vs. wholesale buying, using energy from storage or storing energy for future use, etc.). The demand generator is actually crude and starts from a yearly and a daily pattern, to which a lot of random noise is added. The model allows both for independent noise (each consumer is different) and dependent variation (weather variations are shared by everyone in the same city).

-          Each actor can play his game through a few decisions (what we call its tactic in GTES). The utility most important decision is its pricing tactic. Prices are defined through a bunch of linear formulas, the coefficient of which are GTES tactical parameters. For those who are not familiar with GTES, an evolutionary algorithm (local optimization) is run to optimize (find the best value) these tactical parameters. Other decisions from the utility involve further investments in its production plant.  The local operator has a similar range of choices to make. It needs to define its operational production mode; it also needs to set up its pricing scheme. Once a year, it needs to decide about new investments, whether they are additional fossil fuel production capacities, renewable energy investment or additional storage. The end consumer can decide to reduce her demand when the price gets too high, at the expense of comfort (the tradeoff between the two being a design parameter of the model). The consumer may also switch from national to local providers and back. Last, the consumer may invest in “negawatt equipment” (such as house insulation or more energy-efficient equipment).

This whole model defines a “game”, which could actually be turned into a real game, SIM-style. Each actor is trying to maximize its long-term goal while making the proper “tactical choices”.

What are the next steps ?

1. The code was written last month but I still need to run “20 years simulations” and make sure that the model is credible (the story told by one simulation run makes sense)
2. I then need to “explore the tactics”, which means that when one of the actor make a decision, the impact on the game outcome is credible. This is the most time-consuming part, even with a simple model like this. This ensures that the game is “realistic”, even if it is obviously naïve.
3. Apply game theory to find (Nash) equilibriums –  This is the fun part, since I have nothing to do and will leverage code that I have written for other problems. This is what GTES is designed for: looking at the conflicting strategies of the different actors.
4. The last step of GTES is to randomize the “design parameters”. The model relies on a number of design parameters such as the demand-generation curve, the sensitivity to price or the efficiency of peak shaving, to name a few. I have no way to calibrate the S-curves that I am using, so I randomize the choice of these design parameters (Monte-Carlo simulation) to see if their value changes the conclusion that I would like to draw from these repeated simulations.

I’ll post a summary of my results if and when the computational experiments are successful. Today’s goal was just to share my overall analysis of the “smart grids” domain.

Darwin, Lamarck and Service-Oriented Architecture

This blog has been sleeping in 2010 since I was writing a third book on « Business Processes and Enterprise 2.0 », an attempt to capture my past years involvement with lean management and information flows. Now that the book is over (I expect it to be published this spring), I am turning my attention back to autonomic/autonomous systems, networks and grids.

Although one could say that the promises of "Autonomic computing" (circa 2003/2004) have not materialized in the world of IT, the premises remain valid. My belief is that it will simply take longer to get effective technology in place in the world of corporate IT. As a research theme (which started much earlier and was very active in the 90s), "autonomous information technologies" (the combination of artificial intelligence, distributed control, adaptive software, quality of service monitoring … to name a few) is still very active.

I predict that there will be significant additional R&D efforts deployed in the coming decade, because of two related fields which are becoming "extremely hot", while requiring the same kind of scientific advances to transform hype into practical innovations:

• Smart Grids, where the ambition captured by the world "smart" mirrors the goal of autonomic computing : self-adaptive, self-organizing and self-healing. There is no need to explain why smart grids are strategic to this century, but it is also easy to recognize the implicit difficulty of the endeavor. The heart of the smart grid principle is to evolve from the centralized management of current power networks towards a distributed and adaptive design, which feasibility remains to be proven on a large scale.
  
• Home Networks, which are growing like mushrooms in our houses, and which have already reached a complexity level that is unacceptable to most households. "Smart houses" are necessary to fulfill the promises of multiple industries: energy – where smart, energy-efficient houses are actually parts of the previously mentioned smart grids -, content and entertainment – IP content anywhere, any time, on any device, home security and management, healthcare – for instance, out-care within the home, etc. The various control/distribution networks may all share the IP protocol, the complexity of provisioning, pairing, routing and interconnecting is rapidly becoming an impossible burden. Here also, the words "self-provisioning", "self-repair" and "self-discovery" are quickly becoming requirements from the customers.
  

It would not be difficult to define a SDT (Smart Distributed Thing) that regroups the challenges of distributed information systems, smart grids and smart home networks … With this first post of the year, I'd like to explore three ideas which I have been toying with during the past few months and which I intend to explore more seriously in the future.

1. There is a lot of wisdom in applying biomimetics to replicate evolution to produce autonomous systems. This is especially true for smart home automation networks. Rather than designing "a grand scheme" of "the smart house's nervous system", it is much safer to start with simpler subsystems, add a first layer of local control, then add a few reflexes (limited form of autonomy), then add a second layer of global control … and end up with a "cortex" of advanced "intelligent" functions. Multi-layered redundant designs such as those produced by evolution are more robust (a key feature for a home automation control network), more stable (a key insight from complex system theory which is worth a post by itself) and more manageable. The need for recursive/fractal architecture is nothing new: I wrote about it with respect to information system architecture many years ago in my first book. I went from a global architecture (which was common when EAI was a catchwordJ) to a loosely coupled collection of subsystems (so called fractal enterprise architecture), for the same reasons: increase robustness, reduce operational complexity and, most importantly, increase the manageability (the rate of evolution is not constant over a large information system). There is much more than fractal design involved here: the hierarchy of cognitive functions from low-level pulse, reflexes, to skills and then "creative" thinking, is equally suited to the design of a SDT.
   
2. Autonomous systems tend to scare end-users unless they embody the principles of calm computing. Calm computing is derived from the concept of ubiquitous computing (cf. the pioneering work of Mark Weiser at Xerox Park), and addresses the concerns that emerge when "computers are everywhere (ubiquitous)". Calm computing is very relevant to SDT, I could summarize the three main principles as follows: a smart ubiquitous system must act "in the background" and not "in your face" (it must be discrete J), it needs to be adaptive and learn from the interaction with its users (the complexity of the users must be recognized in the overall system) and, most importantly (from the user's perspective), it should be stoppable (you must be able to shut it down easily at any time). This becomes much easier with a fractal/layered design (previous point) and more difficult with a monolithic global design. There is a wealth of ideas in the early papers about calm technology, such as minimizing the consumption of the user's attention.
   
3. The emergence of software ecosystems most often needs to be guided/shepherded, and rarely occurs in the wild as random events. This is a key point since it is widely acknowledged that software ecosystems (such as iPhone applications) are where innovation occurs (and where value is created from the end-user's point of view). In the realm of home network, I have been advocating for open architecture, open standards and (web) service exposition for many years, thinking that open standards for the "Home Service Bus" would attract an ecosystem of service providers. You create the opportunity and evolution/selection of the fittest does the rest (Darwin). The last two years spent thinking about sustainable development (i.e., analyzing complex systems' architectures) and looking at successful software ecosystems have made me reconsider my Darwinian position. I am much more a follower of Lamarck these days: I see a "grand architect" in the success of many application stores, iPhone being the obvious example. Open standards and open API is not enough. The spectacular failure of major Web Service exposure programs from large telco is a good example. You need to provide SDKs (more assistance for the developper), a "soul" (a common programming model) and a sense of excitement/challenge/cool (which obviously requires some marketing).
   

 

This third point is precisely the cause for SOA (Service-Oriented Architecture). This is an observation that I have made earlier: reuse in the world of corporate IT does not occur easily or randomly, it requires serious work. To put it differently, to come up with a catalog of reusable services is not to deploy a service-oriented architecture (with a Darwinian hope that the "fittest" services would survive). To make SOA work, you need to organize (hence the word "architecture"), promote, plan and communicate. There is a need for a "grand architect" and a "common sense of destiny" for SOA to bring its expected benefits of sharing, reusing and cost reduction.

Taming Information Systems Complexity

This blog has been silent for a long time. I'll resume with a topic which is drawn from my course at Ecole Polytechnique: measuring and mastering the complexity of information systems. I have written many times, including in this blog, that the first job of a CIO is to master the complexity of her/his company's information system.

1. Which complexity ?

Although the difference between complex and complicated is fuzzy and varies according to the source (for instance, it is not supported by the TLF, the official dictionary of French language), it has emerged as follows: complicated is matter of size and scope, where complex describes the nature of the relationship between the components of a system. Complexity (in the sense of complex systems) arise when the finality and the behavior of a system cannot be derived from those of its component (hence the concept of emergence). The three most common ingredients in a complex system are:

• Feedback loops (and the non-linear resulting behavior that result from amplification)
  
• Delays (especially long-term delays) that generate "temporal complexity" which can easily puzzle us.
  
• Human factor, that is, the presence of humans as components of the global system.
  

Information systems are both complicated and complex. The fact that information systems are complex systems is something that I have touched upon in previous posts. I will give example of "complexity and emergence" in a future post, here I want to address complexity from a practical angle, as it appears to the CIO. Here is a summary of what makes information systems complex:

• Too many things: the sheer number of components, of apps, of interfaces. Although standardization and automation of component management help to master this dimension, it is obviously part of the problem (i.e. the information system of a small company is neither complex nor complicated).
  
• Too many interactions: these numerous components interact in many ways both explicit and implicit. Reducing the number of explicit interaction is the goal of enterprise architecture, and technology (integration middleware) may help. Implicit interaction, such as the use of a common resource, is more subtle to track. Reducing implicit interaction is the goal of a modular architecture, which is more an art than a science.
  
• Temporal complexity: many relevant time scales coexist, with both very short term delays which requires mastering so-called "real time" behavior and long term life cycles that demand to step back and anticipate
  
• Human complexity: information systems are centered (or should be) around human users. This is the source of uncertainty, of plain errors (e.g. typing errors) and interaction errors when users try to second-guess the system (which is unavoidable since humans are intelligent – cf. Charles Perrow's remarkable book "Normal Accidents").
  

2. Measuring Complexity ?

Measuring complexity is indeed difficult and I know of fewer measures than there exist "dimensions of complexity" as explained previously.

The first dimension (size) means to associate a weight to the information systems, which is a combination of counting and associating a weight to each component. This is the best understood part:

• Applications, or software components, can be measured using function points
  
• Computing resources may be measured using TPM-C (the most obvious choice for "commercial software" but other, more specialized metrics/benchmarks are available for specific purposes.
  
• Storage resources are easily measured in Teraytes or Petabytes.
  

  
  

The second dimension is structural complexity, which measures the richness of explicit interactions between components. The most common example of such a measure is cyclomatic complexity, which counts the number of elementary cycles within a graph (the interaction graph). Cyclomatic complexity was popularized a few decades ago and found useful to measure software architecture. A better approach for information systems is Euclidian Scalar Complexity (ESC). Given an architecture diagram with n objects with associated weights (w1, … wn) and m edges between these objects, the Euclidian scalar complexity is defined as:

• The square root of the sum of products (wi X wj), if i = j or components i and j are linked through an edge.
  

It is one of the rare metric that is scale invariant (insensitive to the "zoom effect") and invariant to extension without information loss. For more information, you may download a research article from Caseau, Krob and Peyronnet.

The third dimension is the complexity of implicit interaction, which is precisely the definition of modularity. Although one may define a co-evolution distance as the probability (among all possible changes) that an impact on component A also yields a change for component B, this definition is too theoretical to be useful. My own experience suggest to make specialized architectural diagrams for co-evolution (called "coupling") and to used ESC to measure the complexity of the resulting diagram. What are the possible causes for co-evolution? Here is a short and incomplete list to explain this concept:

• Objects: components that share business objects are co-dependent.
  
• Processes: similarly, the existence of a business process that uses both components A and B makes these two components linked (even if no direct reciprocal calls are made)
  
• User interfaces coherence: for instance, the requirement for a coherent multi-channel access may create dependencies among components that are functionally independents.
  
  
  
  
  To summarize, this is the "measuring discipline" that I suggest to my students:

• To be performed continuously : counting, sorting, weighing components (e.g.: function points)
  
• To be performed once in a while: applying ESC to the usual architecture diagrams and maps, producing coupling specialized architectural charts (i.e., process interaction, business object lifecycles …)
  
• To be performed "on demand": a detailed complexity analysis to decide between two architectural options

3. Taming Complexity

I have collected the following list of approaches, from simpler to more complex, which is actually quite thorough and effective, while still being quite practical (any comments on how to extend the list are welcome!). It is not a list drawn from "complex system theory", but rather from practical experience.

• Simple approach: draw diagrams and maps (cartography). This may sound silly, but drawing architecture diagram is still the best way to cope with complexity, assuming that the meta-diagram (the meaning of the graphical conventions) is well-understood. This is what makes UML2 so useful.
  
• Systematic approach: Enterprise Architecture (what we French call urbanization). Enterprise Architecture is, by construction, a method geared to reduce the information system complexity.
  
• Technology approach: Infrastructure (middleware). As mentioned earlier, integration infrastructures have a clear benefit over the structural complexity. It can actually be proven using ESC (one of my course favorite exercise).
  
• Common sense approach: Energetic Standardization. Reducing the heterogeneity of the components effectively reduces the complexity.
  
• Hardest approach: modularity (de-coupling), that is producing a modular architecture. As explained earlier, there is no guaranteed method, but it is a skill that is learned through trials and errors.
  
• Strategic approach: SOA (governance) as a strategic answer to complexity. SOA has a very positive impact on modularity and favors mutualization and reuse (hence a mechanical reduction of complexity). It also plays a crucial role in the governance of information systems, reducing the human complexity of satisfying the complete range of stakeholders.
  
• Sustainable development of the Information System. This is a topic which I have already covered in a previous post. Sustainable development, as advocated by SITA, is a way to master temporal complexity and to avoid painful paradoxes.
  

If this is the practical list, what would the "theoretical one add" ? Clearly, I would add the influence of biology (hence the theme of this blog) and "autonomic computing" to build systems that self-organize and self-manage their own complexity. This is an ongoing topic of reflection, to be covered in a future post.

New Shared Document

For reasons explained in my other blog, I am keeping quiet for a while. However, I have added the "Shared Documents" gadget on this blog and added a new presentation about SOA and BPM.

I gave this invited talk at the SOA & BPM IDC Conference on September 17th.

As usual, it is offered under creative commons rules.