Biology of Distributed Information Systems

Lean IT, Devops and Cloud Programming

2011-11-26T23:24:00.001-08:00

I have becomemore and more interested with lean IT over the years. It started with the book "TheArt of Lean Software Development" by Curt Hibbs.

I enjoyed this book because itsintroduction of lean is 100% compatible of whatI learned from Jeffrey Liker and other great authors about TPS. This simple book helped me drawthe lines between good software development methods such as extreme programming or agileprogramming, and lean management. For those of you who are not familiarwith lean or extreme programming, here is a very crude summary of some of themost salient points of lean software development :

No delays: avoid as much as possible workthat is sitting between two steps of the development process (what is calledWIP – work in process – in the lean jargon). This is true at the process level(the goal is to design a streamlines/ “single piece flow” developmentorganization) and the developer level. A practical goal is to avoid switchingtasks as much as possible: focus on one thing and do it right!
Quality in (right first time) : this lean principle translates into testingas early as possible (a tenet from agile programming) but also to use alltechniques that improve the quality of the code, even at the expense ofsource code productivity since we all know that it is cheaper not to produce abug than to remove it later. Here comes, for instance, the practice of pairprogramming and code reviews, but also “good practices” such as programmingguidelines and standards
Fast delivery: the lean principle is to reducethe “lead time” of the software development process, which requires working onall stages. Removing in-between delays (cf. earlier) is necessary but notenough. Continuousintegration is a core technique to achieve this goal, such as fast deploymenttechniques.
Short deliveries: It is more efficient to producesmall pieces of software at a high rate than bigger pieces at a lower rate.This is another key principle from lean (“small batches”), which is doubly truefor software development: not only are l smaller batches easier to build (awell-known law of software engineering), but the continuous evolution ofcustomers needs and environment makes it easier to adapt with a small batchapproach.
Less code: this is the application of the KISS principle ! Lean software development tries to stay awayfrom complexity (see later in this post). Unnecessary code may pop up from many sources, lean applies a techniquecalled VSM (Value-Stream Mapping) and a posture of “muda removal”. Muda (waste) removal means to go through theprocess with the “eyes of the customer” and remove all that does not producevalue from her perspective. VSM is a tool that tracks the value creation andassigns it to each step of the process. Lean software development aims atproducing the right product, without unnecessary features. It is also anarchitecture principle (stay away from complexity) designed at simpler andfaster maintenance over the years.
Customer participation: the best way to produce only whatis necessary from the user’s perspective is to ask her frequently ! This is whyend-user/customer participation is a tenet of agile programming. When thecustomer is not available, the principle mixes with “small batches” to become :deliver fast, failfaster to succeed sooner.
Regular work effort : the leveling of the effort is akey principle from extreme programming, the equivalent of “heijunka” from leanprogramming. A few years ago, when I was still a CIO, I started thinking about “extreme IT” (applying extremeprogramming at the information system level), and the sustainabilityof the effort is a crucial point. Regularity has a counterpart, which isdiscipline. Using methods and tools (such as configuration management, sourcecode versioning, automated testing) is crucial. One should take a look at the wonderful talk from Mark Striebeck on“Creating a testing culture” at Google.
Minimize handovers: complex tasks, such as writingsoftware, are better accomplished when the responsibility chain is nottruncated into too many segments. Working as a team is the only alternative to delivercomplex projects (this is the topic of mythird book). This is another insight from the Agile Manifesto, this is why today’s bestpractice it to assemble multi-disciplinary teams including software developers,marketers and designers.

There are anumber of interesting books on this topic. The most classical ones are thosefrom Mary Poppendieck, such as “ImplementingLean Software Development: from Concept to Cash”. I had the pleasure of presenting my last bookat the Lean IT Summit a month agoand to meet with Mary Poppendieck (a number of great talks, including hers,Michael Ballé’s and Georges Striebeck’s, are now available online).

There arealso books which are not about software development per se but very closelyrelated. I am especially fond of two books which Ihave reviewed in my French blog:

The Lean Startup, from Eric Riess,
Theprinciples of Product Development Flow, from Donald G. Reinertsen

Thanks to Guillaume Fortaine, I have come to learn about Devops. Quoting from Wikipedia, “DevOps" is anemerging set of principles, methods and practices for communication,collaboration and integration between softwaredevelopment (application/software engineering) and IT operations (systems administration/infrastructure) professionals”. The more I read about Devops (Devops.com is a treasure trove of greatarticles), the more I find that Devops is the missing link betweenAgile/Extreme Programming and lean management (Toyota-style, henceLean-Startup-style). Although Devopsclaims to “help finishing what Agile Development started”, there are differentflavors in what one may read on the web: bridgingthe development /admin & operations gap, makinglean IT happen, makingcollaboration happen (and also), bringingagility to operations, which inturns makes it easier to leverage the benefits of new computing resources suchas cloudprogramming. My first obvious interest with Devops has been the practicaldeployment of lean IT, as a followup of what I just explained earlier. Forinstance, Devops promote a number of interesting tools related to deploymentautomation, such as GLU.It turns out that the “Cloud and Devops intersection” is also quite promising.Indeed, leveraging thestrengths of cloud programming requires a shift in development/architectureculture that requires a “Devops-like” approach.

It happensthat I gave a talk last week at the Model-DrivenDay about « Complexity, Modularity and Abstraction” (the talk isavailable in the “My Shared Document” box on the left). The talk is about,among other things:

Complexity (why it is important, how to measureit and to tamecomplexity – since avoiding complexity altogether is not an option at theIS level)
Sustainability: how to transform enterprisearchitecture into a regular practice. This is related to the concept of “extreme IT” : avoid the “heroicstruggle” to move towards the “continuous transformation of the informationsystem”. This is a major reason why I have been advocating for SOAas a company-wide enterprise architecturepractice for many years.
“Architecture-oriented services” : Imade this pun to emphasize the difficulty to produce the “right” servicesthrough SOA. “Architecture-oriented” means services that have the right levelof abstraction, that are modular and “easy-to-compose”. To my knowledge, thereis no easy recipe for this, but the wisdom and folklore of 40 years of softwarearchitecture design apply.
Cloud computing: I have added a small additionabout “cloud-ready architecture” in the 4th edition of myfirst book. I strongly believe that Information Systems will change in aspectacular manner when we learn how to exploit massively parallel architecture(using tools/approaches such as Hadoop/MapReduce). Using Cloud Computing toprovide with a few Saas-based front-office services is nice and relevant, butthe big change comes when cloud-computing is applied to back-office services (provisioning,billing, data mining). This requires an architecture change, but mostlyrequires a culture change.

The thoughtthat prompted me to write this post is as follows. Devops is the missing link betweenthe themes of my MDD talk: managing complexity, delivering agility/modularityand moving to the new century of massively parallel computing (including, butnot restricted to, cloud computing). I speak of "Cloud Programming" in the title because I agree with a comment made by Georges Reese: the Cloud is a computing resources defined by its API, i.e., a resource that is managed through programming. The exact quote is:

Cloud is, for the purposes of this discussion, is sort of the pinnacle of SOA in that it makes everything controllable through an API. If it has no API it is not Cloud. If you buy into this then you agree that everything that is Cloud is ultimately programmable.

Not only the cloud is more agile (up/down scalability), it may be controled by the piece of software which is using it. From a systemic perspective, it's a "whole new game" :)

Systemic Simulation of Smart Grids

2011-09-25T07:38:00.000-07:00

This blog hasbeen extremely quiet for two years, mostly because I was busy with othertopics, including Lean and Enterprise2.0. In my previous post, I mentioned that I was looking at “SmartDistributed Things” through two instances, Smart Grids and Smart HomeNetworks. I will not talk about the second in this blog, since it iswork-related (we’ve come up with the acronym BAHN : Bouygues Autonomous HomeNetwork).

Smart Gridsmake an interesting instance of the “smart distributed network” because of theextraordinary amount of interest/excitement/hype that exists around this topic.There is a range of conflicting opinions, ranging from “this is a straightforwardand marginal improvement of existing networks which are already quite smart” to“smart grids are the backbone of a new sustainable society, based on communities,subsidiarity (organic,multi-scale, resilient organization) and self-aware optimal management ofresources”.

I am not anexpert with any of these topics: electricity, power grids, energy, storage …but like any citizen, I would like to build my own opinion about the future ofour country and our planet, as far as energy and global warming are concerned. Hencethe idea of a small systemic simulation has emerged during my vacation month(last month). I tried to assemble a crude model of all the different aspects ofthe “smart grid ecosystem” as we may understand it, without too much detail,just the broad principles. Each aspect is actually quite simple to explain, iftaken in isolation (e.g., anyone may understand why it is smart to couple asolar panel with local storage). It is the combination of all viewpoints,together with the huge amount of uncertainty (about the economy, the speed atwhich technology will get cheaper, the speed at which behaviors will change,etc.), which makes this a hard topic.

This iswhere I got this insight: I actually have a method for complex embedded modelswhere half is unknown and the other half is unclear: GTES(Game-Theoretical Evolutionary Simulation). GTES is the perfect platform to assembleconflicting views of what a smartgrid should be, conflictingviews about how the actors should behave, and try to generate some sense.In a word, I am planning to build a “serious game” to havea closer look at smart grids.

Let mefirst clarify what I said about the range of opinions and introduce three viewsof the smart grid, which I could label: the “Utility view”, the “Googleview” and the “Japanese view”:

The“Utility view” defines a smart gridas adapting the power network to localsources (as opposed to a one-way distribution network that goes from fewlarge GW production units towards millions of consumers), adapting to intermittent production sources (thoughstorage and favoring flexible production units that can adapt to the powersurges of intermittent sources like solar or wind) and using price incentivesto “shave” demand peaks. This is a “no-brainer” program (“what to do”is clear), where the major issue is price: most techniques show a cost/benefitratio that is worse than current practice. To believe in this approach, you mustbelieve that new technology prices will go down (e.g., solar, storage), or that electricity prices will go “throughthe roof” in the next 20 years, or that global warming fears will drive asignificant price for CO2.
The“Google view” defines a smart grid asa change from a tree structure to a networkstructure (centralized to de-centralized), the use of market forces tocreate a dynamic and more efficient equilibriumbetween supply and demand, and the useof IT to provide information to all actors, including end consumers. Theimportance of signals (pricing and power grid control) is so important in thisview that it is often said that telecom, IT and energy network will merge(something which I don’t believe for a second – but it shows the spirit of theimportance of “smart” in “smart network”). Calling this the “Google view” is afriendly reference to “WhatWould Google Do” J Thecore of this approach is the principle that a significant amount of efficiencywould be obtained with a “hyper fluid market”, made possible through ITtechnology.
The“Japanese view” is human-centeredinstead of being techno-centered. The goal is to change human behavior to adaptto new challenges (lack of resources, global warming, …). Smart grids are the backboneof a multi-scale architecture (smarthome, neighborhood, city, region, country) where each level has its ownresources and autonomy, resulting in a system that is more resilient and withmore engagement as a consequence of more responsibilities. Smart grids supportthe necessary behavioral transformation through communities and constant feedback. I call this a “Japanese vision” becauseI have heard it beautifully explained in Tokyo, but the systemic approach iscommon to Asia as a whole. Because active communities are “engaging”, SmartGrids help get rid of waste (muda in the leansense) such as useless transportation, un-necessary usage, etc.

These viewsdo not necessarily conflict; it is possible to envision the union of all theseideas. However, as soon as one tries to imagine a convincing deploymentscenario, there are a number of questions that pop to mind. Here are a fewexamples:

What part does local storage play? Can there be a smart grid withoutdistributed storage? What price hypotheses are necessary to make thisrealistic, considering that current energy storage price are too high tojustify large-scale deployment? Even if solar or wind becomes free, having tostore the energy with today’s techniques make this approach uncompetitive (withtoday’s parameters, obviously)
What CO2 price would changesignificantly the cost/benefits analysis? Actually, this extends more generally to the pairenergy prices + CO2 price, but CO2 price is especially significant. Because of theabundance of coal, and because of the increasing availability of new forms ofgaz, CO2 price is, from a naïve point of view, the key factor that could changethe economic analysis of introducing alternative intermittent sources ofenergy.
What is the systemic benefit oflocal management,that is, giving autonomy to local community to handle a part of their energy decisions,at different scales? This is a “system dynamic” issue, related to the handlingof peaks, shortages and crisis. In a regular mode, there are obvious benefitsto the centralized approach, from economies of scale to averaging pseudo-independentdemand. The “pseudo” is a tiny word but full of consequences: the absence of independenceis what produces complex systems and disasters, from the financialcrisis to industrialaccidents. It is the study of bursts and dynamic scenarios, with feedbackloops, that may show the benefit of a “smart system” with fastercounter-measures and learning abilities.
What could be the large-scale effectof dynamic pricing on self-optimization of customer demand? The “marginal story” of “peak demand shaving”is beautiful: to remove the few hours of peak-production with gaz/oil turbineswhich produce CO2 at a high marginal cost, by returning some of the value to theconsumer, to convince her/him to postpone parts of her usage. The most commonexample being heating (house or water) since inertia makes postponing a viablealternative. However the story does not necessarily scale so well, nor does itnecessarily address the “resilience issue”. If additional capacity is needed nomatter what to cope with some kinds of peaks, its marginal cost for dealingwith the “other types” become quite small. We are back to a “system dynamicissue” that cannot be resolved with a “back-of-the-envelope” ROI computation,but requires a large-scale simulation.

My goal isto get a first simple set of answers to these questions. Obviously, using GTESis not going to give me a price or a definite answer, but rather a “sense ofhow things are interrelated and react as a whole (system)”. Here is asimplified description of the model that I have built last month:

Thisis a “game theory model” with four actors:

(1) Theregulator (government) whichcontrols the price of CO2 and may both favor renewable energy (investmentincentive through tax breaks) or regulate them (impose joint storage with newintermittent sources). The long term goal (what we call a strategy in GTES) of the regulator is to reduce CO2 whilemaintaining the economic output of the country.

(2) Theutility (national energy supplier)which runs its production assets, distribute electricity and sets its pricedynamically according to demand and primary energy (oil / gaz) pricevariations. One will notice that I implement an ideal world where price can beset up freely and change constantly, which is very far from the truth, but Iwant to address question #4. The long term goal of the utility is to makemoney, deliver a proper return on investment for its new acquisition (if any)and ensure resilience (the ability to serve the necessary amount of energy inthe future).

(3) Thelocal operator who operates a smartgrid associated to a city or a county. The operator manages all localproduction and storage capacity that is linked to the grid (wind turbines,solar panels, storage, etc.). It also operates a fossil fuel small-scale plantto provide additional electricity when required, although it may also buy it atwholesale prices from the utility. The long term goal of the operator is simplyto make money :)

(4) Theend consumer who tries to reduce herelectricity bill (both its average value and its expected maximum value) whilepreserving her comfort. Optionally, this may mean to reduce her CO2 emissions :)

The smart grid architecture is pretty naïve. I simulate a country that could beFrance or another European country, with one electricity utility, one thousandsmart cities/ smart communities (with their own local operator), and twentymillions households. The national supplier produces most of the electricitywhen the game starts, using a mix of nuclear and fossil plants. The electricityis either sold to the local operators, or sold directly to the consumers (henceeach operator has a given market share, that is the percentage of householdsthat get their energy from this alternative supplier).

- Theheart of the model is the demand generation. Running the model once demand isestablished is not difficult, although the operational mode of the operatorrequires a careful description, since many choices need to be made (localproduction vs. wholesale buying, using energy from storage or storing energyfor future use, etc.). The demand generator is actually crude and starts from ayearly and a daily pattern, to which a lot of random noise is added. The modelallows both for independent noise (each consumer is different) and dependentvariation (weather variations are shared by everyone in the same city).

- Eachactor can play his game through a few decisions (what we call its tactic in GTES). The utility mostimportant decision is its pricing tactic. Prices are defined through a bunch oflinear formulas, the coefficient of which are GTES tactical parameters. Forthose who are not familiar with GTES, an evolutionary algorithm (localoptimization) is run to optimize (find the best value) these tacticalparameters. Other decisions from the utility involve further investments in itsproduction plant. The local operator hasa similar range of choices to make. It needs to define its operational productionmode; it also needs to set up its pricing scheme. Once a year, it needs todecide about new investments, whether they are additional fossil fuel productioncapacities, renewable energy investment or additional storage. The end consumercan decide to reduce her demand when the price gets too high, at the expense ofcomfort (the tradeoff between the two being a design parameter of the model).The consumer may also switch from national to local providers and back. Last,the consumer may invest in “negawatt equipment” (such as house insulation ormore energy-efficient equipment).

This wholemodel defines a “game”, which could actually be turned into a real game,SIM-style. Each actor is trying to maximize its long-term goal while making theproper “tactical choices”.

What arethe next steps ?

Thecode was written last month but I still need to run “20 years simulations” andmake sure that the model is credible (the story told by one simulation runmakes sense)
Ithen need to “explore the tactics”, which means that when one of the actor makea decision, the impact on the game outcome is credible. This is the mosttime-consuming part, even with a simple model like this. This ensures that thegame is “realistic”, even if it is obviously naïve.
Applygame theory to find (Nash) equilibriums – This is the fun part, since I have nothing to do and will leverage codethat I have written for other problems. This is what GTES is designed for:looking at the conflicting strategies of the different actors.
Thelast step of GTES is to randomize the “design parameters”. The model relies ona number of design parameters such as the demand-generation curve, thesensitivity to price or the efficiency of peak shaving, to name a few. I haveno way to calibrate the S-curves that I am using, so I randomize the choice ofthese design parameters (Monte-Carlo simulation)to see if their value changes the conclusion that I would like to draw fromthese repeated simulations.

I’ll post asummary of my results if and when the computational experiments are successful.Today’s goal was just to share my overall analysis of the “smart grids” domain.

Darwin, Lamarck and Service-Oriented Architecture

2011-01-02T03:50:00.001-08:00

This blog has been sleeping in 2010 since I was writing a third book on « Business Processes and Enterprise 2.0 », an attempt to capture my past years involvement with lean management and information flows. Now that the book is over (I expect it to be published this spring), I am turning my attention back to autonomic/autonomous systems, networks and grids.

Although one could say that the promises of "Autonomic computing" (circa 2003/2004) have not materialized in the world of IT, the premises remain valid. My belief is that it will simply take longer to get effective technology in place in the world of corporate IT. As a research theme (which started much earlier and was very active in the 90s), "autonomous information technologies" (the combination of artificial intelligence, distributed control, adaptive software, quality of service monitoring … to name a few) is still very active.

I predict that there will be significant additional R&D efforts deployed in the coming decade, because of two related fields which are becoming "extremely hot", while requiring the same kind of scientific advances to transform hype into practical innovations:

Smart Grids, where the ambition captured by the world "smart" mirrors the goal of autonomic computing : self-adaptive, self-organizing and self-healing. There is no need to explain why smart grids are strategic to this century, but it is also easy to recognize the implicit difficulty of the endeavor. The heart of the smart grid principle is to evolve from the centralized management of current power networks towards a distributed and adaptive design, which feasibility remains to be proven on a large scale.
Home Networks, which are growing like mushrooms in our houses, and which have already reached a complexity level that is unacceptable to most households. "Smart houses" are necessary to fulfill the promises of multiple industries: energy – where smart, energy-efficient houses are actually parts of the previously mentioned smart grids -, content and entertainment – IP content anywhere, any time, on any device, home security and management, healthcare – for instance, out-care within the home, etc. The various control/distribution networks may all share the IP protocol, the complexity of provisioning, pairing, routing and interconnecting is rapidly becoming an impossible burden. Here also, the words "self-provisioning", "self-repair" and "self-discovery" are quickly becoming requirements from the customers.

It would not be difficult to define a SDT (Smart Distributed Thing) that regroups the challenges of distributed information systems, smart grids and smart home networks … With this first post of the year, I'd like to explore three ideas which I have been toying with during the past few months and which I intend to explore more seriously in the future.

There is a lot of wisdom in applying biomimetics to replicate evolution to produce autonomous systems. This is especially true for smart home automation networks. Rather than designing "a grand scheme" of "the smart house's nervous system", it is much safer to start with simpler subsystems, add a first layer of local control, then add a few reflexes (limited form of autonomy), then add a second layer of global control … and end up with a "cortex" of advanced "intelligent" functions. Multi-layered redundant designs such as those produced by evolution are more robust (a key feature for a home automation control network), more stable (a key insight from complex system theory which is worth a post by itself) and more manageable. The need for recursive/fractal architecture is nothing new: I wrote about it with respect to information system architecture many years ago in my first book. I went from a global architecture (which was common when EAI was a catchwordJ) to a loosely coupled collection of subsystems (so called fractal enterprise architecture), for the same reasons: increase robustness, reduce operational complexity and, most importantly, increase the manageability (the rate of evolution is not constant over a large information system). There is much more than fractal design involved here: the hierarchy of cognitive functions from low-level pulse, reflexes, to skills and then "creative" thinking, is equally suited to the design of a SDT.
Autonomous systems tend to scare end-users unless they embody the principles of calm computing. Calm computing is derived from the concept of ubiquitous computing (cf. the pioneering work of Mark Weiser at Xerox Park), and addresses the concerns that emerge when "computers are everywhere (ubiquitous)". Calm computing is very relevant to SDT, I could summarize the three main principles as follows: a smart ubiquitous system must act "in the background" and not "in your face" (it must be discrete J), it needs to be adaptive and learn from the interaction with its users (the complexity of the users must be recognized in the overall system) and, most importantly (from the user's perspective), it should be stoppable (you must be able to shut it down easily at any time). This becomes much easier with a fractal/layered design (previous point) and more difficult with a monolithic global design. There is a wealth of ideas in the early papers about calm technology, such as minimizing the consumption of the user's attention.
The emergence of software ecosystems most often needs to be guided/shepherded, and rarely occurs in the wild as random events. This is a key point since it is widely acknowledged that software ecosystems (such as iPhone applications) are where innovation occurs (and where value is created from the end-user's point of view). In the realm of home network, I have been advocating for open architecture, open standards and (web) service exposition for many years, thinking that open standards for the "Home Service Bus" would attract an ecosystem of service providers. You create the opportunity and evolution/selection of the fittest does the rest (Darwin). The last two years spent thinking about sustainable development (i.e., analyzing complex systems' architectures) and looking at successful software ecosystems have made me reconsider my Darwinian position. I am much more a follower of Lamarck these days: I see a "grand architect" in the success of many application stores, iPhone being the obvious example. Open standards and open API is not enough. The spectacular failure of major Web Service exposure programs from large telco is a good example. You need to provide SDKs (more assistance for the developper), a "soul" (a common programming model) and a sense of excitement/challenge/cool (which obviously requires some marketing).

This third point is precisely the cause for SOA (Service-Oriented Architecture). This is an observation that I have made earlier: reuse in the world of corporate IT does not occur easily or randomly, it requires serious work. To put it differently, to come up with a catalog of reusable services is not to deploy a service-oriented architecture (with a Darwinian hope that the "fittest" services would survive). To make SOA work, you need to organize (hence the word "architecture"), promote, plan and communicate. There is a need for a "grand architect" and a "common sense of destiny" for SOA to bring its expected benefits of sharing, reusing and cost reduction.

Taming Information Systems Complexity

2010-01-25T09:28:00.001-08:00

This blog has been silent for a long time. I'll resume with a topic which is drawn from my course at Ecole Polytechnique: measuring and mastering the complexity of information systems. I have written many times, including in this blog, that the first job of a CIO is to master the complexity of her/his company's information system.

1. Which complexity ?

Although the difference between complex and complicated is fuzzy and varies according to the source (for instance, it is not supported by the TLF, the official dictionary of French language), it has emerged as follows: complicated is matter of size and scope, where complex describes the nature of the relationship between the components of a system. Complexity (in the sense of complex systems) arise when the finality and the behavior of a system cannot be derived from those of its component (hence the concept of emergence). The three most common ingredients in a complex system are:

Feedback loops (and the non-linear resulting behavior that result from amplification)
Delays (especially long-term delays) that generate "temporal complexity" which can easily puzzle us.
Human factor, that is, the presence of humans as components of the global system.

Information systems are both complicated and complex. The fact that information systems are complex systems is something that I have touched upon in previous posts. I will give example of "complexity and emergence" in a future post, here I want to address complexity from a practical angle, as it appears to the CIO. Here is a summary of what makes information systems complex:

Too many things: the sheer number of components, of apps, of interfaces. Although standardization and automation of component management help to master this dimension, it is obviously part of the problem (i.e. the information system of a small company is neither complex nor complicated).
Too many interactions: these numerous components interact in many ways both explicit and implicit. Reducing the number of explicit interaction is the goal of enterprise architecture, and technology (integration middleware) may help. Implicit interaction, such as the use of a common resource, is more subtle to track. Reducing implicit interaction is the goal of a modular architecture, which is more an art than a science.
Temporal complexity: many relevant time scales coexist, with both very short term delays which requires mastering so-called "real time" behavior and long term life cycles that demand to step back and anticipate
Human complexity: information systems are centered (or should be) around human users. This is the source of uncertainty, of plain errors (e.g. typing errors) and interaction errors when users try to second-guess the system (which is unavoidable since humans are intelligent – cf. Charles Perrow's remarkable book "Normal Accidents").

2. Measuring Complexity ?

Measuring complexity is indeed difficult and I know of fewer measures than there exist "dimensions of complexity" as explained previously.

The first dimension (size) means to associate a weight to the information systems, which is a combination of counting and associating a weight to each component. This is the best understood part:

Applications, or software components, can be measured using function points
Computing resources may be measured using TPM-C (the most obvious choice for "commercial software" but other, more specialized metrics/benchmarks are available for specific purposes.
Storage resources are easily measured in Teraytes or Petabytes.

The second dimension is structural complexity, which measures the richness of explicit interactions between components. The most common example of such a measure is cyclomatic complexity, which counts the number of elementary cycles within a graph (the interaction graph). Cyclomatic complexity was popularized a few decades ago and found useful to measure software architecture. A better approach for information systems is Euclidian Scalar Complexity (ESC). Given an architecture diagram with n objects with associated weights (w1, … wn) and m edges between these objects, the Euclidian scalar complexity is defined as:

The square root of the sum of products (wi X wj), if i = j or components i and j are linked through an edge.

It is one of the rare metric that is scale invariant (insensitive to the "zoom effect") and invariant to extension without information loss. For more information, you may download a research article from Caseau, Krob and Peyronnet.

The third dimension is the complexity of implicit interaction, which is precisely the definition of modularity. Although one may define a co-evolution distance as the probability (among all possible changes) that an impact on component A also yields a change for component B, this definition is too theoretical to be useful. My own experience suggest to make specialized architectural diagrams for co-evolution (called "coupling") and to used ESC to measure the complexity of the resulting diagram. What are the possible causes for co-evolution? Here is a short and incomplete list to explain this concept:

Objects: components that share business objects are co-dependent.
Processes: similarly, the existence of a business process that uses both components A and B makes these two components linked (even if no direct reciprocal calls are made)
User interfaces coherence: for instance, the requirement for a coherent multi-channel access may create dependencies among components that are functionally independents.

To summarize, this is the "measuring discipline" that I suggest to my students:

To be performed continuously : counting, sorting, weighing components (e.g.: function points)
To be performed once in a while: applying ESC to the usual architecture diagrams and maps, producing coupling specialized architectural charts (i.e., process interaction, business object lifecycles …)
To be performed "on demand": a detailed complexity analysis to decide between two architectural options

3. Taming Complexity

I have collected the following list of approaches, from simpler to more complex, which is actually quite thorough and effective, while still being quite practical (any comments on how to extend the list are welcome!). It is not a list drawn from "complex system theory", but rather from practical experience.

Simple approach: draw diagrams and maps (cartography). This may sound silly, but drawing architecture diagram is still the best way to cope with complexity, assuming that the meta-diagram (the meaning of the graphical conventions) is well-understood. This is what makes UML2 so useful.
Systematic approach: Enterprise Architecture (what we French call urbanization). Enterprise Architecture is, by construction, a method geared to reduce the information system complexity.
Technology approach: Infrastructure (middleware). As mentioned earlier, integration infrastructures have a clear benefit over the structural complexity. It can actually be proven using ESC (one of my course favorite exercise).
Common sense approach: Energetic Standardization. Reducing the heterogeneity of the components effectively reduces the complexity.
Hardest approach: modularity (de-coupling), that is producing a modular architecture. As explained earlier, there is no guaranteed method, but it is a skill that is learned through trials and errors.
Strategic approach: SOA (governance) as a strategic answer to complexity. SOA has a very positive impact on modularity and favors mutualization and reuse (hence a mechanical reduction of complexity). It also plays a crucial role in the governance of information systems, reducing the human complexity of satisfying the complete range of stakeholders.
Sustainable development of the Information System. This is a topic which I have already covered in a previous post. Sustainable development, as advocated by SITA, is a way to master temporal complexity and to avoid painful paradoxes.

If this is the practical list, what would the "theoretical one add" ? Clearly, I would add the influence of biology (hence the theme of this blog) and "autonomic computing" to build systems that self-organize and self-manage their own complexity. This is an ongoing topic of reflection, to be covered in a future post.

New Shared Document

2009-09-19T23:39:00.000-07:00

For reasons explained in my other blog, I am keeping quiet for a while. However, I have added the "Shared Documents" gadget on this blog and added a new presentation about SOA and BPM.

I gave this invited talk at the SOA & BPM IDC Conference on September 17th.

As usual, it is offered under creative commons rules.

Kolmogorov and the measure of competitive value

2009-07-11T23:07:00.000-07:00

I was fortunate enough to attend USI and, although I could not participate to all the sessions, it has been quite fruitful. The (summer) "University of the Information System" is organized by Octo, BCG, le Monde Informatique and TV4IT. It is a great gathering for "bosses and geeks", with lots of opportunities for networking (and meeting old friends as far as I am concerned) and learning exciting stuff (the list of keynotes is amazing).

1. Complexity

I'll start with the key idea that attending a brainstorming session moderated by Luc de Brabandere generated. It may be stated in a pompous manner as:

CompetitiveValue(IS) = f(complexity)
The competitive value from Information Systems is a function of their complexity (in the Kolmogorov sense)

It starts as follows: what is not complex is easily reproduced and become a commodity, something that anyone can use and that may not, therefore, seen as a competitive advantage. For instance, the chisel in the hand of the stone carver is such a commodity tool. Although it is crucial to the task, and is taken great care of, the chisel is not a differentiating factor. Anyone can get a great chisel. What makes a great stone statue is the talent and the craft from the hands of the stone carver. For those companies for which IT is a differentiating factor, there is a fair amount of complexity that has been mastered, from a size, a technological or a business integration perspective.

This is actually very close to the concept of information measure as defined by Kolmogorov. Let's recall that Kolmogorov measures the complexity of an information sequence as the size of the smallest program that can generate the sequence. Anything that is very rich but generated from a set of few rules has a small Kolmogorov's complexity, while chaotic and random structures have a high Kolmogorov complexity. Here the measure of the information systems is precisely what cannot be reduced to a set of rules and a few enabling technology. If you have a large information system which is using standard tools, standard techniques in a usual manner, its complexity measure will be small. If you have an information system that is uniquely tuned to the business, where the practical know-how built over years has helped to resolve technical difficulties, its complexity measure is high.

This approach is strikingly compatible with Nicholas Carr's position on "IT does not matter", that is, IT without complexity is a commodity. One must read the original article or the book to see that Carr is talking precisely about the competitive value of information systems. His vision, which is fairly optimistic in its timing but generally accepted as target architecture, is that Web Service mash-ups will transform IT into a commodity. This Web Service / Cloud IT will not be without value (as necessary as electricity) but without differentiating value. I disagree about the availability of this “commodity IT” (cf. my book “Information Technology for the Chief Executive”, whose second chapter talks about N. Carr’s position), but I definitely agree with the (obvious) statement that there is no differentiating value with a commodity service.

One could say IT without complexity does not exist, but it is not true. Software as a service, for instance, is clearly one direction to remove a fair amount of complexity. Really simple IT exists; unfortunately it cannot solve all problems. At the end of the day, there remains a lot of complexity, irrespective of the technology or the procurement options that are chosen (cf. the Web site of the SITA: Sustainable IT Architecture). This is why companies need a CIO in the first placeJ. For me, the first job of the CIO is to manage complexity. This includes:

Reducing complexity through an Enterprise Architecture approach,
Removing complexity whenever possible, that is empower users to manage their own information system;
Taming complexity through collaboration and training

The paradox is that the CIO’s mission is to constantly reduce the perimeter of her/his job. But since the mission of any enterprise is to be smarter than its competitors, new challenges keep been thrown at the CIO …

Cloud computing is about complexity “de-materialization”: the complexity does not vanish (cf. SOA is not scale free), only its nature changes. If the IS is managing a complex set of business processes with a lot of service interactions, rapid changes and performance constraints, moving the IT on the cloud does not make the complexity disappear.

Back to this idea of Enterprise Architecture, the challenge is to produce a flexible architecture which may sound as an oxymoron. More precisely, one must create structure without rigidity. This prompts a few suggestions:

Be wary of “invariants” - there are few of them. So-called invariants are traps to install rigidity (for French readers, this is a topic covered in my first book)
Reference designs are living objects. Architecture relies on a number of reference designs: data models, service catalog, integration framework, etc. Any good Enterprise Architecture methodology will tell you how to build extensible designs. It looks like design violation is closely associated with innovation (?)
Diversity is key (a theme from the second day of the USI, I’ll be back in a moment).

This line of thoughts brings us back to biology and the general theme of this blog. In the living world, “invariants” (building blocs) are small and they are versatile (better than flexible). As explained by Albert Jacquard during his magnificent talk, diversity comes from reproduction, hence from randomness.

2. Uncertainty

The brainstorming session which generated all these ideas was about uncertainty. How to live, how to create, how to be relevant in a uncertain world?. Luc de Brabandere used a different set of scenarios, which are somehow similar to the four scenarios of Dan Rasmus in his book “Listening to the Future”.

I am a big believer in this approach to define a proper strategy for information system (cf. previous reference to the second chapter of my book). A scenario is not forecast, it is a virtual situation designed to foster creativity. This is a key point: the scenario’s value is not to be as close to what will happen in the future, it is to help build skills that will prove useful in the future (for the information systems or for the employees). In a world, the goal of the “scenario exercise” is to develop one's situation potential.

I have developed a “theory” over the years (cf. my other french blog), that the only tool to master uncertainty is gaming (as in "serious gaming"). Games are based on virtual scenarios but may develop true skills or help better understand the possibility of the future. I will return to this idea in a future post. What came to me as a conclusion of this afternoon session is that “Participants must participate”: passive viewing is of (almost) no value. This is deeper than it looks: since the scenario is not interesting per se, the value is the thought experiment and the collaboration that occurs between the participants while they play with the scenario. If a summary is proposed to a set of external listeners, most of the time it sounds dull or strange. I came to express this as a communication rule: if you need to report the result of a scenario-brainstorming session to your managers (or some other managers), it must keep the form of a role-playing exercise where the audience is actively engaged.

3. Value

This first section of this post dealt with “differentiation value”. What about “regular” value? The classical issue of the value that is produced by information systems was, as one would expect, central to this year’s USI, with one dedicated session on this topic. A good reference on this topic, by the way, is Amhed Bounfour’s book “Organizational Capital”.

That session started with an outlook on the issue, stating that there is neither consensus nor any method that would be applicable to the whole spectrum of issues (I agree, I wrote as such in the previously mentioned bookJ). Octo’s proposed approach is to define a “usage value”, very similar to Adam Smith’s definition: the value of a component of the information system is the additional amount of time it would take to perform the task without this component. It is expressed as a monetary value and actualized (over a given amount of time, such as the life expectancy of the software component.

It is a convenient measure, because it is easy to understand and relatively easy to evaluate, at least when an order of magnitude is concerned. Obviously the value must be capped by the total amount of money generated by the associated business process, in order to cover activities that would not exist without an IT platform (e.g., something that would require a million hands and generates little value). It has a few nice properties: it takes the quality of service into account, as well as the true deployment of the component. A beautiful application that is almost never used has a null value with this approach, a desirable property which is not true of all methods!

It is also a shortsighted measure, proving once again that it is hard to conciliate all objectives (cf. the introductory point). This measure does not take the future into account and how the information system is ready to embrace change. One could argue that “usage value” could be made future-oriented with a scenario approach, following the tracks of the previous section … and that’s true but that’s hard.

Adam Smith’s definition was quoted one day earlier by Daniel Cohen during a great evening speech. He talked about Philippe Askenazy's work on the Solow’s paradox (the absence of evidence of productivity gains due to IT). Philippe Askenazy, through a careful study though a large sample of data points, was able to show that IT can bring value only in conjunction with re-organization:

Value = Information System + Re-engineering

Those company who decided to reorganize themselves as they introduce IT in their processes showed significant returns after a few years, while those whom embraced IT but did not change, had nothing to show but costs. This is a great piece of evidence since it supports a claim easy to understand for any CIO: IT revolution only works if used as a lever to re-organize and re-optimize work (hence the importance of business processes).

I will conclude with a great idea from Pierre Pezziardi and Laurent Avignon: foster innovation through opening new territories, places where anyone can contribute to the information system. Obviously one cannot allow anyone to touch anything anywhere, hence the concept of “new territories”. This would be a “zone” where almost anyone can make a contribution (a piece of software) that may be used by others. There is a lot of value there:

Foster creativity and innovation
Improve the image of the information system, from a dark mystery to a friendly tool J
Support collaboration and engage a dialog with users
Find all the talents that hide in the company (i.e., not in the IT department) and who could contribute
Introduce diversity: the peaceful coexistence of safe, structured islands with active (even chaotic) agile platforms

For instance, one could open a few web services that give access to the heart of the information system (in a read mode for a first experiment J) and pick a sand box (such as Excel, Microsoft, Salesforce.com, Google App Engine) that is exposed with a SaaS (Software as a Service) philosophy. The ease of deployment (and adoption) is crucial here to make this a meaningful event. This should turn into a big innovation contest!

The general theme of Pierre and Laurent’s show “L’informatique conviviale” (it was nicely executed on stage in a very lively manner):

Diversity is necessary, Pleasure is key.

The first point is well developed in OCTO’s book “Une politique du Système d’Information”. Their dialectic analysis is too rich to be reported here, but may be summarized as follows: there are too many conficting constraints on the information systems to adopt a unique set of policies. Different parts of the information system must be governed with different approaches, to reconcile the needs for innovation, agility, safety, performance, reliability, etc. This “zoning” of the information system requires clear “passage rules” to support the exchange flows between the different zones.

The second point was the heart of their talk: build an information system that brings pleasure to the users and pleasure to the developers. This is a great insight since biologists tell us that there is no learning (continuous improvement) without pleasure. I will conclude with something that I learned a year ago at a conference about complex systems, on the process of learning for all living organisms, from the smallest to the most complex (us).

I had learned ten years ago about the PDCA cycle of continuous improvement from Deming: Plan, Do, Check, Act. There is a similar cycle that nature has invented for learning: Desire, Plan, Execute and Please. Desire creates the will to plan, to formulate a goal (for conscious livings), to get ready. The execution yields pleasure, which strengthen the desire and reinforces the cycle. I believe that pleasure is an integral component of corporate/collective learning and continuous improvement. Pleasure can take many forms, from pride (in Japan) to simple fun (in a Silicon Valley's start-up).

Sustainable IT Budget in an Equation

2009-06-07T07:00:00.000-07:00

This post is going to be technical, if you are allergic to math formulas, I suggest that you skip this or download my talk (in french) about IT value :)

The topic is the search for a sustainable IT budget, a thought that occured to me during the SITA conference on April 30th. I figured that I could get some characteristic equation of a sustainable state as a fixed point of the IT budget equations. I have spent quite some time (over 10 years) modelling IT costs. Some of it may be found in my last book, it is also a key part of my Polytechnique course. This course is focused on Enterprise Architecture, following the analysis of the CEISAR, but is also heavily oriented towards economic analysis (cf. the previously published bibliography).

Before I dive into the technical analysis, I'd like to stress out that I am not looking for a stable information system. Information systems are living objects, with new parts beeing constantly added and (hopefully) other parts being removed. What I am looking for is homeostasis, that is a property that is verified by the IS considered as an open system, even though the parts are changing constantly. Here, the property that I consider is that the budget grows at the same rate as the turnover of the company.

What I did is to take the cost model that is published in my book, and simplify it to reduce the number of necessary variables. The biggest simplification comes from using acquisition costs as opposed to function points to measure the application portfolio. I won't go into details today, it actually makes sense.

The budget is defined as the sum of the project budget P and the Operation budget O. Project are used either to add new applications or to improve/maintain/renew/update existing ones. The two ratios that are of interest (based on what on hears in IT conference or benchmarking) are:

R = (E / E+O) = the % of the IT budget spent for projects (we also use r = E/O)
n = Pn/P = % of the project budget spent on creating new applications

The model that I use is straightforward:

each year, Pn € is spent acquiring/building new applications
a given percentage (d) of the application portfolio is removed/destroyed
(P - Pn) is spend renewing a fraction of the remaining apps, so that the average life expectency of apps is A (measured in year, or we can say that the renewal rate is 1/A)
We suppose that the operation cost for an application that costs 1k€ is w k€ - A typical value is 25% (once again the references may be found here).
We also suppose that there is a productivity gain over the years of p%. That is, until the application is renewed (or discarded) its operation cost decreases by p% each year.

The IT budget is considered stable if both P and O grow at the same rate g as the turnover (such that r is also a constant). Looking for a stable state yields three equations:

stability of the application portfolio (dS/S = g)
we get: w* = g+d / nr (weighted operation cost, smaller than w because of p)
renewal rate of the apps (P - Pn = S/A)
we get n = 1 - 1/(w* Ar)
stability of the operation expenses (dE/E = g)
we get a huge formula:
dE/E = [(1-p)(1-d)(1-1/A) - 1] + (w/w*)(1-d)/A + wnr

The math, although high-school grade, is a little tricky. I implemented an Excel version of this model, in order to get a first hand opinion about the behaviour of this simple model and also to check that my equation algebra was correct !

The good news is that I managed to get it right eventually, that is, the equations may be resolved to express r and n as a function of the other parameters. Here are my two theorems (I will write a paper with the proofs one day):

n = [A (g + d) ] / (1 + (g+d) A)
r = [g +d +p +(1-d)/A] / [w (1 - d + A(g+d))]

The second formula is approximated (I left aside a few "second-order" subterms) but is still quite accurate when compared with the real value from the simulation. Note that if you suppose that g = 0 (i.e., a stable IT budget for a stable company), the equations become even simpler.

What can be derived from these formulas ?

First, they actually provide useful values. If w = 25%, d = 5%, p = 4%, A = 10, we get R = 42% and n = 33%. For someone who has been a CIO for many years, these values make sense. It is important to realize that if n is, say, 50%, then the IS is unstable and will grow !

Second, they show something that I have claimed for a long time : you cannot compare companies using R as a benchmarking ratio if you do not know A ! I have heard so many hours of bullshit SOA infrastructure presentation, which tell how to increase R without taking the proper parameters into account (this is actually why I decided to write my first book: I was fed up listening to consultants who had no clue what they were talking about). I heard the same mistake made by experts (at the VP level of large US software companie). Hence I am glad to have such a compact counter-argument, something I have been looking for over ten years.

Last, the r formula gives a good summary of what can be done to increase the amount of money that can be spent on projects:

clean-up old applications
increase productivity for operations

Nothing new here, but it is nice to see that age-old sound advice (often ignored) may actually be proven.

SOA : A Tale of Two Cities

2009-05-02T22:33:00.001-07:00

This post is an attempt to illustrate a key message: the benefits of SOA are dwarfed by the inconvenient of not doing it. When I write SOA, I mean the combination of MDM (Master Data Management), EA (Enterprise Architecture) and Service Architecture on a global scale. This has been the topic of many of my posts, so I won't go into details. I won't detail the expected benefits either (cf. the previous post on SOA's presumed death) : cost reduction, increased agility, sharing and complexity reduction.

What is clear for everyone is that this approach has a cost. It can be a large set-up cost for a first project or a moderate "architectural investment" if SOA is a sustained practice. I have witnessed the two alternatives when a large-scale new project is launched: with and without such an effort.

It is now clear for me that the true difference in the outcome is not the set of expected benefits of a "disciplined/architected" approach, but rather the unexpected benefits (what one could call the strategic agility) and even more the avoidance of major problems in the future life of the IT system that is being built, mostly with respect to data integration.

Separating between strategic and tactical agility makes sense. Tactical agility may be defined as the ability to make the "easy transformations" to the information system (IS) as easily and cheaply as possible (they go hand in hand). Strategic agility is measured by the cost of making "the hard changes" (those that are deemed "impossible" the first time the need is mentioned). Easy vs. hard is both a matter of anticipation (strategic agility is the ability to move the IS towards a brand new direction) and scope. Most of the technology, such as middleware, is geared towards tactical agility. It helps to implement "reasonably easy changes" faster (sometimes much faster). But what helps to "turn around the ship a full 180 degrees" is the hard work on architecture (mostly, data architecture and then, service architecture). See the recently posted bibliography for more pointers. Strategic agility is difficult to evaluate, but not impossible. Playing with scenarios seems to be the best approach. See my book, or "Organisational Capital", edited by Ahmed Bounfour. For the lack of a better word, I will call structural agility the "status of being able to avoid major problems" through the taming of complexity. When complexity is not curved, unseen consequences start to happen. This is when we see huge overruns in budget and time, or even complete failure of large projects.

I am always worried when I see a "core-centric" project, usually motivated by the introduction of a new technology. A "sacred alliance" occurs between the client who sees the new technology as a quick relief of a precise pain, some itching that has been going on for a while, and the IT folks who are always happy to try on a new thing. New technology here may be a rule-based engine, a new database techno, some learning/recommendation engine, a new Web/Interface generation tool, etc. What I mean with "core-centric" is the focus on the core "new thing", the core "new benefit", as opposed to the edge: the integration with the rest, the way the new system interacts with the old stuff. Core-centric projects always follow a successful proof of concept. When the focus is on the core, it is actually hard to fail a proof of concept … I have, obviously, nothing against proofs of concepts. They are clearly necessary, there is no reason to work on the hard stuff (the edge) if the core benefits is not worth it. But one must keep in mind that a successful "proof of concept" is theeasy part. A common plague of core-centric projects is that they are often "designed by committees". This will be the topic of another post … an IT component need to have on clean business customer (a physical person) and one identified architect.

Core-centric projects tend to start well and to end in misery when adding the last 20% of data integration seems to cost 80% of the effort. The sad thing is that one cannot buy an enterprise architecture from an outside vendor (although help/consulting works J). A "turnkey" project, even with a high quality supplier, delivers exactly what you pay for, but no more. Precisely, the level of integration is what is defined in the specification. Extensibility, the ability to add a new data source, to take a new business practice into account, to adapt to a new competitor… are why architecture is necessary before adding a new IT component. One cannot expect to get this kind of "forward thinking" from an outside vendor. ISVs cannot, as a general rule (and each rule has its exception) carry the weight of the integration issue. Another way to say this is that integration should be "inward-bound" and not "outwards-bound": what matters most is outside the new project, not inside. This applies especially to SOA: it is much easier to define the services that a new component may expose than the services that it may use.

IT Strategic Alignment, a buzzword of these last 10 years is indeed a difficult exercise, because of the dynamics of the target – constantly moving - and the inertia of the IS. Enterprise Architecture is about system dynamics and trajectories. Aligning over the "strategy" is necessarily difficult because the target is vague and shifting its shape continuously. The fast movement of the target coupled with the slow speed of IT transformation means that both anticipation and abstraction are required. Anticipation is necessary because transforming the IS takes time. Even within the framework of a sustained SOA effort, herding the set of services towards the desired service architecture takes time. Abstraction is required to filter out the "micro-variations" and to focus on the key long-term changes.

Let us pick an analogy: imagine that we want to wrap objects of various shapes that are given to us randomly, with sheets of a rigid material. A good example would be statues, since they exhibit very different forms. To prepare beforehand, we pre-fold the sheets. Obviously, the objects represent the business opportunities … while folding the sheets represent undergoing IT projects to fit the opportunities. The preparation beforehand is similar to enterprise architecture: a little effort in advance to speed up things when the real problem occurs. However this preparation only helps if the folds are actually useful to wrap the new shapes. It is actually an interesting analogy since finding a set of versatile preparatory folds is a hard problem.

The following set of illustrations is taken from wrappings by Christo. They are not the best illustration of this fictitious example (no pre-folding since the wrapping material is not rigid) but they look nice J

From this analogy we can pick three key principles:

One cannot do the architecture without knowing what "the future holds" from a business perspective : Service Architecture is about business
Going from the shape to the folds is tricky : Architecture is not for "dummies", it requires thinking and abstraction
One can overdo it and spend more time solving the puzzle than solving the business problem. It is easier to wrap a complex form than find the optimal set of folds that would help wrapping the most.

It would be easy to transform this post into a fictional story to contrast two approaches for a new complex IS project, with and without a SOA investment. I might do this as a new "Caroline story" for a new edition of my book, in a Tale-of-two-cities' style. So, to return to the initial question, what could motivate Caroline to "do it right", if it means a slower start and an increase in the total cost ? Purely defensive arguments are hard to sell, even if perfectly correct (e.g., a higher cost estimate but a higher probability of avoiding overruns). This brings us back to the concept of "situation potential", borrowed from Chinese strategy and which I have mentioned earlier. This is truly a powerful idea, worth yet another post; it unties the Gordian knot that mixes complexity, the difficulty to forecast and the different time scales. It may be seen as the combination of tactic agility, strategic agility and structural agility. Being able to demonstrate and sell the increase of "situation potential" is what it takes to develop a long-term Enterprise Architecture effort.

Selected Bibliography

2009-04-24T00:21:00.001-07:00

Here is a short selection of my favorite books about IT and Enterprise Architecture. I will try to update and develop this selected bibliography in the future, and I am always on the lookout for additions. A more detailed list may be found in the bibliography section of my last book, but this is a "selection from the heart".

The first list I have assembled for my course at the Ecole Polytechnique. These books are both inspiring and reasonably easy to read J

R.J. Wieringa, "Design Methods for Reactive Systems: Yourdon, Statemate, and the UML», Morgan Kauffman (2002)
A really great book about design methods, with both a lot of structure (theory) and practical insights from the domain of reactive systems. Very relevant for complex fields such as telecommunications.
P. Roques, "UML 2 en action : De l'analyse des besoins à la conception », Eyrolle (2007)
This is not a reference book (there are better books to learn about UML) but this is the best book I know to understand how to generate value from the practical application of UML.
P. W. Keen, « Shaping the Future: Business Design Through Information Technology », Harvard Business School Press (1991)
A key reference (heavily quoted) that is still the most comprehensive book on the topic of IT economics. Much better than newer books, especially nice for rebuking naïve statements about SOA, Web Services … or other "silver bullets". The best counterargument against "IT does not matter", from N. Carr, that I have read.
L. H. Putnam, "Five Core Metrics: The Intelligence Behind Successful Software Management», Dorset House Publishing (2003)
The smartest book I have found about software metrics. All the mistakes I have made previously are neatly identified. Not only the difficult, multi-dimensional aspect of software measurement is well accounted for, but the book provides with practical and efficient methods.
J. Printz, "Coûts et durée des projets informatiques pratique des modèles d'estimation », Lavoisier(2002)
A very nice introduction to Cocomo and other methods.
I. Jacobson, "The unified software development process », Addison Wesley (1980)
A classical reference that still makes a very good reading. Extremely useful to understand the current state of "software development best practices"
P. Grosjean & al., "Performance des architectures IT », Dunod (2007)
Amazing book: very practical yet rigorous, covers a large scope of issues and provides with very relevant solutions to real world problems.
X. Fournier-Morel & al, "SOA, le guide de l'architecte du SI », Dunod (2008)
The best book I have ever read about Service Oriented Architectures. Obviously, covers the technical more than the governance side of SOA but is still the best book that I know of to understand what SOA is really about.
D.Gross, « Fundamentals of queueing theory », Wiley (1998)
One cannot talk about IT performance without a minimal background on Queueing Theory. This is one of the good introduction books (there are many others).
F.A. Cummins, « Enterprise Integration: An Architecture for Enterprise Application and Systems Integration », Wiley (2002)
My favorite book about IT integration. A practical book that hits all the key topics and does not shy away from the hard problem. Only 10% of the books that talk about EAI, SOA or integration infrastructure are actually relevant to "real world usage", most of them are just re-hash of marketing slides (all the glory, no guts J). This is one of the precious few.
M. Tamer Ozsu, P. Valduriez, "Principles of Distributed Database Systems », Prentice Hall (1999)
My reference book on database systems. Although it is quite complete and covers most of the issues relevant to distributed systems, it is still an easy read.
E. Marcus, H. Stern, "Blueprints for High Availability», Wiley (2003)
Wonderful book about high availability. All you need to know, tons of practical advice and many examples. A must-read for anyone in IT operations.
K. Schmidt, "High Availability and Disaster Recovery: Concepts, Design, Implementation », Springer (2006)
More refined and detailed than the previous one, a great reference book on robustness and redundancy.
R. C. Seacord, "Modernizing Legacy Systems: Software Technologies, Engineering Processes, and Business Practices", Addison-Wesley (2003)
The only book that I know of that talks about "re-engineering of legacy systems" (what we call "refonte" in France) in a way that is consistent with my own experience of a CIO at Bouygues Telecom. All the hard issues are covered and the book is full of sound practical advice.

The next list is a reference list. These books are heavier, and are not meant to be read "in one shot". On the other hand, they contain "treasures of knowledge".

C. Jones, "Applied Software Measurement: Assuring Productivity and Quality », Mc Graw Hill (1996)
My "bible" during the last 10 years : all the hard numbers necessary to model software development costs and quality insurance. This is "the survival kit" for anyone who wants to introduce function points measurement.
J. Printz, "Architecture logicielle concevoir des applications adaptables » Dunod (2006)
This is a reference book on software architecture. It is very thorough, explaining all the hows with the whys. Very valuable to get a deep understanding on SE principles.
- Meinadier, "Ingénierie et intégration des systèmes», Hermes (1998)
  Still one of the best reference books about system engineering.
B. W. Boehm, "Software Cost Estimation with Cocomo II ». Prentice Hall (2000)
No one can afford to miss Cocomo II, since it is the scientific reference for almost all intuitions that one may develop after spending years of developing SW projects.
W. Perry, "Effective Methods for Software Testing », Wiley & Sons (1995)
560 pages that tell 90% of what one should know about software testing. I was fortunate to spend many years with people who had spent their live researching this topic at Bellcore, and find this book to be surprisingly complete and accurate.
D. A. Menasce, "Performance by Design: Computer Capacity Planning By Example", Prentice Hall (2004)
The best book that I have read about Performance modeling and capacity planning.

The last list contain fun books to read (at least from my perspective) which actually tell a lot about IT

M. Crichton, "Jurassic Park", Ballantine Books (1991)
This is still the best way I know to get acquainted with chaos theory and why it is relevant to understand IT failures J
C. Perrow, "Normal Accidents: Living with High Risk Technologies", Princeton University Press (1999).
A must read about industrial accidents (such as TMI: Three Miles Island) – The analysis and the proposed patterns are brilliant.
K. Kelly, "Out of Control: the New Biology of Machines, Social Systems and the Economic World", Perseud Books Group (1995)
My favorite book of all time, see earlier in this blog J
C. Hibbs, S. Jewett, M. Sullivan, "The Art of Lean Software Development", O Reilly (2008)
A wonderful very short books that contains one of the best introduction to lean I have ever read, and a wealth of advice about software development. Very concise but incredibly relevant.
F. Brooks, "The Mythical Man-Month: Essays on Software Engineering, 20th Anniversary Edition », Addison-Weslay Professional (1995)
Still relevant after so many years !
T. DeMarco, "Peopleware: Productive Projects and Teams (Second Edition) » Dorcet House (1999).
A treasure trove ! a non-nonsense inquiry into what it takes to be productive when writing software. The most incredible part is that the main contribution (such as the negative influence of disruption) are still unique to this day (to my knowledge)

Let me know about your own favorite IT books.

SOA is much too young to be dead

2009-01-24T22:14:00.000-08:00

Today I'll look into the SOA controversy that was generated by Anne Thomas Manes's SOA's obituary.

This may look like a slow response (90% of the blogosphere has already published their comments) but I will take some length to explain the issue (and this required a little time for thinking over). Surprisingly, one still get a very fuzzy and abstract idea about what SOA from reading all the positive answers to Ann Thomas Manes.

What is much clearer are the benefits one should expect from SOA (and they have been clear from 2003). SOA brings increased agility and reduced costs. The increase agility comes from reusability an the cost reduction comes from sharing services. Reusability comes in two flavors : direct (reuse of an existing service) and indirect=mash-up (quickly mixing existing services to produce a new one).

There are additional benefits (such as the ease to mix internal and outsourced services) but these two are enough to get anyone's attention. The question is: how to get there ?

1. SOA is a 3 step process

First, we decompose SOA into three steps ("doing SOA" means doing all three in sequence):

Service Definition
Service Architecture
Service Integration

Service Definition simply means to come up with the catalog of services. It is a "business oriented" step: the business processes are analyzed and the relevant business services are identified. At first it looks like the "simple" (nothing but simple actually) first part of a functional analysis of a new system. There is (much) more to it, because the role/user analysis is an integral part. The focus on reusability and agility starts here. Hence it is necessary to play with business scenarios, which further means that it is not an information system play. In the "acronym soup", the relevant terms are EA (Enterprise Architecture) and SOBA (Service Oriented Business Architecture) which have been introduced later on to insist on the "business/enterprise" dimension.

Service Architecture is the technical step that takes a large catalog as an input and produces a structured hierarchy as an output. A list of thousands of equally visible services is in no way a feasible approach to manage a large information system. Classical issues such as modularity, encapsulation, visibility, etc. must be taken into consideration. I write "classical" because this is the heart of software engineering and software architecture. Re-factoring techniques (the same that are used when writing a class or a component library) must be applied to produce services that have a better chance of being reused or being combined. I won't dwell into it today, but the role of data & models architecture is key (as I said, this step is the IS step).

Service Integration is what most people think about when they say SOA: linking all the services into an integration infrastructure. This is where all the technologic issues live (ESB or not, SOAP or REST, etc.). The term SOI (Service Oriented Integration - for instance, see this definition) was coined to denote this step.

These three steps help to grasp the huge difference between "SOA in the large" (with a large enterprise as its scope) and "SOA in the small" (SOA for one information system, either for a small company or one department). The effort repartition obviously varies, but here is the pattern from my own experience:

In the large: 40/30/30 (i.e., 40% of the effort is the definition part)
In the small: 20/10/70

It is interesting to notice that there is (almost) nothing new with "SOA in the small": service definition + architecture + flexible implementation is what has been making a good piece of software for 30 years.

2. What is easy/hard ?

Service Definition is hard when the scale gets big. Although producing the service catalog is straightforward for a department-size project, there are mutiple issues of heterogeneity and multiplicity of stakeholders that makes it much more complex to manage at the (large) entreprise level. Not only the management of the catalog gets harder as the size grows, but there is a subtle balance to be found between "common sense" and "formal methods". There are indeed methods/pattern/frameworks available for this first step but:

if this step is too formal, too many stakeholders are discouraged
if it is informal, the process crumble under its own weight.

The same is even truer with Service Architecture. It is quite simple when doing "SOA in the small" but gets harder as the size grows. Contrary to Service Definition, there is little help available as a methodology: hard learned IT experience is necessary. Unfortunately, this is not something that one learns through writing Powerpoints. If that step is poorly executed, the integration is more difficult, but - and this is the key point - little reusability or agility is observed later on. This is something that I am trying to teach during my "Information System Course" at Polytechnique, but this is a hard, technical topic which is better acquired through practice than theory.

Service Integration is actually not so difficult nowadays, because the technology is quite mature. I am not saying that it is easy, but it would be sad to ignore the progress that has been delivered by technology providers. There are still a few difficult issues (cf. next section) and, more generally, a large-scale integration project still exhibits all the possible pitfalls of a complex IT project, but there is nothing new here. Actually, when done "in the small", this is straightforward and there are so many success stories to prove it.

3. What is the maturity state of SOA ?

Using the same decomposition, here is my own assessment:

50% for Service Definition. This step is rather well explained, many good books are available (such as "SOA for profit"). It is still a difficult job at the enterprise level. There is indeed a governance issue (those who mock the "governance issue" have missed what Enterprise Architectuire is, and are still trying to accomplish "small scale SOI"). There is also a "pedagogy issue": an entreprise-scale effort cannot be undertaken if its meaning is not understood, by all stakeholders.
20% for Service Architecture. There are very few books that can help, and even fewer are directly relevant for SOA. The only one that I recommend to my students is "SOA, Le guide de l'architecte." (in French, sorry for my English speaking readers). I have tens of books about IT/software architecture in my private library, including great pieces from Fred Cummins, Len Bass, Richard Shuey, Paul Clements, Jean-Paul Meinadier, Jacques Printz, Robert Seacord, Roel Wieringa - to name a few -, but none of them (including my own two books :)) adress the issue of Service Architecture "in the large" thoroughly.
70% for Service Integration. The technology is there, it is scalable and proven (WebSphere or Weblogic being two classical commercial examples). Obviously, there are still technical issues. For instance,

distributed transaction & synchronisation is still a hard problem.
performance overheads (cf. SOA is not scale-free) still exist and makes the deployment tricky when response time is an issue.
monitoring & autonomic load balancing is difficult and/or not sufficiently developed.
service discovery and pub/sub architecture (EOA) are not as straightforward to implement as one might wish

But those are hard issues, nothing to be ashamed of :)

4. SOA is a practice, not a target or a technology

As David Linthicum said: "SOA is something you do, not something you buy". It is not something that you do once (a project) but that you do all the time, continuously.

This is what brought the creation of the sustainable IT architecture approach. Done right, SOA is a practice that transforms information systems for the better. According to the money that you invest, to the legacy and to the business constraints, the transformation is slower or faster, but it takes time and is never ended. The sustainable IT approach comes from a change of perspective: instead of always focusing on what is to come, on the new projects, one should search for more value from the "existing assets". Another "sustainable" principle is to save your strengths to last longer (hence the continuous improvement approach as oppose to the big bang).

It's a "culture thing". Part of it (cf. Service definition) is everybody's concern, hence it must be simple and must be pushed at the CxO level. Therefore, IT governance is a key component for changing the culture.

I will conclude by stating that SOA represents a "Chinese strategy" for IT management, in sense of François Julien. I could not urge you enough to read his book, but if you have not, a key principle is to distinguish between planned strategy (top-down, in the Greek fashion) and "building up your situation potential". A Chinese proverb says than one does not grow a plant faster by pulling its stem.

Similarly, instead of pushing "SOA technology projects", a better approach is to build up your "situation potential" : build a common data model, train everyone, build a shared consensus about what the service catalog should be, etc. Each project is then seen as an opportunity to move one step in the right direction.

Information Technology for the Chief Executive

2008-11-29T04:15:00.000-08:00

At last, the english edition of my second book is available. It is somehow simplified (no technical appendix and a shorter, leaner introduction) but the body is the same as "Performance du Système d'Information".

At the same time, I just received the "Best IT book award" from the AFISI. The AFISI is the French association for engineering and information systems. I am quite honored to receive this prize when I look at the list of the previous winners.

Since I have used a self-publishing company and performed the translation myself (with the help of a professional interpreter), trying to promote this book is going to be an adventure.

Very Cloudy Weather: will it rain on the « Cloud Computing » Parade?

2008-11-02T09:09:00.001-08:00

This blog has been very quiet for the last few months … and will remain as such until I am finished preparing my lecture about Information Systems for Polytechnique (2009). I should have, at that point, accumulated quite some material for future posts, since I am digging quite hard into the theoretical foundations of "Information Systems".

However, I have been reading so many newspaper articles about "cloud computing" during the last few weeks that I feel like putting a few thoughts on paper. The turning point was the article from "The Economist" last week: as is rightfully noticed, too much hype will likely cause "disillusion". I am actually totally convinced that "cloud computing" is relevant to many businesses and that it represents a revolution that is coming. I have actually explained some of this in my other blog. If you are new to this, a really simple description of the benefits could the following:

Fault-tolerance through the implicit redundancy. This is, however, an architectural issue. It is not enough to rent your computing infrastructure from Amazon, Google or Microsoft to get this benefit. You also need to implement your information system with an architectural paradigm which takes advantage from the availability of multiple servers.
Super-computing performance through parallelization. The same remark applies: it is true that new techniques for data mining or real-time event processing (two examples) may be tried successfully on the cloud through a MapReduce approach; it also requires a significant amount of work if you start from your legacy application.
Reduced costs of operation (TCO) through the use of standardized and mass-produced units. Look at the price of the TPMC according to the type of hardware and you will get the idea. I won't dwell on this, this is explained everywhere in the newspaper articles I was mentioning.

There is an implicit warning here: although "cloud computing" is "the way to go" for many cases/enterprises, there is a learning curve and a price to be paid. Especially, it will take time and energy to move from an existing architecture to one that can be "migrated on the cloud". However, "Cloud Computing" is not the "ultimate solution" for everything and I do not believe for a second that "software is dead". Let me give three simple reasons for which one may decide to stay on the "firm ground" as opposed to ""move to the cloud":

The risk of loosing privacy and control (cf. R. Stallman's reaction). There are many aspects, including legal and societal, to this issue. This is the part that is reasonably well covered in the papers (such as The Economist). It is clearly a valid point, but technology and service segmentation might alleviate this issue in the future. There may exist private clouds, secure clouds, encrypted clouds (where the encryption is managed by a third-party), etc.
Latency : accessing the cloud is not instantaneous. Even if the protocols were truly optimal (and they are not, web service invocation carries a significant overload), it takes some time to access to a distant data center (the light takes 10 ms to travel 3000km, and 10ms is significant for many high-performance computations or transactions). This is why MMORPGs rely on a RCA (rich client architecture – a significant part of the work is done locally).
Computational overhead: making each service invocation a web service invocation is not practical for high performance computing. There is a proper level of granularity for encapsulating a piece of computation/transaction into a cloud service. This is actually something for which there exists a significant amount of history: when companies try to develop a SOA architecture, they have to get the service granularity right. Otherwise, they "discover" that application servers cannot carry an infinite load and that they exhibit a performance overhead when compared with more traditional approaches (a RPC – remote procedure call- is more expensive than a procedure call, and a WS call is not the cheapest RPC).

Hence we get two negative answers to the following questions:

"Can I move all my IT onto the cloud without changing my apps and their architecture ?" (the cloud as the universal IT outsourcer). The answer is negative when performance (throughput or latency) constraints exist.
"Can I migrate all my apps towards a SOA architecture that will live in the cloud, made from SaaS (Software as a Service) ?". The answer is negative because some services are better produced locally. The economy of scale of the SaaS requires some form of mutualization (running a service remotely for a unique client is not a clear winner). If performance constraints prevent from breaking an application into "tiny components" (because of the cost of recomposition), many apps are too specific to be run as SaaS.

To summarize, I would propose two "axioms/theorems/conjecture" about "cloud computing":

"Cloud Computing" will succeed because of the combination of two facts:
1. The grid architecture is the best architecture to provide true scalability and superior availability (fault-tolerance)
2. Grid architecture exhibits "economy of scale", that is, can be run more efficiently when operated on a very large scale. The more identical servers one operates, the lower the TCO per server is.
"Service Oriented Architecture" (SOA) is not scale-free: there is a balance to be found between "recomposability" and performance. Hence the cloud may not host all the necessary services that any enterprise may need.

The first "theorem" could be labeled the "Google Theorem" since Google operations are an existence-proof of these two affirmations. They could be justified independently … and make perfect sense to anyone who has been in charge of running the IT department of a large-scale company. However, the same experience of being the CIO of a large company also suggests that performance and data synchronization/distribution issues will prevent quite a piece of the IS application portfolio to be moved to a cloud architecture, at least for the next few years.

Flexibility and Biology

2008-06-02T11:31:00.001-07:00

In a previous post I have listed flexibility as one of the four pillars of agility. I defined flexibility very loosely with the following sentence: " the technical ability to cover a large span of business situations and processes with the existing set of IT services and capabilities."

This requires a little bit of further explaining. Especially, there are two ways to obtain this ability which are both relevant:

The ability to implement change in order to achieve functional flexibility. This is foremost a matter of architecture. Hence it is the first thing that comes to mind, in the context of "SOA and Agility". This flexibility is expressed at the component level (the ability to parameterize: change the functional behavior though a change of parameters –cf. the 12^th chapter of my book on SOA), and at the "integration level", i.e. the ability to integrate a new component quickly and efficiently.
The ability to adapt to a new situation "without change", that is without having to stop and modify one or more components. This kind of flexibility is a combination of "meta-data architecture", that is, the ability to change the behavior of information systems through the change of parameters and the ability to do so dynamically. This is the most important part, and the one that is most often overlooked. In many cases, changing parameters is still a new project that requires testing, synchronization and going live with a new version.

We can summarize by defining flexibility as the combination of functional flexibility (ease of change) and operational flexibility ( at run-time, without change to the existing binaries).

The first kind may be labeled "mechanical flexibility". The analogy makes sense because when you modify a mechanical device, you must (most often) shut it down before changing or adding a new piece. Most of what you can read about IT agility is actually of this kind. New technology, rule-based engines, meta-data parsers are all designed to extend the functional flexibility. This is definitely true of integration infrastructure technology that I have come across during the last few years. There are a few noticeable exceptions (such as using UDDI for dynamic discovery of services) but … surprisingly … they have not made it to mainstream IT yet.

I will call the second kind "organic flexibility" for obvious reasons: changing the SI without shutting it down (without the "go live" step) reminds us of living organisms which grow, change and adapt themselves without having to halt their biological processes. The boundary is subtle: everyday running systems are modified by the input of their users. So what? It turns out that for the same systems (at least those that I known of) there also exists "hidden"/admin parameters that cannot be changed without stopping and restarting the system. It also turns out that most major changes require writing new code, to re-compile, re-link or re-connect, hence a new "go live".

I must confess that I have been trained for twenty years to think about flexibility in a mechanical way. However, I now believe that the flexibility that matters the most is the organic kind. This will become truer and truer as the stringent requirements of real-time digital business will demand no shut-downs of any kind.

Organic flexibility is a matter of organization, people and processes as much as of technology. However, there is a technological part: one may not achieve this behavior (easily) with legacy technology. A key is obviously the availability of dynamic meta-data that is interpreted as parameters and modifies the behavior in a deep way. The challenge starts when different components must share a change of parameters to achieve a new behavior. This means and requires that propagating new parameters becomes a IT run-time service. The technology for this type of propagation is there, once again we find rule-based engines, scripting engine, automation tools such as processflows. If I reflect on the IT projects that I have been a part of, 80% of the energy is spent making sure that there is enough flexibility ("enough parameters") to make the component "future-proof", and 20% is spent thinking about the processes that are used to change these parameters. Designing an organic information system requires to change the ratio to the opposite.

Software Development Productivity Gains ?

2008-05-08T03:34:00.000-07:00

As I am close to getting my last book published, I figured I should publish a few pages once in a while to trigger the interest and to gather a few comments.

This first extract deals with the matter of productivity gains in the information system. This is a complex topic, and it is really the whole of this book that provides an answer to it. We will merely highlight the most important points and indicate the chapters where they are described in greater detail. In this initial overview we will distinguish the project aspect, dealt with in this section, from the operations aspect, dealt with in the following section. These can then be broken down into three major parts: technology, industry and architecture.
Since the dawn of IT, every year has seen the advent of new technologies that promise spectacular gains in productivity[1]. In the past, people talked about programming languages with a high level of abstraction, artificial intelligence, computer assisted graphic design, synthesis and automatic program checks, etc. Today, the hot topics in the literature consist of meta-programming, service architecture, component assembly, to name but a few. We will go back over these technologies and their promises in chapter 9. There is certainly progress, in terms of function points per dollar and in terms of function points per man-day, but this progress is coming at a very slow rate. If we had to provide an order of magnitude based on the comparative analysis of the productivity figures from the late 80s, and those that are found today, I would say that we have gained a factor 2 in 20 years.[2]. It is a matter of overall productivity (on the project perimeter) according to the technical axis (in FP/man-day). The economic aspect is different, because the IT industry has undergone significant changes.

The software and system integration industry brings with it structural gains in productivity according to the following factors:

Globalization, which makes it possible to move a part of production into countries where there is a lower cost of labor. This delocalization produces a substantial economic advantage, when it is relevant, particularly for stable needs. We shall come back to this in chapter 9.
Mutualization of needs, which leads to the continuous release of software packages that capitalize on the needs shared by different companies. The associated gains may be quite sizeable at the level of a project, but are only moderate at the level of the average of the ISD.
Maturing and professionalization of development practices in relation to the development of tools. The continued formalization and optimization of development processes by using standardized repositories as support[3], is a good practice that is vital for an ISD and its providers.

These productivity factors may have greater effects than technology factors, but their implementation is not easy, and requires discernment in terms of the scope of application.
The third area of progress is the information system architecture, which we distinguish from the technology section, because it involves what is known as “enterprise architecture”, a subject which spans information technology, business processes and “strategic alignment” of the information system. Enterprise architecture is used as a complexity control strategy so that the integration costs of a new application do not reach an irreversible level as the IS grows. This is mostly a defensive strategy that is used when the complexity is precisely such that integration becomes the predominant part of IT projects. As a result, the effect that is noticed is not a “productivity miracle” but the return to a “comfort zone” where integration costs are proportional to the functional complexity of the application.
The conclusion we can make at this stage of the analysis is that there are undisputed productivity factors in terms of information system development but these are not necessarily easily deployed; the gains remain measured.

The visible part of these gains is all the more modest that another part is consumed by three heavy trends of enterprise IT[4]:

Additional requirements, as regards availability, security, ergonomics, etc., which induces more complex IT solutions with an equivalent functional perimeter.
Inter-operability requirements, whether within the company or in conjunction with partner companies. Enterprise IT has been experiencing a double revolution for 20 years. Firstly, its functional scope grew along with inter-application connectivity requirements (unlike departmental IT in the 70s and 80s). Secondly, the concept of the extended company arose, i.e. the necessity to connect the information systems of companies that are collaborating on processes.
The abstraction of software functions to control complexity. Controlling complexity leads to the creation of layered software, with the layers corresponding to different levels of abstraction, and makes it possible to have them evolve without having to get a sense of them as a whole[5]. These methods make it possible to control application systems that are very rich and complex but generate additional costs (the actual size exceeds the expected size).

Users of general-public software such as office suite tools will have noticed that more and more resources are required to run their favorite applications (e.g. word processing), mostly due to these three reasons.

[1] We have already referred to F. Brooks’ article entitled “No Silver Bullet: Essence and Accidents of Software Engineering”, which examines the intrinsic difficulties in terms of software productivity. This article can be found in T. DeMarco and T. Lister’s Software State-of-the-art, which includes a selection of the best articles from the late 80s.
[2] This assessment is an average ratio which hides a great level of variability, according to the type of problems to be dealt with. This will become clearer when inter-company benchmark test results are published every year.
[3] The repository which is gradually becoming the standard one in IT project development is CMMI (Capability Maturity Model Integration). For more on CMMI, see the first section of M.B. Chrissis, M. Konrad and S. Shrum’s CMMI – Guidelines for Process Integration and Product Improvement. You can also find many sources online, including those of the SEI (Software Engineering Institute).
[4] This phenomenon is more commonly known in technology circles as the “rebound effect”. For example, the efficiency increase for heating a square foot of living space is offset by the increase in the average house size; the progress in car fuel consumption has helped to develop the usage, instead of simply generating savings, etc. In IT, the “complexity race” absorbs a proportion of the gains by the rebound effect.
[5] The concept of “layers” is hard for non-computer specialists to grasp. It is quite relevant to compare it to geology: layers corresponding to age, with a successive deposit of “functions”. For example, I have the experience of precisely analyzing the code of a major American publisher on typical functions whose performances seemed to be disappointing. Using the debugger as an analysis tool showed several “layers” which have been built up as the software has undergone different changes (from an initial effective kernel to multiple generic interfaces). This concept of “sedimentation” will be introduced in section 2.4.1.

Agility, Biology and SOA

2008-04-13T06:00:00.001-07:00

I have just finished writing a new chapter for the third edition of my book « Entreprise Architecture & Business Process Management » (which is only published in French, unfortunately). This new chapter deals with "SOA and Agility", two buzzwords of these IT times.

A part of this chapter deals with the concept of "sustainable architecture" following what I wrote in my previous post. I have proposed to paraphrase the definition from the Brundtland report:

"Sustainable development is development that meets the needs of the present without compromising the ability of future generations to meet their own needs."

into a definition of that spells out the "sustainable development" of the IT architecture:

"Sustainable architecture is the Entreprise Architecture that enables the development of today's IT services without compromising the ability to develop tomorrow's services because of an ever-increasing complexity or the induced scarcity of resources (money or skills)."

To gain this sustainability, I have been advocating for quite some while for "fractal" architectures (hence, "fractal" methods). I mean methods that can be broken into pieces and applied differently (and at different rates) to different situations. This fractal view applies to middleware, to enterprise architecture, but also to data models or to process models. The information system (as a whole) needs to grow (as a living object) according to different rates of pressure that are applied to its different parts. The CEISAR architecture documents make a very elegant and clear point about the value that exists when breaking long "end-to-end" business processes into smaller, more manageable pieces.

The major part of this new chapter is devoted to agility (mostly because I hear too much nonsense about solving the agility issues with technology). To give a short summary, I see four forces that contribute to agility:

Anticipation: being agile is implementing changes rapidly enough to meet TTM (time-to-market) constraints. A smart way to think about it is to start early. It is not a "solution in itself" (since many changes may not be anticipated) but it is definitely part of the "whole package" since many "smart decisions" (the one that will increase agility) take time to get implemented.
Flexibility: the technical ability to cover a large span of business situations and processes with the existing set of IT services and capabilities.
Leanness: the process of applying changes to the information system (for instance, developing a new IT project) needs to be as lean as possible (cf. my other blog for more details, although I will most certainly be back on this topic). Lean obviously means short from an organizational perspective (the longer the chain between the developer and the problem owner, the less agility is observed). The ultimate goal is to see IT project disappear. The "absolutely agile" IT department is the one who is not needed to let the users adapt their information system to the business changes. Although it may sound silly, it is actually a sound goal, but one that requires a lot of anticipation.
Skills: one needs to be competent to be agile! This also sounds dumb, but actually it is a rather profound observation, which is backed by observation. The fear of failure and the pressure in today's modern organizations means that any lack of full understanding or full confidence will translate into multiple redundant checks, back-and-forth, "safety precautions", etc. Another dimension is the ability to profit from the opportunity provided with the constant progress of technologies (from new IHM techniques (Ajax, mash-up, etc.) to new middleware tools, including more radical options to leverage the value in SaaS – software as a service).

Notice that, except for the second one, these are organizational traits and not technical ones. This, by the way, is also true of being able to leverage the value of SaaS. The software services provided by SaaS are not especially great. Using them is smart from a business and an organizational perspective, and requires "business agility/flexibility" even more than technical skills or abilities.

As I wrote earlier, SOA's main contribution is to provide a framework for sharing and reusing services. It also helps to achieve agility, but more as a second-order consequence rather than a direct cause-effect path, and not without efforts and dedication. The first-order benefits of reuse and share are cost and complexity reduction. On the other hand, as I have already explained earlier, SOA is foremost a governance principle, before being an architecture framework. If the stakeholders do not play "the game", the existence of a beautiful ESB (enterprise service bus) will not provide as much benefits as expected. As a governance principle, SOA sets principles and rules that must be followed.

If this sounds very different from a crude idea of what a "biology-inspired" IS could be … it is indeed! The strength of biomimecry as a design principle is found in what nature does very well:

Flexibility/ adaptability (to new situations)
Reliability / Survival ability

One thing that nature does not do so well is the "economy principle", the "simplicity of design". When biologists study mechanisms in living objects, they are often surprised by the level of redundancy, by the complexity of the processes (which have been obtained through evolution). By the way, when we look at legacy code, we often get the same impression J …

There are two consequences:

Biomimecry is not enough as a design principle. The economic reality of business requires implementing a "principle of simplicity" and a constant strive to reduce costs. I advocate for a "fractal architecture", which may sound like an oxymoron to some, as a weakened form of the necessary dictatorship of principles and rules. Rules are necessary to produce simplicity.
For all the good reasons mentioned in earlier posts, it is a good idea to borrow from Nature to design systems which are at the same time: complex, reliable, flexible (doing any of the two is OK, doing the three is a challenge). However, this will come at a cost (mostly redundancy). To justify the effort, one must look at the "big picture" and consider full "life cycles" of IT services and take multiple business scenarios into account. This means that the projects that will bring biomimecry into the IS architecture will not have a simple ROI justification, but rather a sophisticated one. This is discussed in my book, which should be made available in English (at last) in a few months.

Nature is organized around a simple principle that says that evolution cannot be stopped, which also applies to information systems. Hence the only mechanism to cope with complexity is to clean, to remove unnecessary things. Nature does great job of cleaning (and recycling) unnecessary things in living organisms. This is another aspect which may be borrowed, especially when managing a SOA. A key issue is to master the complexity (and the sheer number) of services. The book will mention a few tools and techniques to manage this catalog of shared services, but the first principle is to limit its size. Since evolution cannot be stopped, old services need to be removed.

This blog is moving very slowly (although one would think that there are enough provoking statements in this message to generate a few comments). I actually thought that it was dead, but there are a few readers coming everyday (that's what the stats tools say). I may accelerate the posting rhythm in a few months, when I'll be preparing a new course on the "Theory of Information System".

Sustainable Enterprise Architecture

2007-10-20T10:14:00.001-07:00

After five years spent as a CIO, I have developed the following conviction:
[1] Major architecture re-engineering projects are fuelled by pain (one could say : no pain, no gain :)).

Enterprise Architecture (EA) projects (I should rather say programs, as in a family of projects) require an unusual amount of effort and alignment throughout a long period of time. The alignment is even more difficult to achieve than the sustained level of effort, especially with a large IT organization. Alignment here means the fact that a large group of people decide to make sub-optimal choices, from their own viewpoint, to achieve a larger-scope goal. It may be strange way to look at alignment but it makes sense: if the logical choice for each actor was to move towards the same direction, there would be nothing to talk about :)

Introducing an EA scheme (what we French call "urbaniser" the information system) usually occurs because a blazing limitation of the IS has been found. It is too slow, not flexible or agile enough, not reliable enough and, most often, too expensive, etc. The level of pain is necessary to break through a "decision/action" threshold, since there are obvious risks. From a technology perspective, an EA relies on an integration infrastructure (ESB, EAI, ETL, and so on). From a culture perspective, new concepts and a new vocabulary are introduced.

What happens if the program is successful? After a while (it could be a long while :)) the pain recedes. As a consequence the alignment starts to weaken. This is not simply an internal/organizational issue for the IT department. Deploying an EA approach is a corporate endeavour, which requires from all business division a common wish to build a global system. The alignment here means that each division is ready to relinquish some of its interests for the common good.

Because of this weakening of the common resolve, the master plan becomes too heavy to carry and one returns to (some of) the previous faulty behaviors that caused the pain in the first place… I have been thinking about this for the last three years and I have come to build a second conviction:

[2] Service Oriented Architecture is the sustainable approach to develop a "well-architectured" information system over time.

This statement probably sounds lame and dull to anyone familiar with the SOA (Service Oriented Architecture) concept. All possible "claims to fame" have already been made when SOA is concerned :). I should first state a few caveat:

I do not mean SOA as a technical architecture framework (Web Services, ESB, …) but as a governance method for building reusable information system assets. I do not want to dwell on this today, I'll come back to it some other time (one may look at the discussion about the SOBA acronym to grasp this ambiguity).
I do not mean a unique, shared, common architecture. As I explained in my first book, I am a strong believer in diversity. Actually, one of the first papers to talk about "sustainable enterprise architecture", from Marten Schoenherr and Stephan Aier, precisely considered sustainability a benefit of a distributed approach. I refer my customary readers to page 98 of my book where I also develop this idea (e.g., the main benefit of SOA compared to earlier approach, such as EAI, is the ability to decentralize the EA program).
I really push the analogy with "sustainable development" to the core: a sustainable EA approach is one that produced benefits without requiring so many efforts from the culture, the people, the organization that it stops whenever the actors have the freedom to do so. This is really about people, and especially the relationship between business process owners and their IT providers.
I am discussing about a large-scale enterprise and its information system as a whole. I consider the problem of "SOA at a departmental scale" solved (this is illustrated by the existence of so many successful implementations …). The sustainable alignment of a medium-size information system is not such a formidable task :)

Hopefully my second book will be available soon to English-speaking readers (since I have finished the translation). They will see that one of my central theme is that a "well designed IS" is a corporate responsibility, not something that may be left to the CIO. The CIO may take the leadership for a "special re-engineering" program/effort, but this cannot last. Eventually it is a matter of management culture (unless the CIO wants to become a "dictator" but he or she usually gets fired quickly if this temptation is too strong :)).

Pierre Bonnet recently opened a web site about this very topic: http://www.sustainableitarchitecture.com/home. His book about the same topic will be out next month. The web site is really interesting, together with the companion site about the Praxeme method. If you go and read through it (which I encourage you to do :)), you may think that it develops a similar line of ideas (obviously, with more details and more thought-though principles). I actually agree with everything … but I do not think that it reflects truly what sustainable development is really about: people. I am personally a big fan of the ACMS approach (Agility Chain Management System). Unfortunately, (or fortunately, since no so many people may understand what it :)), this is not where I see the issue for deploying a SOA Enterprise Architecture in a sustainable way. There (rightfully so) a lot of talk about SOA governance nowadays. Unfortunately it remains complex and abstract, whereas the issue is the appropriation from all stakeholders in the company. I will return to this topic in further postings, since I believe that this (SOA governance) is the key to agility. I fear for those who will promise agility from the sole technical merits of a SOA architecture.

It turns out that there is a totally different meaning for "sustainable IT architecture" ! If one looks at the electric consumption of a data center, it is raising dangerously over the years (with respect to the double issue of energy price increase and greenhouse gas emissions). Electric consumption here includes both the powering of the computers and their cooling. Both tend to be proportional to the square of the processor frequency (one way to look at it, although the resistance decreases with the smaller scale designs). Both tend equally to be proportional to the amount of computation that is made, which is clearly growing fast in most companies.

This is why Google is seeing energy consumption as a key issue. For instance, read this newspaper article to find out about Urs Hoelzle approach to reduce server electricity footprint (more technically-savvy readers may look at this :)). Since then, Hoelzle has said Google is looking into neutralizing its carbon emissions by the end of the year.

The link with architecture is as follows. The simplest way to increase the computing power without increasing the consumption is to use massive parallelism. I don't have time to go into details today. One may look at the StorageMojo web site to get a lot of interesting stuff.

I have just finished Ray Kurzweil book "the Singularity is near" (2005). As usual, this is a fascinating book, especially from this "sustainable development of IT" perspective. From a general perspective, it is a refreshing view from a "technology optimist" which offers a clear break from the prophets of doom. As far as computing is concerned, Kay Rurzweil offers hope for software designers to be able to use much, much faster hardware (although, if you read the book, you'll see that the name may no longer be appropriate :)), something that I am dreaming off each time I run of my "game theory simulation" :)

Ray Kurzweil's optimism does not cancel the validity of Google concerns (different time scale). One might say, then, that a sustainable architecture needs to run on a grid-like structure (or any other form of massively parallel system architecture).

Lean Information Systems

2007-10-11T21:25:00.000-07:00

Lean Manufacturing is a powerful concept, which is often misunderstood. It was made popular by Toyota’s implementation and Taiichi Ohno’s vision (one of Toyota charismatic leaders). A very simple way to explain what it is would be to compare two production shops:

one shop is organized so that each machine is run at optimal capacity, in its best operating conditions. Buffers are introduced and the transport between machines is a little longer (so set up the machine optimally)

the second shop is organized so that the flow is shortened as much as possible. Buffers are reduced (and eliminated as much as possible) and the transport is optimized. The consequence is that each machine is no longer working optimally. Some are underutilized and others are working in operations mode that do not yield the best productivity.

What does Lean Manufacturing (and experience) say ? Obviously the first shop costs less to operate (cost for producing one unit) on paper, but unless it operates in a ideal world with no variations at all, it actually costs more in real life. The second approch costs less from an inventory perspective, but mostly it is more flexible (with respect to priority changes) and more robust (with respect to load variations).

Let us now consider two information systems, from our scope of large-scale, distributed information systems (many parallel nodes running business processes) :

The first one has been designed so that each node is running close to its optimal capacity. A node here may be a group (cluster, farm, blade) of servers that run services which are the elementary components of the business processes. The computing power of the node is designed so that the node is running at 85% capacity when the load is full (i.e. when the business processes are running at their maximal expected load).

The second one has been designed to speed up the process running and to avoid "queuing waiting time". Hence the computing power of the node is adjusted so that the average utilization ratio is closer to 50%

Here also, the first data center is clearly cheaper to build than the first one. The second one has a few advantages: better SLA (service level agreement) may be promised to the customer (tighter = faster garanteed response time), and the upgrade process (when the company grows) may be planned in a more regular way) ... but let us assume that these are not compelling advantages. That is, let us suppose that the customer accepts the two different SLA:

in the first case, the SLA is such that the target response time will be obtained in 98% of the time with regular business conditions.

in the second case, the SLA is also such that the response time will be obtained in 98% of the time (hence, a smaller number than the first one).

I have made some interesting computing experiments last month to see how these two data centers would behave when "a little stress occurs". Stres here may come from one node unavailability, from a process overload, or from a higher-than-usual variation in the processing load. Anyone who has any experience with operations will recognize that these are the common issues of day-to-day production life.

These experience were reported in a talk that I gave at the "Colloque d'Automne du LIX", from which I have extracted the last slide:

You may find the complete presentation on the CAL web site. To keep things simple, the curves describe the behavior of the systems (1) and (2), with different stress scenario. The different curves correspond to different strategies of "adaptive middleware" (recall that I have this interest for autonomic computing :)). What matters here is that the lower curve reproduces the strategy that ALL existing systems use today (first-come, first served). What you may see is a tremendous difference:

The lean IS (on the left) does actually very well under stress. Only the loss of a node creates a real problem (and it is not major, the SLA drops to 75%)
The loose IS (on the right) is definitely not robust. The stess conditions cause a significant drop of the SLA (down to 20% !).

There is another way to say it: if your IS is run in such a way that message queues are often full of pending requests, setting up a proper SLA is a very difficult job, because predicting the behaviour (response time) of an overloaded queing system is hard science. It is not enough to add reasonamble margins (such as, promise a 10 minutes response time because the average processing time is 1 minute).

There is nothing new here. This experiment confirms what experience or intuition shows. What is interesting (and what surprised me) is the HUGE difference that the computing experiment reveals.

I plan to do similar experiments within the (global) entreprise context. I need a model that links the behavior of the IS with that of the company itself. Fortunately, I can rely on the great work (and models) just released by the CEISAR.

The CEISAR is a French initiative, under the patronage of the Ecole Centrale, to create a repository of models and practical knowledge about Enterprise Architecture. A first gem is their global model (follow "main concepts" then "Core Business System" on their web site), an attempt to define Enterprise Architecture with 10 key concepts. Another extremely useful piece that is part of the first release is a document about entity modeling. In one of my books I complained that this type of knowledge was not accessible (and could only be obtained from experience). It is nice to see real-life-experts, such as Jean-René Lyon, share their knowledge about such topics.

I definitely plan to adhere to CEISAR's terminology and framework for my own future work about IS architecture. One of the most pressing issue (as I have already testified on this blog) is to build a framework/model to explain, discuss, simulate data distribution and synchonization protocol. The only way to make this a relevant topic is to keep a very broad perspective, that includes a model of the coupling between IS and business. The nice conclusion is that this type of work falls neatly between my two topics of interest (cf. my other blog) : IS efficiency and Enterprise efficiency.

Long term Research Agenda

2007-06-24T00:02:00.000-07:00

I have published my research agenda on my web site.

I will resume my work on OAI this summer to prepare a lecture that I need to give in October. However, my first priority is to translate my book into English.

I am currently finishing my computational experiments on Social Networks. Contrary to what I have posted on the first message of this blog, there is a fair amount of common grounds between my two research topics. Mostly, the importance of information propagation ... how is it relevant to business performance, and how does information techhnology help to achieve it.

The idea that fast is better, that reactivity is a crucial quality ... is everywhere. However, as the study of LeanSixSigma from an operations research's viewpoint shows, this is not so obvious. Lean-ness, Speed, Reactivity come at a price. It is easy to be convinced that the price is small compared to the benefits, but this argument is rarely heard.

This is a thread of thought that I will follow ... It is related both to my old job about Information Systems Efficiency and to my new job of VP in charge of business process optimization and total quality management.

Complex Systems and Autonomy

2007-05-05T04:02:00.000-07:00

This blog is “moving” very slowly because I am currently much more involved with the theme “Information flow and Enterprise architecture”. I am currently working on Social Networks, and it turns out that there are a number of themes which are equally relevant to my two research interests.
For instance:

I will soon post a review of Robert Axelrod’s book “The Complexity of Cooperation”, which could appear in both my blogs. This book introduces a few key concepts, such as the use of game theory and genetic algorithms to study patterns of (stable) cooperation.
I attended a few sessions of the conference on “Complex Systems”, which I thought was appropriate considering that DIS (Distributed Information Systems) are indeed complex systems, and it turns out that social networks, including those that represent the information flows within a company, are the focus of “complex system approaches” as well. The 7th PCRD has made “socially intelligent IT” a hot priority.

My program nowadays is mostly to continue my education and read books. I will give a lecture in October for which I would like to extend the OAI research I have already talked about. More precisely, here are the two directions that I will pursue this summer when I resume my experimentation work:

Explore different scales and designs of systems to see what kind of influence they have on the behaviour of “smart” routing rules for process control (through message passing). Most of the experiences that I have made so fare try to reproduce business processes from Bouygues Telecom. Since I am using different “Enterprise Simulation Scenarios” in my SIFOA experiments, I plan to reuse this modelling effort. By scale & design, I mostly mean the number of processes, the number of tasks and systems involved in a process, and the interaction topology induced by the SOA architecture.
Try to model the handling of exceptions, that is, to model the alternate processing path which is used once a component is unavailable to deliver a business process. The difficulty here is that it is often an ad hoc approach (cf. my papers and my books). On the other hand, with the current trend of virtualization, more systematic alternate approaches will become available. I attended an interesting lecture on Autonomic computing from IBM which made the obvious-but-profound point that there is no autonomy without choice, and that choice comes (mostly, in the world of IT) from virtualization. There is little room for improving the handling of exceptional situations and failures with autonomic computing if a traditional architecture (hard links between dedicated resources) is used. On the other hand, in a virtualized world, there are many interesting options (hence a choice) when a server becomes unavailable.

This means that I am currently focusing on the points (1) and (2) of my previous list.
Following the suggestion from Cedric Nicolas, I just read “the age of spiritual machines” from Ray Kurzweil. It is a fascinating book, with very insightful thoughts, especially about consciousness and intelligence in a machine (or a network of). I am less enthusiastic about his prediction of the future (2009-2019-2099), but it is only a fourth of the book and the first three quarters are enlightening. I will return to the topic of consciousness when I post a review of Kevin Kelly’s book. I also finished a book from Jean-Claude Ameiesen, “La sculture du vivant” (following a suggestion from Pierre Haren) which is totally different (a biology book about programmed self-destruction of cells) but equally fascinating. This type of biology book makes it even more convincing that large scale man-made systems must draw their inspiration from living organisms. There are a number of mechanisms designed to yield both stability and flexibility/adaptability (somehow antagonistic) which are worth reproducing.

Five Challenges for Entreprise Architectures

2007-02-11T05:33:00.000-08:00

I have defined previously the subject of this blog in a very general manner, trying to look at the “big picture” (almost from an epistemological point of view). The “big question” is : how much autonomy and “intelligence” needs to be fed into an information system in order to achieve the properties that are expected - resilience, performance, reactivity … and so forth. The reasoning is as follows. Large-scale information systems are, truly, complex systems which exhibit all the classical properties of such systems : fractal/recursive structure, non-linear behaviour when they receive an incoming flow of event (cf. the interesting amplification loops of acknowledge/resend messages due to fault-tolerant mechanisms), “emergence” of large-scale properties that are different from those at the component scales, etc. This is why I will use this blog as a forum to relate my (slow and progressive) journey into the world of complex systems and their theory.

Today I will look at the topic from the other side, taking my former “CIO hat”. I will describe five architectural/design challenges that face a modern (large) enterprise. The following is a list of the issues that I have been struggling with during my five-years tenure, and which, I believe, are relevant to many companies with similar scale (size of IS) and scope (role of IS within the business processes). This is not to say that these are “open problems”. Indeed, we found solutions to each of those, as did other corporations. On the other hand, no definite answer has been proposed yet :

these are ad hoc solutions and they leave some questions un-answered,
the state-of-the-art, as defined in books or architectural frameworks that are sold by consultant, is surprisingly shy on these topics,
the academic research is only touching the “surface” of these problems.

These are very practical questions, one of the goal of my research is to build a link with the more forward-looking / science fiction / philosophical discussion on autonomic and intelligent systems. As a matter of fact, one of my long term goal is to address these issues in my computational experiments, such as those I made for the OAI research :

Self-adaptive middleware: Supporting business process priorities and service level agreements
Advanced Engineering Informatics, Volume 19, Issue 3, July 2005, Pages 199-211

Here is a short description of each of them, I will return with a more detailed description on some other occasion.

(1) How does one guarantee the Quality of Service, defined at the business process level, from the system-level properties of the individual components? This is a key a challenge for any IS architecture framework, including service-oriented architecture (SOA). I have coined the OAI term to describe this issue and a simple description may be found on my web site (http://claire3.free.fr/yves_research.htm), while a more detailed presentation is included in the afore-mentioned AEI paper. This is clearly an open-ended issue, my preliminary work has led to more questions than answers.

(2) What is the proper approach to achieve resilience (lowest possible impact of system-scale failures of one or many components) ? This is where the biology comes in (cf. my previous message) : the mechanical view of robustness through redundancy (multiple copies and spare parts) shows its limit in the real world, and, most often, real life crisis are resolved through alternate scenarios (an “organic approach”). It turns out that there already exists a number of alternate approaches: an older system that has not been disconnected yet, a simplification of the business process that is still acceptable, a different component that may render a simplified version of the service, a raw computer utility/patch/batch that fixes 80% of the problem (measured in dollars) with a fraction of the effort (measured in function points), etc.

(3) What kind of data architecture is best suited to distribute business objects in a coherent and efficient way ? Business objects need to be distributed in a large-scale system for performance reasons, but they participate to business process executions, which require some kind of coherence. This leaves roughly three options:

Assume a separate mechanism that will ensure the coherence of the distributed objects (precisely, a distributed database system management system - DDBMS J). On a small-scale system, or with a homogeneous system, this is obviously the most logical approach. We sub-contract this coherence issue to another system, and run the business processes assuming that distributed objects are coherent. It turns out to be difficult (the so-called “snapshot problem” of DDBMS cannot be solved in its full generality) and quickly unpractical once one builds an information system out of COTS (commercial, of the the shelf) software components (heterogeneous).

Take responsibility of “business object distribution & coherence” as part as the business process management. In other words, ensure that the business process flow pushes all relevant updates so that it guarantees that, as far as the execution of the process is concerned, the objects always are in a coherent state. The synchronization of business process events and object management events is, indeed, a tricky issue.

Define an acceptable level of “chaos”, that is accept that complete coherence is not necessary ! This is actually a generalization of the previous approach, and is closer to reality (real large-scale systems survive with a fair amount of non-synchronization).

Finding a proper approach that is robust to errors, (i.e., contains a proper long running transaction mechanism) is, as far as I can see, both a truly practical question (this problem is everywhere, although many fail to recognize it) and a difficult one.

(4) What is the nature of service-level contracts which will yield the ability to evolve in a flexible and distributed manner? This is the quest for “upper compatibility” when designing service interface (for instance in a SOA approach) which should enable a truly distributed evolution of the IS as a whole. My experience is that this is a difficult topic, and that large IS projects are often necessary when one components is upgraded from one version to another – a lot of interface work and a lot of non-regression testing.

(5) How does one achieve a modular architecture that still takes advantage of the opportunity to share and reuse services? In the world of SOA, this translates into the definition a common, global directory of shared services. There is an obvious (and classical) tension between distribution and centralization. A common shared service is the opportunity to reduce cost, reduce size (hence complexity) and ensure more coherence. On the other hand, it may render the evolution of the system more difficult, and certainly requires mechanisms to operate the community of stakeholders (who use the service), to govern the roadmap of the shared component. It is also essential (as for large-scale software) to introduce some level of abstraction and encapsulation, which prevents a unique, common and shared directory of all services. The structure that defines a distributed directory of services with its proper governance mechanism (with visibility and decision rules) is yet to be defined to implement an enterprise-wide service-oriented architecture.

Since one of my current goals is to draw as much relevant information as possible from the literature on complex systems, I will post regularly my findings and try to explain how they related to these five questions.

Strong AI and Information Systems

2006-12-17T06:24:00.000-08:00

The following is a translation of a discussion with three students from ESIEE: Philippe Glories, Erwan Le Guennec, Yoann Champeil.

- According to you, where mostly does one finds IA in everyday’s life ?

There are two types of answers, according to the scale of systems. Large scale systems require distributed AI (multi-agents, etc.) , whereas smaller scale systems may rely on the many technologies that have been developed so far for “local use” (such as expert systems, constraints, knowledge bases and inductive or case-based reasoning, fuzzy logic, etc.). Actually AI is already everywhere ! In picture cameras, cars, central heating, … and equally in existing applications from current Information Systems. As far as embedded systems are concerned, fuzzy logic (and neural nets) are heavily used for intelligent control. Information Systems applications make use of rule-based systems (sales rules, monitoring rules, and so on).

Large-scale applications of distributed AI are scarcer, as far as I know.

- May one talk about “strong AI” (as opposed to “weak AI”, see http://en.wikipedia.org/wiki/Strong_AI) to deal with autonomy ? (The autonomy would require self-awareness to manage oneself)

This is both a possible and useful distinction, although it is difficult to manage because of the continuous nature of autonomy. However, one may distinguish between AI based on applying rules to a situation and AI that uses a model of the problem it tries to solve, together with a model of its own action capabilities (a meta-representation of oneself being a first step towards consciousness), to that it may adapt and devise different appropriate reactions.

I would present a different type of distinction: made (or “built”) AI vs emergent (or “born”) AI. In the first case, a piece of software produces (intelligent) solutions that are predictable as a consequence from the original design. In the second case, the nature of solutions is harder to foresee, they emerge from the components together with the reflexive nature of the application (meta-model). It is another way to look at this weak/strong difference.

- Is the creation of autonomous AI already feasible ? Do we hold the technical means ? If missing, are we about to get them in a few years ? Are there any special theories that are requires to develop these ?

I am no longer enough of an expert to answer this question with any form of authority. I believe that the answer is positive, in the sense that we have all that we need, from a technology standpoint, to create autonomous AI. It is mostly a knowledge representation issue.

- Do you feel that the creation of autonomous AI is advisable and desirable ? From an industrial perspective? From a society perspective? From a scientific perspective ?

This is a large and difficult question !

I would answer positively, since I believe that only a strong AI approach will enable us to break the complexity barrier and to attack distributed AI problems. This is especially true for Information Systems issues, but I believe this to hold for a more general class of problems. To say it differently, solving successfully distributes problems may require to relinquish explicit control and to adopt an autonomous strategy (this is obviously the topic of this blog and of Kelly’s book).

There are associated risks, but one may hope that a rigorous definition of the meta-model, together with some form of certification, may help to master those risks.

Obviously, one of he risks, both from an industrial or social perspective, it to see the emergence of systems with “too much autonomy”. As a consequence, a research field that needs to be investigated is the qualification of “degrees of freedom” that are granted to autonomous systems. A precise answer with collide with classical indecidability problems, however abstract and “meta” answers may be reachable.

- From a philosophical point of view, do you see autonomous artificial intelligence as a threat to mankind ?

No, from a philosophical point of view, autonomous AI is an opportunity. There is a danger, however, from both an ethical and practical standpoint. Practically, the abuse of autonomy without control may have negative consequences. From an ethical point of view, there is a potential impact on society and work economy, as the delicate balance between production and consumption roles may be affected (which is true, by the way, for any method of automation).

- To summarize, would you qualify yourself as an opponent or an advocate of autonomous AI ?

Without a doubt, I see myself as a proponent of AI ! The reasons are, in part, those expressed in this blog: autonomous AI is the only approach to resolve complex problems, for which a solution is really needed. I see delivering the appropriate level of quality of service in an information system an example of such a worthy cause :)

A last remark : the scale issue is really key here. The same rules should not apply …

(1) On the small scale, components should be built using a “mechanical vision”, with proper specifications, (automated) testing and industrial quality using rigorous methods. When “intelligent” behaviour is needed, classical AI techniques such as rules or constraints should be used, for which the “behavioural space” may be inferred. Although this is just an intuition, I suspect that components should come with a certification of what they can and cannot do.

(2) On the other hand, large-scale systems, made of a distributed network with many components, should be assembled with “biomimetics” technology, where the overall behaviour will emerge, as opposed to be designed. My intuition is that declarative, or policy-based, assembly rules should be used so that a “overall behavioural space” may be defined and preserved (which is why we need certified components to start with). The issue here is “intelligent control”, which requires self-awareness and “freedom” (autonomy).

Autonomic Computing

2006-12-03T07:57:00.000-08:00

Since "Autonomic Computing" is a key concept related to the topic of this blog, I have translated an extract from my book ("Urbanisation et BPM"). It is not the best reference on the topic :) but it will give an overview for readers who are not from the IT community. This extract was written in 2004 (therefore, it is no longer precisely up to date ...)

Autonomic computing (AC) is the name given to a large-scale research initiative by IBM, on the following: mastering the ever increasing complexity requires to make IT systems « autonomous ». More precisely, « autonomic » is defined as the conjunction of four properties :

(1) self-configuring : the system adapts itself automatically and dynamically to the changes that occur in its environment. This requires a « declarative » configuration, with a statement of goals and not a description of means. A good example of such a declarative statement is the use of SLA as parameters. When something changes, the system tunes its internal parameters to keep satisfying its SLA policy.

(2) self-healing : the management of most incidents is done automatically. The discovery, diagniosis and repair of an incident are made by the system itself, which supposes a capacity to reason about itself. Therefore, such a system holds a model of its own behaviour, as well as reasoning tools that are similar to so-called “expert systems”. This is often seen as the comeback of Artificial Intelligence, although with a rich model, simple and proven techniques are enough to produce an action plan from incident detection.

(3) self-optimizing : the system continuously monitors the state of its resources and optimizes their usage. One may see this as the generalization of load balancing mechanisms to the whole IT system. This requires a complex performance model, which may be used both in a reactive (balancing) and proactive (capacity planning) manner (cf. Chapter 8).

(4) self-protecting : the system protects itself form different attacks, both in a defensive manner, while controling and checking accesses, and in a proactive manner though a constant search for intrusion.

(note: To start with an excellent introduction to « autonomic computing », one must read the article « The dawning of the autonomic computing era » from A.G. Ganek and T.A. Corbi, which may be found easily on the Web)

The interest of these properties and their relevance to the management of IT systems are self-evident. One may wonder, however, why a dedicated research initiative is called for, instead of leveraging on the constant progress of the associated fields of computer science. The foundation of the Autonomic Computing initiative is the belief that we have reached a complexity barrier: the management cost of IT infrastructure (intallation, configuration, maintenance, …) has grown relentlessly with their size and power, it represents more than half of the total cost. Next generations of infrastructure will be even more complex and powerful. IBM’s thesis is that their management is possible only if it becomes automated. In the article that we just quoted, a wealth of statistics is given that shows the ever-increasing part of operational tasks, while at the same time the business reliance on IT system is equally growing, resulting into financial consequences of IT outage that are becoming disastrous.

Therefore, a breakthough is necessary to overcome this complexity barrier : in the future, IT systems need to manage and repair themselves automatically, with as little human intervention as possible. This may sound far too ambitious and closer to a “marketing public relation initiative” than a directed research project : after the eras of distributed computing, autonomous computing and ubiquitous computing, here comes autonomic computing. This initiative flows continuously from computer science research themes of the past 30 years. Some of the problems and solution directions are actually old. However, two paradigm shifts have occured. First, the center of attention has evolved in the last few years from software (component, application) to software system (hence the focus on enterprise archictecture and business process). In the 80s and 90s, the focus on problems similar to those of AC has given birth to the concept of intelligent agents, but their application to the full scale of corporate IT has proven to be difficult. From a CIO perspective, the focus on enterprise architecture and infrastructure is a welcome shift, especially since the research budgets are impressive. On IBM’s side, this is the main theme for a R&D budget of over 5 billions dollars. Most other major players also work on similar initiatives, even though the vocabulary may vary.

The second paradigm shift is the emergence of the biological model of incident processing. This is a departure from an endless search of methods that would produce software free from bugs and multiple back-ups that would guarantee a complete availability of infrastructures. Autonomic Computing applies to the “real world”, where software contains many defects and run on computers that experience all sorts of outage, and draws analogy from “living organisms” to deliver fault-tolerant availability. Living organisms cope with a large spectrum of aggressions and incidents (bacteries, viruses, …) with a number of techniques: redundancy, re-generation, self-amputation, reactive aggression, … etc. Similarly, autonomous IT systems need to be designed to perform in an adverse environment. The analogy and inspiration from biology is a long-lived trend in computer science. For instance, the use of “ evolution theory” as an optimization strategy has produced genetic algorithms or swarms approaches. To quote a NASA expert, who is applying the concept of swarm to micro-robots, some part of the design is replaced by “evolution as an optimization strategy”.

Even though there is a form of utopia in this field and many goals are definitely long-termed, this is not science-fiction. There is an evolutionary course towards autonomic computing and the first steps correspond to technologies that have already demonstrated in research labs. Actually, there is a symmetry in the founding argument: if we are not able to provide our new systems with more autonomy, there progress in scale and complexity will soon reach a manageability limit. It is, therefore, logical to bet on the forecoming availability of autonomic capabilities. The issue is not: “If the systems become autonomous”, is is “when” and “how”..

Automic Computing is not a philosophy, it is an attitude, according to the experts. Systems should be designed to be autonomous from the very early design, even though some of the choices and options may still be very rudimentary. An illustration drawn from the practice of real-time systems is the heartbeat principle. A heartbeat is a periodical simple signal that each component broadcast as a testimony to its status (alive, weakened, distressed, etc.). The « heartbeat » principle comes from the real-time systems community. For instance, it is used for NASA ‘s satellites under the « beacon » designation. On this topic, one may read, the papers from the NASA research center in the proceeding of the autonomic computing workshop that was part of EASE’04.

This proactive attitude seems relevant to the design of information systems. The paper form Ganek and Corbi makes a strong argument in favor of evolution, as opposed to revolution, based on a roadmap that goes from “basic” to “autonomous”. The progression is labeled through the steps: managed, predictive, adaptative and autonomous. It is completely relevant to the transformation of business process management, as described in Chapter 8.

AC and Enterprise Architecture

The foundations of autonomic computing are well suited to the field of Enterprise Architecture, since IT components make a large and complex system. Hence, many aspects of the AC approached are relevant to the re-engineering of IT. We shall now consider three: autonomic computing infrastructure, autonomic business process monitoring and “biological” incident management. There is a clear overlap, but we shall move from the the most complex to the more realistic.

The concept of autonomic IT infrastructure has received the most media attention since it is the heart of IBM strategy, being a cornerstone of the on-demand computing (cf. 11.3.2). An autonomic infrastructure is based on computing resource virtualization; it manages a pool of servers that is allocated to application needs on a dynamic basis. The global monitoring allows load balancing, reaction to incidents through reactive redistribution, and dynamic reconfiguration of resources to adapt to new needs. The ultimate model of such an infrastructure is the GRID model, which implies that grid computing is de facto a chapter of autonomic computing. A grid is a set of identical anonymous servers that are managed as one large parallel computing resource. The management of Grid computing has already produced valuable contributions to the field of AC. For instance, the most accomplished effort to standardize with “quality of service” means in the world of Web Services, WSLA, comes from the world of grid computing. One may read « Web Service Level Agreement (WSLA) Language Specification”, and see, for instance, how performance-related SLA are represented (under the measurement headins, 2.4.8). One may see the grid as a metaphor of tomorrow’s data center. Therefore, even though the grid is often reduced to the idea of using the sleeping power of a farm of PC during the night, most CIOs should look into this field as a guideline for their own infrastructures.

A grid is by construction a resource that may be shared. It has interesting properties of robustness, fault-tolerance and flexibility. It is, therefore, a convenient infrastructure to bridge the gap between the autonomic and the on-demand computing concepts. A “on-demand” infrastructure is a flexible computing infrastructure, whose capacity may be increased or decreased dynamically, according to the needs of the company’s business processes. Such an infrastructure may be outsourced, which yields the concept of “utility computing” as a service, or it may be managed as an internal utility. There is more to “on-demand computing” than the infrastructure aspects, which will be covered at the end of this chapter. The synergy with autonomic computing is obvious: the business flexibility requirements of the on-demand approach demand a breakthrough in terms of technical flexibility which translates into autonomic computing, and is well illustrated by the grid concept.

This vision is shared by most major players in the IT industry. However, we consider this as a long-term target, since the majority of today’s application portfolios in most companies may not be migrated smoothly onto this type of infrastructure. This does not mean, as was previously started, that some of the features are not available today. For instance, “blade server” infrastructures deliver some of the autonomous benefits as far as self-configuration, self-management and self-healing are concerned.

On a mid-term perspective, one may expect the field of autonomic computing to have an impact on business process management. The field of OAI (Optimisation of Application Integration, cf. chapter 8) is equally a long-term, complex research topic, but which will benefits from small-steps improvements. Two family of software tools are currently evolving towards autonomous behavior: integration middleware and monitoring software. BAM (business activity monitoring) software is integrating capabilities to model and to simulate the occurrence of business incidents. Adaptative middleware or “fault-tolerant” middleware are emerging. For instance, look for the Chameleon project, which belongs to the ARMOR approach (Adaptive, Reconfigurable and Mobile Objects for Reliability). Chameleon is an integration infrastructure which combines adaptability and fault-tolerance.

On a short-term perspective, one may apply the “biology metaphor” of autonomic computing to formalize the management of incidents, as it often occurs in the “real world”. To summarize, one may say that system operation relies on two visions: a mechanical and an organic one, to deliver the continuity of service that is required by its clients. The mechanical vision is based upon redundancy, with back-up and spare copies of everything. When something fails, the faulty component is replaced with a spare. Depending on the recovery time constraints, this approach translates into a disaster recovery plan, into clusters, fault-tolerant hardware, etc. For instance, one may look at the discourses from Carly Fiorina, HP’s former CEO, who focuses on mutualization, consolidation and virtualization under « Adaptative Enterprise » vocable. The same keywords appear in SUN’s or Veritas’s strategy about servers and data centers.

The organic vision is based upon alternate scenarios and re-routing. It is derived from an intimate knowledge of business and its processes. It consists of a network of “alternate sub-processes”. It requires a strong cooperation between operation managers, application developpers and business owners. Only a precise knowledge of the business and its priorities enable a transverse team to find valid “ alternate approaches”. We named this approach « organic operations », which may be described with the following goals :

Create an operations model which supports the definition of scenarios and takes all operations tools and methods into account (some of which are represented with hatched boxes in the figure). Stating existing recovery procedure in a formal and shared manner is both the first stage of maturity and a possible step towards automation.
Create multiple levels of “reflexes and reaction”, which are automated incident management rules. An interesting contribution from the biological metaphor is the separation between reflexes (simples, distributed) and reactions (centralized, require a « conscient » intelligence).
Create the tools which allow us to make “intelligent” decisions. These may be representation/cartography tools (which is linked to the topic of business activity monitoring) or planning/simulation tools : what would happen if the message flow from component A was re-directed towards B ?

Welcome Message

2006-11-19T05:53:00.000-08:00

The first message of a blog is always the most difficult : one does not know where to start. Today I’ll just describe some of my motivations and objectives.
I have become fascinated over the last few years with the topics of emergence, biomimetics (http://www.bath.ac.uk/mech-eng/biomimetics/about.htm), autonomic computing (http://www.research.ibm.com/autonomic/) and artificial life. I look at these topics from a dual perspective : as a sientist and as the CIO of a large organization.

1. Motivations :

The opening of this blog is the result of three converging threads :

OAI : Optimization of Application Integration.
OAI is the field of optimizing the quality of service in a business process-oriented IT infrastructure. The link with EAI is obvious ... and dated. Today, it would be smarted to talk about an SOA architecture. The problem is the same, though : how does one optimize the quality of service, measured at a business process level, in a real-life information system (i.e., with failures, bursts, and so on).
I have looked at this problem with my operations's research culture, it is a beautiful problem : rich, complex and very relevant to real life operations.
You may look at my last published paper on this topic: Self-adaptive middleware: Supporting business process priorities and service level agreements • ArticleAdvanced Engineering Informatics, Volume 19, Issue 3, 1 July 2005, Pages 199-211Caseau, Y
Autonomic Computing.
I won't start telling with AC is for me in this short message. I wrote quite intensely about it in my book about Enterprise Architecture and Business Processes. I'll post a message with a translation of a short fragment to state my positition. The key point is that I am a believer, from my CIO standpoint: the only way to achive the kind of cost reduction, complexity reduction and improvement of QoS is to let Information Systems become "autonomic".
The reading of the fascinating book "Out of Control" by Kevin Kelly.
It will take me a number of messages to share all the insightful comments which I have found in this book.
For instance, these are two my favorite quotes from this book.
« Investing machines with the ability to adapt on their own, to evolve in their own directions, and grow without human oversight is the next great advance in technology. Giving machines freedom is the only way we can have intelligent control. »
"The great irony puzzling cognitive scientists is why human consciousness is so unable to think in parallel, despite the fact that the brain runs as a parralel machine. [...] That's why the first computers were programmed in the von Neumann's serial design : because that's how human think. And this, again, is why parallel computers must be evolved rather than designed: because we are simpletons when it comes to thinking in parallel."

It turns out that these three threads are totally related, and the combination is the foundation for the "theory of the live information system" :)

I will illustrate this with a quote from my own new book (coming out in a few months) :

"Information Systems complexity will be mastered once the IS has become alive. This "livelyhood" must be understood according to the meaning that is proposed by Kevin Kelly : "Life is is an emergent property, contingent upon the organization of inanimate parts but not reducible to them" [...] By stating that the ideal IS must not be designed but grown (through emergence), I include information systems in the large family of (truly) complex systems. Kevin's Kelly analyses leads us to think that the "satisfactory behavior" of complex systems (in the case of IS, this would be the quality of service - availability and performance) is precisely an emerging property, at the global system level, and not a feature that would be built at the component level. In that sense, the ideal IS is a "vivisystem" which blends properties that are built intentionally (for instance, through the architecture of the system) and properties that emerge through self-regulation and self-adptation to the environment.

2. Objectives of this Blog

This blog has three main objectives:

Act as an "idea workbench" (as for most blogs of the same kind) : improve the depth of my own analysis (though writing) and expose my thoughts to early criticism.
Create a network of people (who share similar interests) and bibliographic references.
Develop a unified theory of high avalaiblity / high efficiency / highly adaptable information systems with a practical roamap.

This last point about "practicality" is worth a remark. There is a tension between an "Artificial Life Theory" perspective and the "Adaptative Middleware Experimentation" perspective. One may wonder if I see my own thinking as a philosopher's pastime ... or as the resolution of a CIO practical issue. This would require a lengthy debate, I will simply say that I believe that both a "science-fiction" and a "computational experiment" twists are necessary to achieve success.

3. Methods

I plan to use this blog in a way that is similar to my other blog on organizational architecture:

share bibliography : post book reviews and ask for further readings.
share thoughts, once I am ready to formulate them.
share the results of computational experiments. I plan to use computer simulation to explore some the ideas, as I did for the work on OAI.

I decided to write this blog in English, because most of my bibliographic references are written by Americans. The topics that are mentionned here were born in the US and the likelyhood of joining a thought-network with relevant contributors is much higher if I use an intenational language (a litle bit of provocation here :)). However, comments posted in French are welcome !

One may ask if there exists a relationship with my other blog (http://organisationarchitecture.blogspot.com/). The simple answer is negative: there are no links but the common use of simulation tools. At a "far, far level" a company may be seen as a distributed information system, thus a few connections may be drawn in the future.