SOA : A Tale of Two Cities

This post is an attempt to illustrate a key message: the benefits of SOA are dwarfed by the inconvenient of not doing it. When I write SOA, I mean the combination of MDM (Master Data Management), EA (Enterprise Architecture) and Service Architecture on a global scale. This has been the topic of many of my posts, so I won't go into details. I won't detail the expected benefits either (cf. the previous post on SOA's presumed death) : cost reduction, increased agility, sharing and complexity reduction.

What is clear for everyone is that this approach has a cost. It can be a large set-up cost for a first project or a moderate "architectural investment" if SOA is a sustained practice. I have witnessed the two alternatives when a large-scale new project is launched: with and without such an effort.

It is now clear for me that the true difference in the outcome is not the set of expected benefits of a "disciplined/architected" approach, but rather the unexpected benefits (what one could call the strategic agility) and even more the avoidance of major problems in the future life of the IT system that is being built, mostly with respect to data integration.

Separating between strategic and tactical agility makes sense. Tactical agility may be defined as the ability to make the "easy transformations" to the information system (IS) as easily and cheaply as possible (they go hand in hand). Strategic agility is measured by the cost of making "the hard changes" (those that are deemed "impossible" the first time the need is mentioned). Easy vs. hard is both a matter of anticipation (strategic agility is the ability to move the IS towards a brand new direction) and scope. Most of the technology, such as middleware, is geared towards tactical agility. It helps to implement "reasonably easy changes" faster (sometimes much faster). But what helps to "turn around the ship a full 180 degrees" is the hard work on architecture (mostly, data architecture and then, service architecture). See the recently posted bibliography for more pointers. Strategic agility is difficult to evaluate, but not impossible. Playing with scenarios seems to be the best approach. See my book, or "Organisational Capital", edited by Ahmed Bounfour. For the lack of a better word, I will call structural agility the "status of being able to avoid major problems" through the taming of complexity. When complexity is not curved, unseen consequences start to happen. This is when we see huge overruns in budget and time, or even complete failure of large projects.

I am always worried when I see a "core-centric" project, usually motivated by the introduction of a new technology. A "sacred alliance" occurs between the client who sees the new technology as a quick relief of a precise pain, some itching that has been going on for a while, and the IT folks who are always happy to try on a new thing. New technology here may be a rule-based engine, a new database techno, some learning/recommendation engine, a new Web/Interface generation tool, etc. What I mean with "core-centric" is the focus on the core "new thing", the core "new benefit", as opposed to the edge: the integration with the rest, the way the new system interacts with the old stuff. Core-centric projects always follow a successful proof of concept. When the focus is on the core, it is actually hard to fail a proof of concept … I have, obviously, nothing against proofs of concepts. They are clearly necessary, there is no reason to work on the hard stuff (the edge) if the core benefits is not worth it. But one must keep in mind that a successful "proof of concept" is theeasy part. A common plague of core-centric projects is that they are often "designed by committees". This will be the topic of another post … an IT component need to have on clean business customer (a physical person) and one identified architect.

Core-centric projects tend to start well and to end in misery when adding the last 20% of data integration seems to cost 80% of the effort. The sad thing is that one cannot buy an enterprise architecture from an outside vendor (although help/consulting works J). A "turnkey" project, even with a high quality supplier, delivers exactly what you pay for, but no more. Precisely, the level of integration is what is defined in the specification. Extensibility, the ability to add a new data source, to take a new business practice into account, to adapt to a new competitor… are why architecture is necessary before adding a new IT component. One cannot expect to get this kind of "forward thinking" from an outside vendor. ISVs cannot, as a general rule (and each rule has its exception) carry the weight of the integration issue. Another way to say this is that integration should be "inward-bound" and not "outwards-bound": what matters most is outside the new project, not inside. This applies especially to SOA: it is much easier to define the services that a new component may expose than the services that it may use.

IT Strategic Alignment, a buzzword of these last 10 years is indeed a difficult exercise, because of the dynamics of the target – constantly moving - and the inertia of the IS. Enterprise Architecture is about system dynamics and trajectories. Aligning over the "strategy" is necessarily difficult because the target is vague and shifting its shape continuously. The fast movement of the target coupled with the slow speed of IT transformation means that both anticipation and abstraction are required. Anticipation is necessary because transforming the IS takes time. Even within the framework of a sustained SOA effort, herding the set of services towards the desired service architecture takes time. Abstraction is required to filter out the "micro-variations" and to focus on the key long-term changes.

Let us pick an analogy: imagine that we want to wrap objects of various shapes that are given to us randomly, with sheets of a rigid material. A good example would be statues, since they exhibit very different forms. To prepare beforehand, we pre-fold the sheets. Obviously, the objects represent the business opportunities … while folding the sheets represent undergoing IT projects to fit the opportunities. The preparation beforehand is similar to enterprise architecture: a little effort in advance to speed up things when the real problem occurs. However this preparation only helps if the folds are actually useful to wrap the new shapes. It is actually an interesting analogy since finding a set of versatile preparatory folds is a hard problem.

The following set of illustrations is taken from wrappings by Christo. They are not the best illustration of this fictitious example (no pre-folding since the wrapping material is not rigid) but they look nice J

From this analogy we can pick three key principles:

• One cannot do the architecture without knowing what "the future holds" from a business perspective : Service Architecture is about business
  
• Going from the shape to the folds is tricky : Architecture is not for "dummies", it requires thinking and abstraction
  
• One can overdo it and spend more time solving the puzzle than solving the business problem. It is easier to wrap a complex form than find the optimal set of folds that would help wrapping the most.
  

It would be easy to transform this post into a fictional story to contrast two approaches for a new complex IS project, with and without a SOA investment. I might do this as a new "Caroline story" for a new edition of my book, in a Tale-of-two-cities' style. So, to return to the initial question, what could motivate Caroline to "do it right", if it means a slower start and an increase in the total cost ? Purely defensive arguments are hard to sell, even if perfectly correct (e.g., a higher cost estimate but a higher probability of avoiding overruns). This brings us back to the concept of "situation potential", borrowed from Chinese strategy and which I have mentioned earlier. This is truly a powerful idea, worth yet another post; it unties the Gordian knot that mixes complexity, the difficulty to forecast and the different time scales. It may be seen as the combination of tactic agility, strategic agility and structural agility. Being able to demonstrate and sell the increase of "situation potential" is what it takes to develop a long-term Enterprise Architecture effort.

Selected Bibliography

Here is a short selection of my favorite books about IT and Enterprise Architecture. I will try to update and develop this selected bibliography in the future, and I am always on the lookout for additions. A more detailed list may be found in the bibliography section of my last book, but this is a "selection from the heart".

The first list I have assembled for my course at the Ecole Polytechnique. These books are both inspiring and reasonably easy to read J

• R.J. Wieringa, "Design Methods for Reactive Systems: Yourdon, Statemate, and the UML», Morgan Kauffman (2002)
  A really great book about design methods, with both a lot of structure (theory) and practical insights from the domain of reactive systems. Very relevant for complex fields such as telecommunications.
  
• P. Roques, "UML 2 en action : De l'analyse des besoins à la conception », Eyrolle (2007)
  This is not a reference book (there are better books to learn about UML) but this is the best book I know to understand how to generate value from the practical application of UML.
  
• P. W. Keen, « Shaping the Future: Business Design Through Information Technology », Harvard Business School Press (1991)
  A key reference (heavily quoted) that is still the most comprehensive book on the topic of IT economics. Much better than newer books, especially nice for rebuking naïve statements about SOA, Web Services … or other "silver bullets". The best counterargument against "IT does not matter", from N. Carr, that I have read.
  
• L. H. Putnam, "Five Core Metrics: The Intelligence Behind Successful Software Management», Dorset House Publishing (2003)
  The smartest book I have found about software metrics. All the mistakes I have made previously are neatly identified. Not only the difficult, multi-dimensional aspect of software measurement is well accounted for, but the book provides with practical and efficient methods.
  
• J. Printz, "Coûts et durée des projets informatiques pratique des modèles d'estimation », Lavoisier(2002)
  A very nice introduction to Cocomo and other methods.
  
• I. Jacobson, "The unified software development process », Addison Wesley (1980)
  A classical reference that still makes a very good reading. Extremely useful to understand the current state of "software development best practices"
  
• P. Grosjean & al., "Performance des architectures IT », Dunod (2007)
  Amazing book: very practical yet rigorous, covers a large scope of issues and provides with very relevant solutions to real world problems.
  
• X. Fournier-Morel & al, "SOA, le guide de l'architecte du SI », Dunod (2008)
  The best book I have ever read about Service Oriented Architectures. Obviously, covers the technical more than the governance side of SOA but is still the best book that I know of to understand what SOA is really about.
  
• D.Gross, « Fundamentals of queueing theory », Wiley (1998)
  One cannot talk about IT performance without a minimal background on Queueing Theory. This is one of the good introduction books (there are many others).
  
• F.A. Cummins, « Enterprise Integration: An Architecture for Enterprise Application and Systems Integration », Wiley (2002)
  My favorite book about IT integration. A practical book that hits all the key topics and does not shy away from the hard problem. Only 10% of the books that talk about EAI, SOA or integration infrastructure are actually relevant to "real world usage", most of them are just re-hash of marketing slides (all the glory, no guts J). This is one of the precious few.
  
• M. Tamer Ozsu, P. Valduriez, "Principles of Distributed Database Systems », Prentice Hall (1999)
  My reference book on database systems. Although it is quite complete and covers most of the issues relevant to distributed systems, it is still an easy read.
  
• E. Marcus, H. Stern, "Blueprints for High Availability», Wiley (2003)
  Wonderful book about high availability. All you need to know, tons of practical advice and many examples. A must-read for anyone in IT operations.
  
• K. Schmidt, "High Availability and Disaster Recovery: Concepts, Design, Implementation », Springer (2006)
  More refined and detailed than the previous one, a great reference book on robustness and redundancy.
  
• R. C. Seacord, "Modernizing Legacy Systems: Software Technologies, Engineering Processes, and Business Practices", Addison-Wesley (2003)
  The only book that I know of that talks about "re-engineering of legacy systems" (what we call "refonte" in France) in a way that is consistent with my own experience of a CIO at Bouygues Telecom. All the hard issues are covered and the book is full of sound practical advice.
  

 

The next list is a reference list. These books are heavier, and are not meant to be read "in one shot". On the other hand, they contain "treasures of knowledge".

• C. Jones, "Applied Software Measurement: Assuring Productivity and Quality », Mc Graw Hill (1996)
  My "bible" during the last 10 years : all the hard numbers necessary to model software development costs and quality insurance. This is "the survival kit" for anyone who wants to introduce function points measurement.
  
• J. Printz, "Architecture logicielle concevoir des applications adaptables » Dunod (2006)
  This is a reference book on software architecture. It is very thorough, explaining all the hows with the whys. Very valuable to get a deep understanding on SE principles.
  
  • Meinadier, "Ingénierie et intégration des systèmes», Hermes (1998)
    Still one of the best reference books about system engineering.
    
• B. W. Boehm, "Software Cost Estimation with Cocomo II ». Prentice Hall (2000)
  No one can afford to miss Cocomo II, since it is the scientific reference for almost all intuitions that one may develop after spending years of developing SW projects.
  
• W. Perry, "Effective Methods for Software Testing », Wiley & Sons (1995)
  560 pages that tell 90% of what one should know about software testing. I was fortunate to spend many years with people who had spent their live researching this topic at Bellcore, and find this book to be surprisingly complete and accurate.
  
• D. A. Menasce, "Performance by Design: Computer Capacity Planning By Example", Prentice Hall (2004)
  The best book that I have read about Performance modeling and capacity planning.
  
  
  

The last list contain fun books to read (at least from my perspective) which actually tell a lot about IT

• M. Crichton, "Jurassic Park", Ballantine Books (1991)
  This is still the best way I know to get acquainted with chaos theory and why it is relevant to understand IT failures J
  
• C. Perrow, "Normal Accidents: Living with High Risk Technologies", Princeton University Press (1999).
  A must read about industrial accidents (such as TMI: Three Miles Island) – The analysis and the proposed patterns are brilliant.
  
• K. Kelly, "Out of Control: the New Biology of Machines, Social Systems and the Economic World", Perseud Books Group (1995)
  My favorite book of all time, see earlier in this blog J
  
• C. Hibbs, S. Jewett, M. Sullivan, "The Art of Lean Software Development", O Reilly (2008)
  A wonderful very short books that contains one of the best introduction to lean I have ever read, and a wealth of advice about software development. Very concise but incredibly relevant.
  
• F. Brooks, "The Mythical Man-Month: Essays on Software Engineering, 20th Anniversary Edition », Addison-Weslay Professional (1995)
  Still relevant after so many years !
  
• T. DeMarco, "Peopleware: Productive Projects and Teams (Second Edition) » Dorcet House (1999).
  A treasure trove ! a non-nonsense inquiry into what it takes to be productive when writing software. The most incredible part is that the main contribution (such as the negative influence of disruption) are still unique to this day (to my knowledge)
  

  
   

  
   

Let me know about your own favorite IT books.

SOA is much too young to be dead

Today I'll look into the SOA controversy that was generated by Anne Thomas Manes's SOA's obituary. 

This may look like a slow response (90% of the blogosphere has already published their comments) but I will take some length to explain the issue (and this required a little time for thinking over). Surprisingly, one still get a very fuzzy and abstract idea about what SOA from reading all the positive answers to Ann Thomas Manes.

What is much clearer are the benefits one should expect from SOA (and they have been clear from 2003). SOA brings increased agility and reduced costs. The increase agility comes from reusability an the cost reduction comes from sharing services. Reusability comes in two flavors : direct (reuse of an existing service) and indirect=mash-up (quickly mixing existing services to produce a new one).

There are additional benefits (such as the ease to mix internal and outsourced services) but these two are enough to get anyone's attention. The question is: how to get there ?

1. SOA is a 3 step process

First, we decompose SOA into three steps ("doing SOA" means doing all three in sequence):

1. Service Definition
2. Service Architecture
3. Service Integration
   

Service Definition simply means to come up with the catalog of services. It is a "business oriented" step: the business processes are analyzed and the relevant business services are identified. At first it looks like the "simple" (nothing but simple actually) first part of a functional analysis of a new system. There is (much) more to it, because the role/user analysis is an integral part. The focus on reusability and agility starts here. Hence it is necessary to play with business scenarios, which further means that it is not an information system play. In the "acronym soup", the relevant terms are EA (Enterprise Architecture) and SOBA (Service Oriented Business Architecture) which have been introduced later on to insist on the "business/enterprise" dimension.

Service Architecture is the technical step that takes a large catalog as an input and produces a structured hierarchy as an output. A list of thousands of equally visible services is in no way a feasible approach to manage a large information system. Classical issues such as modularity, encapsulation, visibility, etc. must be taken into consideration. I write "classical" because this is the heart of software engineering and software architecture. Re-factoring techniques (the same that are used when writing a class or a component library) must be applied to produce services that have a better chance of being reused or being combined. I won't dwell into it today, but the role of data & models architecture is key (as I said, this step is the IS step).

Service Integration is what most people think about when they say SOA: linking all the services into an integration infrastructure. This is where all the technologic  issues live (ESB or not, SOAP or REST, etc.). The term SOI (Service Oriented Integration - for instance, see this definition) was coined to denote this step.

These three steps help to grasp the huge difference between "SOA in the large" (with a large enterprise as its scope) and "SOA in the small" (SOA for one information system, either for a small company or one department). The effort repartition obviously varies, but here is the pattern from my own experience:

• In the large: 40/30/30  (i.e., 40% of the effort is the definition part)
• In the small: 20/10/70

It is interesting to notice that there is (almost) nothing new with "SOA in the small": service definition + architecture + flexible implementation is what has been making a good piece of software for 30 years. 

2. What is easy/hard ?

Service Definition is hard when the scale gets big. Although producing the service catalog is straightforward for a department-size project, there are mutiple issues of heterogeneity and multiplicity of stakeholders that makes it much more complex to manage at the (large) entreprise level. Not only the management of the catalog gets harder as the size grows, but there is a subtle balance to be found between "common sense" and "formal methods". There are indeed methods/pattern/frameworks available for this first step but:

• if this step is too formal, too many stakeholders are discouraged
  
• if it is informal, the process crumble under its own weight.
  

The same is even truer with Service Architecture. It is quite simple when doing "SOA in the small" but gets harder as the size grows. Contrary to Service Definition, there is little help available as a methodology: hard learned IT experience is necessary. Unfortunately, this is not something that one learns through writing Powerpoints. If that step is poorly executed, the integration is more difficult, but - and this is the key point - little reusability or agility is observed later on. This is something that I am trying to teach during my "Information System Course" at Polytechnique, but this is a hard, technical topic which is better acquired through practice than theory.

Service Integration is actually not so difficult nowadays, because the technology is quite mature. I am not saying that it is easy, but it would be sad to ignore the progress that has been delivered by technology providers. There are still a few difficult issues (cf. next section) and, more generally, a large-scale integration project still exhibits all the possible pitfalls of a complex IT project, but there is nothing new here. Actually, when done "in the small", this is straightforward and there are so many success stories to prove it.

3. What is the maturity state of SOA ?

Using the same decomposition, here is my own assessment: 

1. 50% for Service Definition. This step is rather well explained, many good books are available (such as "SOA for profit"). It is still a difficult job at the enterprise level. There is indeed a governance issue (those who mock the "governance issue" have missed what Enterprise Architectuire is, and are still trying to accomplish "small scale SOI"). There is also a "pedagogy issue": an entreprise-scale effort cannot be undertaken if its meaning is not understood, by all stakeholders.
   
   
2. 20% for Service Architecture. There are very few books that can help, and even fewer are directly relevant for SOA. The only one that I recommend to my students is "SOA, Le guide de l'architecte." (in French, sorry for my English speaking readers). I have tens of books about IT/software architecture in my private library, including great pieces from Fred Cummins, Len Bass, Richard Shuey, Paul Clements, Jean-Paul Meinadier, Jacques Printz, Robert Seacord, Roel Wieringa - to name a few -, but none of them (including my own two books :)) adress the issue of Service Architecture "in the large" thoroughly.
   
   
3. 70% for Service Integration. The technology is there, it is scalable and proven (WebSphere or Weblogic being two classical commercial examples). Obviously, there are still technical issues. For instance,

• distributed transaction & synchronisation is still a hard problem.
  
• performance overheads (cf. SOA is not scale-free) still exist and makes the deployment tricky when response time is an issue.
  
• monitoring & autonomic load balancing is difficult and/or not sufficiently developed.
  
• service discovery and pub/sub architecture (EOA) are not as straightforward to implement as one might wish

But those are hard issues, nothing to be ashamed of :) 

4. SOA is a practice, not a target or a technology

As David Linthicum said: "SOA is something you do, not something you buy". It is not something that you do once (a project) but that you do all the time, continuously. 

This is what brought the creation of the sustainable IT architecture approach. Done right, SOA is a practice that transforms information systems for the better. According to the money that you invest, to the legacy and to the business constraints, the transformation is slower or faster, but it takes time and is never ended. The sustainable IT approach comes from a change of perspective: instead of always focusing on what is to come, on the new projects, one should search for more value from the "existing assets". Another "sustainable" principle is to save your strengths to last longer (hence the continuous improvement approach as oppose to the big bang).

It's a "culture thing". Part of it (cf. Service definition) is everybody's concern, hence it must be simple and must be pushed at the CxO level. Therefore, IT governance is a key component for changing the culture.

I will conclude by stating that SOA represents a "Chinese strategy" for IT management, in sense of François Julien. I could not urge you enough to read his book, but if you have not, a key principle is to distinguish between planned strategy (top-down, in the Greek fashion) and "building up your situation potential". A Chinese proverb says than one does not grow a plant faster by pulling its stem.

Similarly, instead of pushing "SOA technology projects", a better approach is to build up your "situation potential" : build a common data model, train everyone, build a shared consensus about what the service catalog should be, etc. Each project is then seen as an opportunity to move one step in the right direction. 

Information Technology for the Chief Executive

At last, the english edition of my second book is available. It is somehow simplified (no technical appendix and a shorter, leaner introduction) but the body is the same as "Performance du Système d'Information".

At the same time, I just received the "Best IT book award" from the AFISI. The AFISI is the French association for engineering and information systems. I am quite honored to receive this prize when I look at the list of the previous winners.

Since I have used a self-publishing company and performed the translation myself (with the help of a professional interpreter), trying to promote this book is going to be an adventure.

Very Cloudy Weather: will it rain on the « Cloud Computing » Parade?

This blog has been very quiet for the last few months … and will remain as such until I am finished preparing my lecture about Information Systems for Polytechnique (2009). I should have, at that point, accumulated quite some material for future posts, since I am digging quite hard into the theoretical foundations of "Information Systems".

However, I have been reading so many newspaper articles about "cloud computing" during the last few weeks that I feel like putting a few thoughts on paper. The turning point was the article from "The Economist" last week: as is rightfully noticed, too much hype will likely cause "disillusion". I am actually totally convinced that "cloud computing" is relevant to many businesses and that it represents a revolution that is coming. I have actually explained some of this in my other blog. If you are new to this, a really simple description of the benefits could the following:

• Fault-tolerance through the implicit redundancy. This is, however, an architectural issue. It is not enough to rent your computing infrastructure from Amazon, Google or Microsoft to get this benefit. You also need to implement your information system with an architectural paradigm which takes advantage from the availability of multiple servers.
  
• Super-computing performance through parallelization. The same remark applies: it is true that new techniques for data mining or real-time event processing (two examples) may be tried successfully on the cloud through a MapReduce approach; it also requires a significant amount of work if you start from your legacy application.
  
• Reduced costs of operation (TCO) through the use of standardized and mass-produced units. Look at the price of the TPMC according to the type of hardware and you will get the idea. I won't dwell on this, this is explained everywhere in the newspaper articles I was mentioning.
  

There is an implicit warning here: although "cloud computing" is "the way to go" for many cases/enterprises, there is a learning curve and a price to be paid. Especially, it will take time and energy to move from an existing architecture to one that can be "migrated on the cloud". However, "Cloud Computing" is not the "ultimate solution" for everything and I do not believe for a second that "software is dead". Let me give three simple reasons for which one may decide to stay on the "firm ground" as opposed to ""move to the cloud":

1. The risk of loosing privacy and control (cf. R. Stallman's reaction). There are many aspects, including legal and societal, to this issue. This is the part that is reasonably well covered in the papers (such as The Economist). It is clearly a valid point, but technology and service segmentation might alleviate this issue in the future. There may exist private clouds, secure clouds, encrypted clouds (where the encryption is managed by a third-party), etc.
   
2. Latency : accessing the cloud is not instantaneous. Even if the protocols were truly optimal (and they are not, web service invocation carries a significant overload), it takes some time to access to a distant data center (the light takes 10 ms to travel 3000km, and 10ms is significant for many high-performance computations or transactions). This is why MMORPGs rely on a RCA (rich client architecture – a significant part of the work is done locally).
   
3. Computational overhead: making each service invocation a web service invocation is not practical for high performance computing. There is a proper level of granularity for encapsulating a piece of computation/transaction into a cloud service. This is actually something for which there exists a significant amount of history: when companies try to develop a SOA architecture, they have to get the service granularity right. Otherwise, they "discover" that application servers cannot carry an infinite load and that they exhibit a performance overhead when compared with more traditional approaches (a RPC – remote procedure call- is more expensive than a procedure call, and a WS call is not the cheapest RPC).
   

Hence we get two negative answers to the following questions:

• "Can I move all my IT onto the cloud without changing my apps and their architecture ?" (the cloud as the universal IT outsourcer). The answer is negative when performance (throughput or latency) constraints exist.
  
• "Can I migrate all my apps towards a SOA architecture that will live in the cloud, made from SaaS (Software as a Service) ?". The answer is negative because some services are better produced locally. The economy of scale of the SaaS requires some form of mutualization (running a service remotely for a unique client is not a clear winner). If performance constraints prevent from breaking an application into "tiny components" (because of the cost of recomposition), many apps are too specific to be run as SaaS.
  

To summarize, I would propose two "axioms/theorems/conjecture" about "cloud computing":

1. "Cloud Computing" will succeed because of the combination of two facts:
   
   1. The grid architecture is the best architecture to provide true scalability and superior availability (fault-tolerance)
      
   2. Grid architecture exhibits "economy of scale", that is, can be run more efficiently when operated on a very large scale. The more identical servers one operates, the lower the TCO per server is.
      
2. "Service Oriented Architecture" (SOA) is not scale-free: there is a balance to be found between "recomposability" and performance. Hence the cloud may not host all the necessary services that any enterprise may need.
   

The first "theorem" could be labeled the "Google Theorem" since Google operations are an existence-proof of these two affirmations. They could be justified independently … and make perfect sense to anyone who has been in charge of running the IT department of a large-scale company. However, the same experience of being the CIO of a large company also suggests that performance and data synchronization/distribution issues will prevent quite a piece of the IS application portfolio to be moved to a cloud architecture, at least for the next few years.

Flexibility and Biology

In a previous post I have listed flexibility as one of the four pillars of agility. I defined flexibility very loosely with the following sentence: " the technical ability to cover a large span of business situations and processes with the existing set of IT services and capabilities."

This requires a little bit of further explaining. Especially, there are two ways to obtain this ability which are both relevant:

• The ability to implement change in order to achieve functional flexibility. This is foremost a matter of architecture. Hence it is the first thing that comes to mind, in the context of "SOA and Agility". This flexibility is expressed at the component level (the ability to parameterize: change the functional behavior though a change of parameters –cf. the 12^th chapter of my book on SOA), and at the "integration level", i.e. the ability to integrate a new component quickly and efficiently.
  
• The ability to adapt to a new situation "without change", that is without having to stop and modify one or more components. This kind of flexibility is a combination of "meta-data architecture", that is, the ability to change the behavior of information systems through the change of parameters and the ability to do so dynamically. This is the most important part, and the one that is most often overlooked. In many cases, changing parameters is still a new project that requires testing, synchronization and going live with a new version.
  

We can summarize by defining flexibility as the combination of functional flexibility (ease of change) and operational flexibility ( at run-time, without change to the existing binaries).

The first kind may be labeled "mechanical flexibility". The analogy makes sense because when you modify a mechanical device, you must (most often) shut it down before changing or adding a new piece. Most of what you can read about IT agility is actually of this kind. New technology, rule-based engines, meta-data parsers are all designed to extend the functional flexibility. This is definitely true of integration infrastructure technology that I have come across during the last few years. There are a few noticeable exceptions (such as using UDDI for dynamic discovery of services) but … surprisingly … they have not made it to mainstream IT yet.

I will call the second kind "organic flexibility" for obvious reasons: changing the SI without shutting it down (without the "go live" step) reminds us of living organisms which grow, change and adapt themselves without having to halt their biological processes. The boundary is subtle: everyday running systems are modified by the input of their users. So what? It turns out that for the same systems (at least those that I known of) there also exists "hidden"/admin parameters that cannot be changed without stopping and restarting the system. It also turns out that most major changes require writing new code, to re-compile, re-link or re-connect, hence a new "go live".

I must confess that I have been trained for twenty years to think about flexibility in a mechanical way. However, I now believe that the flexibility that matters the most is the organic kind. This will become truer and truer as the stringent requirements of real-time digital business will demand no shut-downs of any kind.

Organic flexibility is a matter of organization, people and processes as much as of technology. However, there is a technological part: one may not achieve this behavior (easily) with legacy technology. A key is obviously the availability of dynamic meta-data that is interpreted as parameters and modifies the behavior in a deep way. The challenge starts when different components must share a change of parameters to achieve a new behavior. This means and requires that propagating new parameters becomes a IT run-time service. The technology for this type of propagation is there, once again we find rule-based engines, scripting engine, automation tools such as processflows. If I reflect on the IT projects that I have been a part of, 80% of the energy is spent making sure that there is enough flexibility ("enough parameters") to make the component "future-proof", and 20% is spent thinking about the processes that are used to change these parameters. Designing an organic information system requires to change the ratio to the opposite.

Software Development Productivity Gains ?

As I am close to getting my last book published, I figured I should publish a few pages once in a while to trigger the interest and to gather a few comments.

This first extract deals with the matter of productivity gains in the information system. This is a complex topic, and it is really the whole of this book that provides an answer to it. We will merely highlight the most important points and indicate the chapters where they are described in greater detail. In this initial overview we will distinguish the project aspect, dealt with in this section, from the operations aspect, dealt with in the following section. These can then be broken down into three major parts: technology, industry and architecture.
Since the dawn of IT, every year has seen the advent of new technologies that promise spectacular gains in productivity[1]. In the past, people talked about programming languages with a high level of abstraction, artificial intelligence, computer assisted graphic design, synthesis and automatic program checks, etc. Today, the hot topics in the literature consist of meta-programming, service architecture, component assembly, to name but a few. We will go back over these technologies and their promises in chapter 9. There is certainly progress, in terms of function points per dollar and in terms of function points per man-day, but this progress is coming at a very slow rate. If we had to provide an order of magnitude based on the comparative analysis of the productivity figures from the late 80s, and those that are found today, I would say that we have gained a factor 2 in 20 years.[2]. It is a matter of overall productivity (on the project perimeter) according to the technical axis (in FP/man-day). The economic aspect is different, because the IT industry has undergone significant changes.

The software and system integration industry brings with it structural gains in productivity according to the following factors:

• Globalization, which makes it possible to move a part of production into countries where there is a lower cost of labor. This delocalization produces a substantial economic advantage, when it is relevant, particularly for stable needs. We shall come back to this in chapter 9.
• Mutualization of needs, which leads to the continuous release of software packages that capitalize on the needs shared by different companies. The associated gains may be quite sizeable at the level of a project, but are only moderate at the level of the average of the ISD.
• Maturing and professionalization of development practices in relation to the development of tools. The continued formalization and optimization of development processes by using standardized repositories as support[3], is a good practice that is vital for an ISD and its providers.

These productivity factors may have greater effects than technology factors, but their implementation is not easy, and requires discernment in terms of the scope of application.
The third area of progress is the information system architecture, which we distinguish from the technology section, because it involves what is known as “enterprise architecture”, a subject which spans information technology, business processes and “strategic alignment” of the information system. Enterprise architecture is used as a complexity control strategy so that the integration costs of a new application do not reach an irreversible level as the IS grows. This is mostly a defensive strategy that is used when the complexity is precisely such that integration becomes the predominant part of IT projects. As a result, the effect that is noticed is not a “productivity miracle” but the return to a “comfort zone” where integration costs are proportional to the functional complexity of the application.
The conclusion we can make at this stage of the analysis is that there are undisputed productivity factors in terms of information system development but these are not necessarily easily deployed; the gains remain measured.

The visible part of these gains is all the more modest that another part is consumed by three heavy trends of enterprise IT[4]:

• Additional requirements, as regards availability, security, ergonomics, etc., which induces more complex IT solutions with an equivalent functional perimeter.
• Inter-operability requirements, whether within the company or in conjunction with partner companies. Enterprise IT has been experiencing a double revolution for 20 years. Firstly, its functional scope grew along with inter-application connectivity requirements (unlike departmental IT in the 70s and 80s). Secondly, the concept of the extended company arose, i.e. the necessity to connect the information systems of companies that are collaborating on processes.
• The abstraction of software functions to control complexity. Controlling complexity leads to the creation of layered software, with the layers corresponding to different levels of abstraction, and makes it possible to have them evolve without having to get a sense of them as a whole[5]. These methods make it possible to control application systems that are very rich and complex but generate additional costs (the actual size exceeds the expected size).

Users of general-public software such as office suite tools will have noticed that more and more resources are required to run their favorite applications (e.g. word processing), mostly due to these three reasons.

[1] We have already referred to F. Brooks’ article entitled “No Silver Bullet: Essence and Accidents of Software Engineering”, which examines the intrinsic difficulties in terms of software productivity. This article can be found in T. DeMarco and T. Lister’s Software State-of-the-art, which includes a selection of the best articles from the late 80s.
[2] This assessment is an average ratio which hides a great level of variability, according to the type of problems to be dealt with. This will become clearer when inter-company benchmark test results are published every year.
[3] The repository which is gradually becoming the standard one in IT project development is CMMI (Capability Maturity Model Integration). For more on CMMI, see the first section of M.B. Chrissis, M. Konrad and S. Shrum’s CMMI – Guidelines for Process Integration and Product Improvement. You can also find many sources online, including those of the SEI (Software Engineering Institute).
[4] This phenomenon is more commonly known in technology circles as the “rebound effect”. For example, the efficiency increase for heating a square foot of living space is offset by the increase in the average house size; the progress in car fuel consumption has helped to develop the usage, instead of simply generating savings, etc. In IT, the “complexity race” absorbs a proportion of the gains by the rebound effect.
[5] The concept of “layers” is hard for non-computer specialists to grasp. It is quite relevant to compare it to geology: layers corresponding to age, with a successive deposit of “functions”. For example, I have the experience of precisely analyzing the code of a major American publisher on typical functions whose performances seemed to be disappointing. Using the debugger as an analysis tool showed several “layers” which have been built up as the software has undergone different changes (from an initial effective kernel to multiple generic interfaces). This concept of “sedimentation” will be introduced in section 2.4.1.