SOA is much too young to be dead

Today I'll look into the SOA controversy that was generated by Anne Thomas Manes's SOA's obituary. 

This may look like a slow response (90% of the blogosphere has already published their comments) but I will take some length to explain the issue (and this required a little time for thinking over). Surprisingly, one still get a very fuzzy and abstract idea about what SOA from reading all the positive answers to Ann Thomas Manes.

What is much clearer are the benefits one should expect from SOA (and they have been clear from 2003). SOA brings increased agility and reduced costs. The increase agility comes from reusability an the cost reduction comes from sharing services. Reusability comes in two flavors : direct (reuse of an existing service) and indirect=mash-up (quickly mixing existing services to produce a new one).

There are additional benefits (such as the ease to mix internal and outsourced services) but these two are enough to get anyone's attention. The question is: how to get there ?

1. SOA is a 3 step process

First, we decompose SOA into three steps ("doing SOA" means doing all three in sequence):

1. Service Definition
2. Service Architecture
3. Service Integration
   

Service Definition simply means to come up with the catalog of services. It is a "business oriented" step: the business processes are analyzed and the relevant business services are identified. At first it looks like the "simple" (nothing but simple actually) first part of a functional analysis of a new system. There is (much) more to it, because the role/user analysis is an integral part. The focus on reusability and agility starts here. Hence it is necessary to play with business scenarios, which further means that it is not an information system play. In the "acronym soup", the relevant terms are EA (Enterprise Architecture) and SOBA (Service Oriented Business Architecture) which have been introduced later on to insist on the "business/enterprise" dimension.

Service Architecture is the technical step that takes a large catalog as an input and produces a structured hierarchy as an output. A list of thousands of equally visible services is in no way a feasible approach to manage a large information system. Classical issues such as modularity, encapsulation, visibility, etc. must be taken into consideration. I write "classical" because this is the heart of software engineering and software architecture. Re-factoring techniques (the same that are used when writing a class or a component library) must be applied to produce services that have a better chance of being reused or being combined. I won't dwell into it today, but the role of data & models architecture is key (as I said, this step is the IS step).

Service Integration is what most people think about when they say SOA: linking all the services into an integration infrastructure. This is where all the technologic  issues live (ESB or not, SOAP or REST, etc.). The term SOI (Service Oriented Integration - for instance, see this definition) was coined to denote this step.

These three steps help to grasp the huge difference between "SOA in the large" (with a large enterprise as its scope) and "SOA in the small" (SOA for one information system, either for a small company or one department). The effort repartition obviously varies, but here is the pattern from my own experience:

• In the large: 40/30/30  (i.e., 40% of the effort is the definition part)
• In the small: 20/10/70

It is interesting to notice that there is (almost) nothing new with "SOA in the small": service definition + architecture + flexible implementation is what has been making a good piece of software for 30 years. 

2. What is easy/hard ?

Service Definition is hard when the scale gets big. Although producing the service catalog is straightforward for a department-size project, there are mutiple issues of heterogeneity and multiplicity of stakeholders that makes it much more complex to manage at the (large) entreprise level. Not only the management of the catalog gets harder as the size grows, but there is a subtle balance to be found between "common sense" and "formal methods". There are indeed methods/pattern/frameworks available for this first step but:

• if this step is too formal, too many stakeholders are discouraged
  
• if it is informal, the process crumble under its own weight.
  

The same is even truer with Service Architecture. It is quite simple when doing "SOA in the small" but gets harder as the size grows. Contrary to Service Definition, there is little help available as a methodology: hard learned IT experience is necessary. Unfortunately, this is not something that one learns through writing Powerpoints. If that step is poorly executed, the integration is more difficult, but - and this is the key point - little reusability or agility is observed later on. This is something that I am trying to teach during my "Information System Course" at Polytechnique, but this is a hard, technical topic which is better acquired through practice than theory.

Service Integration is actually not so difficult nowadays, because the technology is quite mature. I am not saying that it is easy, but it would be sad to ignore the progress that has been delivered by technology providers. There are still a few difficult issues (cf. next section) and, more generally, a large-scale integration project still exhibits all the possible pitfalls of a complex IT project, but there is nothing new here. Actually, when done "in the small", this is straightforward and there are so many success stories to prove it.

3. What is the maturity state of SOA ?

Using the same decomposition, here is my own assessment: 

1. 50% for Service Definition. This step is rather well explained, many good books are available (such as "SOA for profit"). It is still a difficult job at the enterprise level. There is indeed a governance issue (those who mock the "governance issue" have missed what Enterprise Architectuire is, and are still trying to accomplish "small scale SOI"). There is also a "pedagogy issue": an entreprise-scale effort cannot be undertaken if its meaning is not understood, by all stakeholders.
   
   
2. 20% for Service Architecture. There are very few books that can help, and even fewer are directly relevant for SOA. The only one that I recommend to my students is "SOA, Le guide de l'architecte." (in French, sorry for my English speaking readers). I have tens of books about IT/software architecture in my private library, including great pieces from Fred Cummins, Len Bass, Richard Shuey, Paul Clements, Jean-Paul Meinadier, Jacques Printz, Robert Seacord, Roel Wieringa - to name a few -, but none of them (including my own two books :)) adress the issue of Service Architecture "in the large" thoroughly.
   
   
3. 70% for Service Integration. The technology is there, it is scalable and proven (WebSphere or Weblogic being two classical commercial examples). Obviously, there are still technical issues. For instance,

• distributed transaction & synchronisation is still a hard problem.
  
• performance overheads (cf. SOA is not scale-free) still exist and makes the deployment tricky when response time is an issue.
  
• monitoring & autonomic load balancing is difficult and/or not sufficiently developed.
  
• service discovery and pub/sub architecture (EOA) are not as straightforward to implement as one might wish

But those are hard issues, nothing to be ashamed of :) 

4. SOA is a practice, not a target or a technology

As David Linthicum said: "SOA is something you do, not something you buy". It is not something that you do once (a project) but that you do all the time, continuously. 

This is what brought the creation of the sustainable IT architecture approach. Done right, SOA is a practice that transforms information systems for the better. According to the money that you invest, to the legacy and to the business constraints, the transformation is slower or faster, but it takes time and is never ended. The sustainable IT approach comes from a change of perspective: instead of always focusing on what is to come, on the new projects, one should search for more value from the "existing assets". Another "sustainable" principle is to save your strengths to last longer (hence the continuous improvement approach as oppose to the big bang).

It's a "culture thing". Part of it (cf. Service definition) is everybody's concern, hence it must be simple and must be pushed at the CxO level. Therefore, IT governance is a key component for changing the culture.

I will conclude by stating that SOA represents a "Chinese strategy" for IT management, in sense of François Julien. I could not urge you enough to read his book, but if you have not, a key principle is to distinguish between planned strategy (top-down, in the Greek fashion) and "building up your situation potential". A Chinese proverb says than one does not grow a plant faster by pulling its stem.

Similarly, instead of pushing "SOA technology projects", a better approach is to build up your "situation potential" : build a common data model, train everyone, build a shared consensus about what the service catalog should be, etc. Each project is then seen as an opportunity to move one step in the right direction.