I have been focused of late (a rare luxury for me) on exploring the integration technologies to the emerging cloud computing providers — especially the SaaS guys. Last week I attended Dreamforce 2009, which was an absolutely fantastic experience. Salesforce.com has posted a bunch of the material on the site, and it is worthwhile.
While at Dreamforce, I attended training on integration with Salesforce.com, both to and from. I enjoyed the class, and found the relatively rich set of alternatives (remember that this is a very young company that has quadrupled its workforce in the last few years) to be a good toolbox for integration. One of the more intriguing aspects is how the company’s engineers balance control of service levels (i.e. their ability to prevent you from hurting their performance when you integrate) with access. One of their key technologies is a set of “governors” to control the resources available.
While I really enjoyed the exposure to the API-level integration, I cannot help but compare all of this to how we traditionally do integration between business partners, as well as to SAP, which is via standard documents. Apparently I’m not alone, as Benoit Lheureux of Gartner wrote a thoughtful blog post on how cloud integration will affect traditional integration.
From my point of view, API level integration (and yes, this includes web services), is usually more powerful and functional than integration driven through documents (disclaimer — yes, I realize at its heart that web services is basically the exchange of XML documents over HTTP, but for this discussion, it feels like an API…), but, document driven exchange is easier, more interoperable, and often enjoys much higher adoption. And a major part of the “document advantage” is the ability to separate the handling of the document from “transport” (communications), which often allows use of familiar technologies to perform the integration.
For my purposes, API level integration is the calling of specific low level services, with a set of arguments, often — though not always — using web services. Although web services is “standard”, the service calls are typically specific to a given SaaS provider (i.e. proprietary). Additionally, the service calls may change based on the configurations done for a given instance setup for a customer (i.e. custom proprietary).
Document level integration, in contrast, uses the basic pattern of sending in a document in a defined format, often — again, not always — in XML. This document may vary from customer to customer, but most of it is the same, and customers can often submit the document in a variety of ways (web services, ftp, etc).
SAP is a pretty popular ERP system in the current world, and it supports a wide variety of integration technologies. In our B2B managed services worlds, IDocs over ALE is by far the most common integration technology — despite the fact that there are often very good reasons to use a lower level approach (directly invoking RFCs after sending data using FTP, for instance).
Why? Among many, many other reasons, customers and integrators like solutions that work predictably across multiple environments. IDocs, like X12, EDIFACT, OAG, etc, are defined entities that do not change based on adding fields or custom logic to an application. But possibly a bigger reason is the ability to use existing technology and processes to perform the work. SAP IDocs can be manipulated using standard translation technology, and then sent via ALE or other communications methods. The ALE protocol can be implemented in a comms gateway that you already own.
Modern communications gateways have extensive support for web services today, but that is really a kind of toolbox for building custom adapters to custom services, with each one a “one-off”. This problem can be intensified if the service you are connecting to changes over time as additional data fields are added to objects (a product object is especially susceptible to change).
API level integration is usually much more brittle for this reason, and it is one of the characteristics that led many enterprises to switch to message oriented middleware, and attempt to impose canonical standards (“canonical standards” is a fancy way of saying a common format — usually in XML — for frequently used documents, like customer; canonical standards are often adopted from outside, especially the OAGi standards). Integrating to a single system via API is often fun, maintaining dozens or hundreds of these is not.
A common pattern is for emerging technology categories to start with low level APIs, and gradually build up to more abstract interfaces that are easier to use and less brittle. Already, in the case of Salesforce, they are offering a bulk load service, and they are also offering a “meta-data” API that allows tooling to simplify integration. Over time, I fully expect that most major SaaS providers will provide integration methods that feel a lot like trading documents, and that B2B teams will take the lead in using them.
In the long battle between document and API integration, document style integration will dominate, though API and service level integration will play critical roles…