Business Intelligence Network - The Vision for BI and Beyond

ADVERTISEMENT

Intelligent Integration Depends on Business Intelligence, Part 1

Published: 5 December 2007

This article covers the issues of integration and how you can successfully tackle these issues - in both a service-oriented architecture and business intelligence context.

There’s been lots of talk over the past few years about business integration – mostly under the banner of SOA (service-oriented architecture). I’ve not been blameless in this area either, as can be seen from my previous articles on the BI Network.

One question about SOA and business integration that comes up regularly is: It’s a nice concept, but what about the implementation? We’ve tried many IT integration projects over the years, and we’ve found that they’re big, slow, expensive and not very popular with the business. What is different about SOA that will make the integration project work this time?

By implication, this question suggests that SOA provides a new set of technological tricks that should make business integration easier. However true that may be, it’s widely accepted that most integration projects fail not for technical reasons but because of underlying project management, organizational and related issues. In this article, I’ll examine these issues and remind you where – and how – you may have tackled them successfully before… when you were building your business intelligence environment and, in particular, your enterprise data warehouse.

The Trouble with Integration

By popular acclaim, the following are the five biggest issues (in chronological order of appearance during the task) encountered when undertaking an integration project:

1. Short-term business needs compete with strategic integration attempts
   
   
2. Prioritization of infrastructure falls foul of rivalries between business departments
   
   
3. Analysis of how existing applications actually work (as opposed to how they are believed or designed to) takes far longer than planned
   
   
4. Initial setup of the infrastructure takes too long and delays business value
   
   
5. Business commitment either fades over time or disappears with the sponsoring executive

Of these, only points three and perhaps four are in any way related to technological issues. Conspicuous by its absence is reference to problems with the actual technology chosen. The underlying source of all of the above issues is organizational in nature. And, interestingly, anybody who has been involved in an enterprise data warehouse project will certainly recognize this. Let’s take a look at each in turn, in both an SOA and a business intelligence context.

Short-term business needs compete with strategic integration attempts. Business executives are often more motivated by achieving short-term business goals or solving immediate business problems than by aiming for longer-term success. Such short-sightedness, driven mainly by stock price mania, leads to IT investing in quick-fix solutions which seldom work with anything else in the organization and often contradict prior strategic technology choices.

Service-oriented approaches do offer relief in the future as products become more interoperable through the defined functional and data interfaces that services expose. However, that doesn’t immediately help with the vast majority of existing applications. So, as we’ve discovered in data warehousing projects, working the organizational politics is key to addressing this issue. The integration manager must be constantly vigilant against emerging business problems or needs; and proactively include at least some of the required solutions within the framework of the new infrastructure.

Prioritization of infrastructure falls foul of rivalries between business departments. In the early stages of the roll-out of a new infrastructure, different business departments obtain differing levels of value at varying stages of the process. Successful data warehousing projects have learned that creating a steering committee consisting of representatives of all the business departments and IT is the best approach. This can create a cooperative environment wherein each business department can see their competing needs as well as the technological realities of a staged implementation and then agree to necessary compromises.

Analysis of how existing applications actually work takes far longer than planned. Old hands at business intelligence have learned this from long experience. There’s no magic answer to this, except to build in sufficient contingency in the plan from the beginning. Data profiling tools have emerged in the data warehousing market that can be used to find and analyse unexpected relationships or discrepancies in the back-end application data. These tools become even more important in the SOA environment where real-time communication replaces batch transfers and reduces the opportunity and time for data cleansing approaches.

Initial setup of the infrastructure takes too long and delays business value. Another old lesson from data warehousing. The explosion of independent data marts in the nineties was a direct result of business discomfort with the time it typically took to create the data warehouse infrastructure. While SOA does not require a “central” data store, it does depend on a common set of infrastructure – usually called the enterprise service bus – to route, carry and manage all inter-service communication. This, in turn, is dependent on the creation of a common and well-structured set of metadata to describe the services and their interfaces.

The lesson to learn from the old business intelligence projects is that business function has to be delivered incrementally even as the infrastructure is being built. Road builders have known this forever – you have to keep the traffic flowing, however awkwardly, while you build the new highway.

Business commitment either fades over time or disappears with the sponsoring executive. While the disappearance of a sponsoring executive is something that is impossible to plan for, keeping business commitment is directly related to ongoing delivery of business value. In the case of business intelligence, this required a continual drip-feed of data marts fed from the evolving enterprise data warehouse. In a wider, SOA-based integration project, business value is also perceived at the frond end. In this case, providing integration on the glass in a portal is usually the best way to give users early benefits of partial, fixed and temporary integration while the full, flexible and robust version is being built in the background via the enterprise service bus.

The Staged Rollout Plan

Many data warehousing projects have been successful by creating an incremental, staged approach to delivering a combination of business function and infrastructure over a period of a number of years. The knowledge and skills gained in these business intelligence projects can be reused in the wider SOA integration effort and promises a greater chance of success. Such an approach is depicted in Figure 1.

The staged roll-out consists of a series of related projects, each of which is designed to deliver a combination of required business function (upward-pointing blue arrows) and a part of the infrastructure (downward-pointing green arrows). The width of the arrows is notionally proportional to the percentage of function delivered in each direction. It can be seen that early projects deliver a larger payload of infrastructure versus business function, while the opposite is true for later projects. As a consequence, we can see that the infrastructure ramps up rather quickly in the early stages, but even from the beginning is supporting the delivery of real business value. As the infrastructure becomes more complete, business function can be delivered more readily.

Figure 1: Rolling Out an Integration Infrastructure

While this is easy to say, there exists significant, underlying complexity and a set of interdependencies in practice. From a business point of view, the business needs are delivered in stages, so some parts of the business must wait far longer than others for their requirements to be satisfied, while other parts of the business receive function based on a sub-optimal subset of the infrastructure. This requires that the business be involved in planning the roll-out and to agree to its staging. Similarly, from a technological viewpoint, there are interdependencies within the infrastructure that dictate the order of delivery of particular pieces of function.

The rather innocuous-looking box on the left of the diagram labelled “Initiation and POC (Proof of Concept)” is where these complexities and interdependencies must be ironed out. This initial piece of work is designed to bring together all players in the overall process – both business and IT – to agree and document a roadmap going forward over the coming projects that balances the needs of the various players. This roadmap is the most important deliverable of the project for management, although the second deliverable, the proof of concept often gets more visibility with users.

Next Up

Having explored the overall similarities between SOA integration projects and enterprise data warehousing projects in this article, I’ll be diving into some more specific skills, techniques and knowledge that business intelligence can bring to the broader integration process in the next article in this series.

If you found this article helpful and would like to receive the latest insights each month from Mike Ferguson and other experts featured on the Business Intelligence Network, please subscribe to the UK Business Intelligence Network Newsletter.