(Article URL: http://www.b-eye-network.co.uk/view-articles/4066)
My last article focused on the demand for “real time” business intelligence (BI) and contrasted its uses with that of business activity monitoring (BAM) solutions. I also addressed how, in an enterprise application integration (EAI) environment, both solutions could be driven by EAI messaging technology without the need for an additional extract, transform and load (ETL) tool. Suggested benefits were lower costs, reduced effort, and the integration of BAM and BI reporting. However, the article also highlighted a number of potential problems when attempting to achieve this level of integration within an existing EAI environment. These included the granularity of the existing messages, lack of integration across processes or business units, gaps in the integration environment, data quality and completeness, and the scope of the existing integration environment.
Let’s drill further in to some of these issues – exploring the practicalities of implementing an event driven architecture which, in addition to enterprise integration, will also deliver both BAM and business intelligence. Although this architecture may not carry the overhead of an ETL tool, it does require significant investment in artefacts such as enterprise data models, business event models, master data services, and data stewardship structures.
Capturing Business Events
Messages created for application integration may not be suitable for either BAM or business intelligence. In general, the messages are designed around the functionality of the application rather than for exposing “seminal” business events for information processing. To achieve true “enterprise” integration, as opposed to application integration,– the published events must be made independent of the existing systems and applications, and be related instead to the logical events and processes of the business – for example, what a business must do to deliver its products and services.
In energy trading, for example, the events could be derived from analysing the “end to end” life cycle of a trade. The advantage of adopting this integration architecture is that all the seminal events of the business would now be published and available for analysis, supporting both BAM and business intelligence.
The second advantage? By relating message events to the business rather than the application’s need, published messages are independent of the currently deployed applications; applications can be replaced or added to the integration environment as and when required without causing a significant change in the integration architecture. The “integration” costs of implementing new applications, often a major element of the overall costs, should then be significantly reduced.
The energy industry also benefits from this integration architecture because of its ability to integrate across divisions that traditionally are strongly siloed. Upstream E&P needs to hand off available for sale volumes to the trading and marketing business, and the trading business needs to optimize the downstream assets, and so forth. This level of cross business unit integration is very difficult to achieve without the notion of business events and activities being modelled and exposed to these integration techniques.
Delivering the Enterprise Architecture
Clearly, there are numerous problems associated with implementing an event driven integration architecture. First, the seminal events must be derived by analysing the logical, not physical, processes required to deliver a particular business service; this is a “top-down” activity requiring extensive input from subject matter experts. The resulting event list must then be mapped to some form of standard language or data model to ensure consistency when applied across the organisation. These events, now transposed to a common data language, must then be associated with the appropriate applications – such as, which will publish, subscribe, request or respond to each message? Having achieved the mapping and created the message structures, usually in XML, the appropriate messages must then be implemented for each application.
The overall effort required to deliver this architecture is significant, and higher in the first instance than implementing a more “application” driven integration environment. The messages will, in general, be of a higher “granularity” than for application integration; an application may need to publish three messages in place of one previously, and a subscribing application may need to subscribe to, and consolidate, three messages instead of one.
A second issue is that messages will now be published, or message “services” exposed, for which there may not currently be a subscriber. This “just in case” deliverable may be difficult to justify. Additionally, messages in an event-driven environment are highly structured, related to a canonical business model, created using standards such as XML, SOAP and possibly WSDL, and designed to meet the needs of BAM and business intelligence. Thus, they will not be the quickest to implement or the most efficient, in terms of raw data throughput, when in operation. These messages tend to carry a very high structural overhead in relation to the amount of data actually carried, and much of the data transferred may not be new or changed. This high payload per message, combined with the increased number of messages, may require an expensive overhaul of the integration backbone. In instances when extremely high throughput is required – for example, for real time commodity pricing – performance requirements simply may not be met.
If the costs of implementing integration solutions were negligible, this proliferation of additional – and possibly superfluous – messages would not be an issue. But in our experience, the cost of publishing a single message can reach $400,000. Therefore, the cost to replace an existing “application” integration environment with an “event driven” enterprise architecture could be considerable. In a recent study conducted by an energy trading organisation, it was found that the cost of implementing such an architecture would total approximately $50 million. This cost was to be invested in an environment which, from an operational viewpoint, was performing satisfactorily and would not be significantly improved after the investment.
The Enterprise Model
Creating the enterprise data model required to support the integration architecture can be a considerable undertaking. As previously discussed, a canonical data model is required to isolate the message structure from the internal structure of the application. As for creating the logical event list, the enterprise model is derived following a “top down” approach by analysing how the business delivers its services. The resulting model should be independent of any existing system or physical process.
An important issue when creating the enterprise model is to define its scope and potential uses. If the model is to support all messages of an event driven architecture, its scope needs to cover all business processes from purchasing to sales, including pre-sales and marketing activities. But the uses of the model should go beyond defining the structure of messages. The same model would form the basis for an enterprise data warehouse (EDW), and of a master data server (MDS). Extending the enterprise model to support the MDS is particularly important as the structure of the master data messages would then map directly to the MDS database schema. Similarly, messages published from the EDW database would map directly to the required message structure.
The enterprise model also would be used when defining the requirements for new business applications. For example, suppliers would need to ensure that any systems they provide can interface directly with the model; and finally, it would be the basis for all communication about data within the business. This means that any systems or users, such as data marts and reporting tools that need to extract data from the EDW, would define their requirements in terms of the enterprise model.
The problems associated with delivering and adopting such an enterprise model are well documented and include the availability of resources such as data modellers and subject matter experts, overall required effort (for example, the challenge to cost justify upfront), time to delivery, and adoption in use. Time to delivery is a significant problem as the business needs to continue implementing new systems during the model development period. And given the dynamics of many organisations, it is possible that the business will undergo substantial organisational change while the model is being created, possibly rendering it obsolete before completion. There may also be a problem of adoption once the model is delivered; systems integration projects are often carried out by project teams dispersed across the organisation that may have their own timetables, budgets, and set of requirements. They also may be tempted to deliver a “point-to-point” solution based on local needs rather than the broader requirements of the enterprise.
Lack of Data and Data Quality
The availability of enterprise “master” and “reference” data is another issue to consider when adopting the integration environment for BAM and business intelligence. The data required to support these services will be broader than that required for “operational” applications integration. In many cases, the data will exist within one or more corporate systems; indeed, versions of the same data may be mastered and used independently in multiple systems, or even multiple versions of the same data will exist within a single system. But even when the data is available, or can be made available using an existing application as the source, the question of data quality remains. One can assume that the quality of the data in any environment is good enough to meet the needs of that environment; but will it be appropriate for any other use? How did the data get into the system? From where, when, and under what controls? Who owns and maintains it? Is the quality guaranteed, and are quality metrics published? Without service level agreements and published metrics, the source must be deemed unreliable. Data may change at unpredictable times, or known errors will be corrected only as required for local uses. Therefore, identifying a system that contains the required data, and extracting it as a source for other purposes, may not be a practical solution.
A Strategic Approach
Clearly, the requirements and issues surrounding the successful delivery of enterprise BAM and “real time” business intelligence, possibly based on an event driven architecture, are daunting and fraught with potential pitfalls. In our experience, there is no magic bullet to ensure the smooth and trouble free delivery of such an architecture; however, we believe the following observations may help mitigate some of the risks and issues.
Build, Maintain, and Embed the Business Case
Implementing an enterprise integration architecture, encompassing BAM and business intelligence, will be time-consuming and expensive. Creating a solid business justification must be at the foundation of such an exercise. The benefits of integrating the enterprise data must be clearly identified and planned for before the project is started. In creating the business case, the overall cost of the programme must first be identified. What will be the costs of implementing an enterprise integration environment? What will be the costs of creating the BAM and BI data hubs, the cumulative data stores, the analytics and portals? Calculating the benefits from this type of programme has traditionally been more difficult, and at least partly subjective. We can value the reduced effort needed to integrate future systems (assuming there is a reduced effort), but the value of fewer data transmission errors, or better process and performance management, is more difficult to measure.
Putting a value on the availability of real-time, reliable, relevant and usable information will always be a challenge. The business case must show that the solution is the most cost effective and that there are internal customers (sales, marketing, purchasing, or finance) committed to using the output. The case will be more robust if the use of the output is embedded within the objectives of the target customer groups. For example, a case will stand up to scrutiny if, without the proposed solution, the users would have a lower probability of meeting their own targets and objectives.
Ensure Data Stewardship Management and Delivery
The requirement to create and maintain a reliable source of corporate data, and to deliver it across the organisation, is not new – the technology to effectively implement such as solution has only recently been made available. Master data management (MDM) solutions are available from a number of vendors and are commonly implemented to provide both real time updates via a message based architecture and “batch” distribution via ETL. The objective is to deliver a “single version of the truth” to all systems of the business (operational and reporting) as and when required for effective service delivery.
But implementing the technology addresses only part of the enterprise data problem. A major element of the solution will be to ensure that the distributed data actually meets the requirements of all the consuming systems. This often will require the appointment of data stewards with responsibility for maintaining the quality of specific data items. Service-level agreements also may agree with a downstream client guaranteeing a predefined level of quality and timelines.
Create a Delivery Framework and Road Map
As discussed previously, producing an integration architecture will require the creation of a reasonably detailed enterprise data model. This is clearly an expensive and time-consuming undertaking. In our experience, the delivery of the model must be incremental and linked to a road map of client projects. The individual project would deliver the required extension to the enterprise data model covering their required scope, if not already available, and would deliver the detailed XML message structures based on the data and the event list. With this approach, there would still be a requirement for a conceptual data model to be created up front to set the scope of the incremental modelling activities, but this would not be as expensive or time-consuming as delivering the full logical model.
An alternative approach to creating a bespoke logical model is to acquire and customise an off-the-shelf industry model, or to base the model on an industry standard markup language such as Financial Products Markup Language. This provides a hierarchical model that can be used to create standard XML message structures.
Create a Governance Structure
Although a detailed road map and framework will define how and when components of the integration architecture should be delivered, a governance structure also needs to be in place. This will ensure that projects, when implementing the road map, adopt all appropriate standards and delivery methodology. However, designing the optimum governance structure can be more difficult than expected; overly complex and bureaucratic processes will cause implementation costs to balloon out of control, whereas weak processes will not have the desired effect on project deliverables.
Conclusions
This article more concretely defined some of the components required to support the implementation of an enterprise integration architecture for delivering BAM and strategic business intelligence. It also highlighted the issues and risks associated with creating these components, and discussed how these risks can be minimised. The core of the problem is that implementing such an architecture is a long term, very expensive programme with few immediate benefits. Even longer term benefits of what is essentially an information solution could be considered nonessential – and, like all such programmes, will be closely scrutinised at the next round of budget reviews.
Apart from sudden budget cuts, the biggest potential threat will come from waning interest. If the target users decide they don’t need the system after all, or forget what they wanted it for, or get reorganised, or do something tactical instead, then the programme is in serious danger. The lesson learned is that a business case for any BAM or business intelligence system must be constantly maintained; benefits must be sold and resold and re-imprinted in the minds and written objectives of the target user groups.
Another trap to avoid is investing heavily in up front “big ticket” artefacts. Wherever possible, the development of data models, architectures, and documentation should be spread over the delivery of the individual project. “Up-front” activities can be limited to defining strategy, concepts, governance and road maps. And finally, we believe it is important to implement a series of smaller deliverables at reasonable intervals rather than going for the big bang solution at the end of the programme. But at all costs, avoid the “quick win” or “proof of concept” followed by little else for a considerable time.
John is a principal with Knightsbridge Solutions. He has more than 20 years of experience in building strategic solutions in the area of business intelligence, data mining and customer retention for the energy, telecommunications and finance industries. He holds a bachelor's degree in engineering and a masters in business administration from