Developing a Service-Oriented Architecture (SOA) for Business Intelligence

Originally published 20 June 2007

In this article, I would like to look at what is involved in developing a service-oriented architecture (SOA) for business intelligence (BI) in order to make it easier to access business intelligence from processes, applications, portals and search engines inside and outside the enterprise. In order to do this, it is, of course, necessary to first provide my definition of a SOA and then look at key technology components of a SOA before turning my attention to how business intelligence fits into this context. I will then look at types of BI web services that can be developed and deployed and also at what is involved in designing BI services to facilitate maximum use of intelligence in an SOA.

What Is an SOA?
A service-oriented architecture provides a standardised, flexible approach to enterprise business process integration and role-based access to process activities, information (including BI), business transactions and collaboration tools. An SOA makes it possible to separate processes from applications and to create on-demand and event driven information, application and collaborative services that can be invoked in an industry standard way. These services can then be rapidly assembled into composite applications associated with individual process activities in an industry standard way.

Therefore, an SOA is effectively a blueprint for integrating all aspects of enterprise computing to support business integration. It, therefore, should support several types of services including:

  • Business process services (whole processes published as services)
  • Operational application services
  • Business intelligence services
  • Content (documents, web content, digital media) services
  • Collaboration (e-mail, web chat, instant messaging, net conference rooms, calendar, etc.) services
  • Data integration services 
  • Common services (e.g., security, search, business rules)

Also, an SOA should provide common technologies to support integration at five key levels. These five levels of integration are shown in Figure 1. They are: user interface integration, people integration, business process integration, application integration and data integration.

Why SOA?
There are many reasons why companied are investing in SOA initiatives at present. The obvious one is that SOA reduces operational costs and increases efficiency, improves effectiveness and increases collaboration. Improving efficiency is typically achieved by standardising on common business processes that have been separated from applications and then integrating and automating them by mapping process activities to applications within the enterprise and across businesses. An SOA can be used to improve effectiveness by leveraging business intelligence everywhere in order to guide employees during every performed activity so that everyone contributes to strategic objectives and executes on a common business strategy. It is also possible to monitor events to automatically detect/predict problems and opportunities for rapid response and business (re-)optimisation. Improving collaboration allows employees, partners, customers and suppliers access to team workspaces where they can share information and services. With collaboration services, people can also be located rapidly and communication can be in real-time.

Figure 1

In the context of the five levels of integration defined in Figure 1, the respective corresponding component technologies you would find in an SOA for each of these integration levels are as follows:

  • Enterprise portal software (user interface integration)
  • Collaboration tools
  • Business process management software (process modelling, execution and monitoring)
  • Enterprise service bus (ESB) and enterprise service registry
  • Data integration software (e.g., ETL, EII, MDM and enterprise content management)

These technologies form the core infrastructure software in an SOA and, therefore, they should be treated as common infrastructure technologies that are shared across the enterprise and that are used repeatedly by different business projects that require integration.

Looking at these technologies, it is the enterprise service bus and enterprise service registry that form the backbone of an SOA. The ESB is the message broker that handles the routing and translation of XML messages as they flow through the enterprise. The enterprise service registry is a central metadata repository for centrally managing services and related metadata artifacts. Services are published to the registry and are then subjected to robust life cycle management. A service registry integrates service metadata to provide a single, comprehensive description of each service and provides much more than basic universal description, discovery and integration (UDDI) registries. Typically, enterprise service registries provide management tools to:

  • Search and categorize services and other SOA artifacts
  • Control service visibility and security
  • Manage versions of services and other SOA artifacts
  • Manage service interfaces and service level agreements
  • Provide impact analysis on proposed changes
  • Monitor service usage

Also, enterprise service registries can integrate with other tools and infrastructure such as ESB products, business process execution language (BPEL) process servers and IDE development tools. Enterprise service registry product examples include: 

Service registry products should support a number of standards in order to be able to manage multiple types of artefacts. These include:

  • UDDI 3.0 
  • XACML for Role Based Access Control Policies 
  • SOAP with Attachments
  • REST 
  • WSDL 
  • ebXML Registry 3.0 
  • BPEL
  • XML Signature
  • XSLT
  • Web Services Security
    • SOAP Message Security and SOAP Message with Attachments (SwA) Profile 
  • WS-I: Basic Security Profile
  • WS-I: Basic Profile
  • SAML

Supporting these standards means that all of these “artifacts” can be typically be stored in a service registry. The service registry is therefore a focal point in an SOA and will be constantly accessed in order to see what services exist, where they are located and how to communicate with them. Many types of services can run in an SOA and are all likely to be managed by a service registry. These include user interfaces visualisation services (e.g., remote portlets accessed by a portal), business process services (the whole processes that can be invoked in a single service call), operational application services, collaboration services (e.g., e-mail, instant message, shared calendar, video conference) and content services (e.g., documents, web pages, digital media). In addition, with respect to BI, it includes BI, performance management (PM) and data integration (DI) services. BI, PM and DI services can be accessed by integration software at all five levels via the ESB. This means that BI, PM and DI services need to be used with integration software in an SOA to maximise business benefit and empower everyone with pervasive business intelligence.

The kind of BI, performance management and data integration artefacts that can be developed and published as web services include:

  • Queries
  • Reports
  • OLAP slice services (MDX queries)
  • Scoring and predictive models
  • Alerts
  • Scorecards
  • Budgets
  • Plans
  • BAM agents
  • Decisions (i.e., automated decision services)
  • Data integration workflows and federated queries

BI Service Design
However, it is not just about constructing many different BI, PM and DI services almost at random hoping you have created something useful. There has got to be something that dictates design of these services to put them into a business context. In my opinion, if you look back at Figure 1, it is business processes that are going to dictate this context (and therefore BI, PM and DI service design), because processes models and the activities within them can guide you in deciding what BI, PM and DI services you need to create. It follows, therefore, that if we are going to get more precise in the delivery of on-demand intelligence in a SOA, then understanding business processes is critical in determining the right BI services to create. More specifically, understanding business process activities helps to determine which BI services to associate with each process activity. This is the key to so-called “right-time” BI. If we are heading toward a process-centric enterprise whereby a user can sit down and choose the task they want to perform (where the task is part of a process), then we can tie specific BI services to that task in order to deliver “right-time” business intelligence on-demand.

The other main contributor in dictating what BI, PM and DI services to design and create is the role of the user. Understanding user roles is also, therefore, very important. For example, I did some work for a shipping company and wondered why port managers were complaining that they did not have access to business intelligence even though it was available via a browser at their desktop. It wasn’t until I went to see a port manager that I realised the problem. The problem was that they were never at their desks! Most of the time in their days was taken up by being out on the port barking orders at people and supervising the day-to-day loading and unloading of cargo on ships. It wasn’t until I understood the role better that I realised that what they actually wanted was BI alerts on their mobile devices with occasional mobile access to basic practical intelligence. Occasionally they might follow up an alert by ringing back into their office and asking an administrator to check something on a report. Therefore, knowledge of process activities plus user roles are critical success factors in the delivery of business intelligence in an SOA. The other thing to realise in a SOA is that people may need a lot more than BI services to perform a task in a business process. They may need a combination of BI services, operational application services, collaborative services etc. to perform a task. Another term for this combination of services is a “composite application”. A composite application may be a series of activities orchestrated in a process and associated with a specific business task (e.g., processing a quote request from a broker in insurance). To do this, an underwriter may need to perform a series of tasks before getting back to the broker, and each of these tasks may require access to operational services, BI services, etc. Understanding this in detail is critical to practical use of on-demand BI, PM and DI services in an SOA.

Staying with insurance, Figure 2 shows an example of an insurance underwriter’s portal with access to services via an enterprise service bus. This is role based computing brought about through portal personalisation in an SOA.

Figure 2

Looking at this, it becomes possible to quickly focus on two things: the role of the user (an underwriter) and the activities (tasks) they perform. Figure 2 shows that an underwriter needs access to operational application services, collaboration services and BI/PM services in order to do his or her job. So right away there is a need for more than business intelligence. With respect to BI and PM services, we can quickly zoom in to what services we need to develop and make available by looking at underwriter tasks. Here you can see two tasks highlighted. The first is policy renewal and the second is new quote request. Looking at each of these tasks, we can zoom in on the precise BI services required to support the underwriter in making the decisions needed with respect to these activities. Of course, a skilled underwriter would use his or her own expertise on top of this to make any final decisions. The point here, however, is that for policy renewal, an underwriter has BI at his/her fingertips at the time he/she performs the renewal. This includes a daily renewals report, an automated rating recommendation for a specific customer and a report about that customer’s loss ratio and recent claims incurred. Equally, when performing a new quote request, the underwriter has access to a claims history report for any claims received and the loss ratio associated with the customer wanting the quote (if the customer is already on the insurance book for another risk), claims on similar risks (if this is a brand-new customer) and an automatic rating recommendation. Therefore, it is not just about what BI and PM services are needed for an underwriter, it is what BI and PM services are needed for an underwriter when performing a specific underwriter task. It clearly follows that if you do not go to the trouble of leaning what roles you have in your enterprise and what tasks people perform, you can get nowhere near the level of precision needed when delivering on-demand BI services in an SOA.

Creating BI, PM and DI Services
As I said earlier, a number of different kinds of BI, PM and DI services can potentially be created, and the good news is that several of the main BI vendors are now supporting BI, PM and DI services. Independent BI vendors such as Business Objects, Cognos, Information Builders and SAS, for example, already have support for web services in their products. IBM and Microsoft and Oracle (Hyperion) also have web service support. Often, this means that BI reports, etc., can be invoked via SOAP requests – but in the case of some products, the tooling to point, click and automatically publish a report as a service to a specific enterprise service registry is not yet released. Some vendors are now rolling out the tooling to publish newly created BI artefacts (e.g., reports) as services in the latest releases of their products. MDX requests can be invoked via an XML/A payload in a SOAP message to render OLAP slices from popular OLAP Servers such as Microsoft SQL Server Analysis Services and Oracle (Hyperion) Essbase. Also, ETL jobs and EII federated queries can now be published as web services. For example, IBM Information Services Director, which is part of the IBM Information Server, can publish data integration workflows as services, as can Informatica. So the capability is available.

However, if you are going to manage this well, then BI developers need to adhere to standard enterprise-wide policies and procedures associated with SOA governance so as to:

  • Develop services using agreed standard interfaces
  • Share and re-use services
  • Categorise services to keep them organised
  • Manage versions of services in a life cycle
  • Adhere to security standards
  • Design services according to enterprise agreed standard patterns
  • Test services in a standard way
  • Publish services to a production environment in a standard way

This means that BI, PM and DI services should be published to an enterprise service registry for service life cycle management and service security management, and to make them available to the wide range of technologies that now integrate with enterprise service registries (e.g., IDE development tools, business process engines, portal servers, search, etc.). This is shown in Figure 3 which includes Software AG’s CentreSite as an example service registry. Ultimately, the intention is to place BI services at the centre of the enterprise for access by anything that wants it.

Figure 3

Figure 4 shows this idea with BI services being accessible for portals, processes, operational applications, performance management scorecards, search engines and office applications all via an enterprise service bus.

Figure 4 

Testing BI Services in an SOA
One of the major differences with respect to BI in an SOA versus standalone BI systems is that BI services may need to be combined with other services to create composite applications for use in business tasks. Therefore the job of creating testing and deploying a report for business users as per a classic BI system development is not the end of the story. When it comes to SOA, you can look upon BI service developers as ‘component’ developers in a way. They tend not to be concerned with the bigger integrated picture. I am now seeing the emergence of two kinds of developers among several of my clients. These are:

  • Component developers
  • Integration developers

The former build the BI, PM and DI services (or indeed other components such as portlets, operational application services, etc.), while the latter are tasked with integrating and testing these components to make sure they work together correctly in an SOA. Figure 5 shows an example of component and integration testing and suggests that we need to consider multiple test environments in order to thoroughly test out what needs to be deployed to the user because BI may be only part of what needs to be rolled out.

Figure 5

Integration Options for BI and DI Services
Looking at the role of an integration developer, he or she has multiple options when integrating BI, PM and DI services into a solution targeted at a specific role. These options are shown in Figures 6 and 7.

Figure 6

Figure 7

All of these are possible. The challenge for us is to work out what makes sense. Please refer to some of my previous articles on the B-Eye-Network: Techniques for Integrating Business Intelligence into the Enterprise, Part 3; Techniques for Integrating Business Intelligence into the Enterprise, Part 4; and Decisions at Your Service.

In conclusion then, BI and data integration services can be created today on a number of BI platforms and data integration products. The challenge is what to build and for what business reasons. Understanding business processes, process activities and business roles is fundamental to successfully deploying BI and DI services in the correct business context, and deployment requires using both business intelligence and business integration software together. Also, BI and DI services need to be published to an enterprise service registry for governance and life cycle management. Finally, integration developers as well as BI developers are needed in order to coordinate re-use of BI and DI services and deliver an integrated solution that a business can use to improve its business performance.

Recent articles by Mike Ferguson



Want to post a comment? Login or become a member today!

Be the first to comment!