Business Intelligence Network - The Vision for BI and Beyond

Web 2.0 and the Highly Evolved Business, Part 1

Published: 6 June 2007

This series of articles will examine Web 2.0 and its relevance to business intelligence.

In previous articles (see: Business Intelligence is Dead – Long Live the Highly Evolved Business Part 1, Part 2 and Part 3), I’ve used the concept of the highly evolved business (HEB) to explore how a service-oriented architecture (SOA) will influence the evolution and direction of business intelligence (BI). This series uses the same foundation to examine Web 2.0 and its relevance to business intelligence.

As discussed previously, a highly evolved business is one where the operational, informational and collaborative environments as well as the people, process and information layers are seamlessly interconnected and working together to provide a single, integrated and flexible IT infrastructure to support on-demand business. This is illustrated in Figure 1.

Figure 1: The Highly Evolved Business

What is Web 2.0?
Mike Chambers of Adobe sums up much of the consensus about the meaning of Web 2.0 with the following definition: “a term often applied to a perceived ongoing transition of the World Wide Web from a collection of websites to a full-fledged computing platform serving web applications to end users. Ultimately Web 2.0 services are expected to replace desktop computing applications for many purposes.”

We’ll need a little more detail to compare Web 2.0 to the HEB concept, but this definition should ring a warning bell for business intelligence, as well as other applications. The second sentence counsels that Web 2.0 services may replace desktop applications, so we may infer that this new Web paradigm could be important in the evolution of business intelligence – but how?

Let’s dig a little deeper into Web 2.0 as described in Wikipedia. The phrase was coined by O'Reilly Media in 2004 and refers to a perceived second generation of web-based communities and hosted services (e.g., social networking sites, wikis, blogs and mashups) that facilitate collaboration and sharing between users. The use of “2.0” implies a new technology version of the Web. However, this is not the case. In fact, it refers to changes in the ways systems developers and the public use and interact with the web platform. According to Tim O'Reilly, "Web 2.0 is the business revolution in the computer industry caused by the move to the internet as platform, and an attempt to understand the rules for success on that new platform."

Note that this is a “business revolution” according to O’Reilly. Although I suspect that “evolution” is turning out to be a better description of the way it’s happening, the key point is that Web 2.0 represents a change in the way business works. That change revolves around the empowerment of users. In this new world, communication, interaction and collaboration between individuals are moving from a centralised and controlled model to a decentralised and chaotic approach. This philosophical change of viewpoint is at least as important as the details of the new application types.

Business Intelligence and the Web 2.0 Mind-Set
One of the key phrases in the original data warehousing lexicon was “a single version of the truth”, the implication being that truth did indeed exist in a single version and that such a single truth could be captured in the data warehouse. Of course, we all recognised that there could be multiple versions of the truth. A classic example is the differences between sales counted in, say, a specific month, as recognised by the salespeople, the accountants and the legal team in a single organisation. Each of these groups is likely to hold a different value as their truth. None of them is technically wrong, they simply use different definitions.

The traditional data warehousing approach to this is both centralised and controlled. Control is exercised by documenting the differing data definitions in an enterprise model that informs the warehouse metadata. The most fundamental definitions of basic data come from the operational environment and carry their influence into the BI environment.

Centralisation means that one value is defined as more correct or more commonly useful than the others and stored in the enterprise data warehouse, while the remaining ones are classified as derived values and made available to the users who need them in data marts. Figure 2 illustrates this model. This approach works well in a classical hierarchical organisation where the definitions of data meaning are agreed upon and relatively stable over time.

Figure 2: Traditional Controlled and Centralised Business Intelligence

When we examine the philosophy of the Web 2.0 mind-set, we see a number of key challenges to the model just described, particularly on the right-hand side of the picture where end users sit. The emerging computer-literate population who are comfortable with collaborative, social computing are less willing to accept definitions of data meaning emanating from the central organisation. Furthermore, they are used to changing and redefining the structures and boundaries of their online experience at will. As a result, the centralised and controlled model for defining and feeding data marts becomes increasingly untenable.

Interestingly, at the same time, businesses are beginning to recognise the value of innovation within their employee population. The Web 2.0 mind-set just described is seen as a powerful driver of innovation, and as a result, we may expect that users will be encouraged to experiment with new models of business measurement in the data mart environment. How can these changes be accommodated?

In order to support this innovative definition of measures and to allow their widespread use in the business, new set of metadata and data flows would need to be introduced into our model as shown in Figure 3, where the new flows are depicted with heavier arrows. Data and metadata from the marts are thus uploaded to the warehouse, from where they can be further distributed.

Figure 3: Additional Data and Metadata Flows Required by Web 2.0

However, it is clear that this also introduces potentially circular flows of both data and metadata, which often leads to conflicts and reconciliation issues. This has been addressed here by defining the uploaded data and metadata as an independent set within the warehouse. This autonomous set of data and metadata would need a further formal process to promote it to “general availability” to the public from the warehouse when any required reconciliation is done.

Incorporating a reverse flow of data from the marts into the warehouse is by no means new. The concept applies equally to so-called operational data marts, where the results of analyses or even newly created operational data are fed back into the warehouse or sometimes to the operational systems themselves. However, these latter flows are still defined and managed centrally, whereas the bottom-line in the Web 2.0 mind-set is the ceding of control to the users.

As described in my series on SOA, there is the added complication that the data updates from the operational systems through the warehouse to the marts are moving closer to real-time, while the original warehouse architecture was optimised for offline, batched transfers of data. Like SOA, therefore, Web 2.0 stretches the data warehouse concept by emphasising the sense and respond loop shown in Figure 1. With Web 2.0, however, we are no longer limiting the sense and respond loop to data, but allowing the metadata or definitions of data to also change in a sense and respond manner. So, the real question is if the data warehouse architecture is being stretched beyond its breaking point.

In Part 2, we’ll examine the characteristics of Web 2.0 applications today and as they evolve to try to answer that question.

If you found this article helpful and would like to receive the latest insights each month from Mike Ferguson and other experts featured on the Business Intelligence Network, please subscribe to the UK Business Intelligence Network Newsletter.