One of the things I got busy on was writing a white paper sponsored by Lyzasoft which they published recently. Speaking to the folks at Lyzasoft, and also to participants in conferences at which I was presenting over the same time period, I found myself looking again at the role and positioning of Business Intelligence tools vs. the people who use them. And I've deliberately phrased that as "versus" - because it's well accepted that many "analysts" who should use BI tools as part of their toolkit end up either fighting the prescribed tool or abandoning it altogether.

Business Intelligence these days is a term that covers a multitude of sins, from executives defining and executing business strategies to automated processes identifying potential fraudulent transactions without any human intervention and everything in between. Depending on the particular business need, software vendor, consultant or industry analyst involved, the focus in different implementations can vary dramatically. All remain BI, but each requires very different thinking and architectural approaches.

One set of users who often sit uncomfortably on the boundary of BI are often called business analysts within their organizations. They tend to use and combine data in new or unusual ways in order to gain new insights into what is going on in the business. They often obtain the data they need from beyond the data warehouse, because the warehouse doesn't hold the data they need or has cleansed it in a way they don't agree with or they simply don't know it's there. They often manipulate and combine data in different ways as an iterative part of their analyses. And many times their tool of choice is the spreadsheet.

It's pretty clear that these business analysts should be supported by the BI community. Within their own organizations, the data they require should come increasingly through the data warehouse in order to ensure data integrity and consistency. The analyses they perform and the outputs of their work also need to be maintained and tracked for compliance and regulatory reasons. Overall, setting these people off to find, manipulate and analyze data in a haphazard way simply doesn't make sense.

Fitting the needs of business analysts into the data warehouse architecture is, however, possible. I've coined the term "playmart" to represent the type of environment these users need. The aim is to balance agility (playing) with control (in a data mart). I've defined eight key characteristics of a playmart in the white paper (see also below) and would be very interested to hear your views on them.

First, we had the interesting revelation that the individual banks and lending institutions all seemed to be blissfully unaware of the extent to which they were exposed by lending in the sub-prime mortgage market. It's difficult to imagine how the information available to decision makers in these companies could have been so scarce or so uninformative. Most, if not all, financial institutions have had extensive and expensive data warehouses in place for many years now. Business Intelligence should easily have warned of the dangers. Was the increasing level of risk unmeasured, overlooked or simply ignored?

More recently, we've had the spectacle of banks being unwilling to extend short-term lending facilities to one another for fear that the borrowing institutions could go belly-up in the next few days! Could the lenders not know? Unfortunately, in this case, the answer is probably that they couldn't. Despite the fact that the worldwide financial market is tightly and instantly interconnected at a transaction level, the truth is that the underlying data remains disconnected and dispersed. Data Management and Data Quality have simply not been considered. Proper business governance in the financial markets as a whole is impossible without a well-defined and credible data foundation.

So, assuming that we can survive the crisis without a meltdown, what has been happening should be a clarion call to Data Management professionals in the financial industry particularly but also beyond. We need to recognize the interconnected and increasingly fragile web of data dependencies that hold the business world together. It's time to get out there and apply the principles we know and preach already. And we had better get moving quickly.

And yet, check in with the users. Indeed, check in with yourself. Why is Excel so popular? Because it makes it easy to play with the data, check out hypotheses, get answers otherwise unavailable, and so on. And once you've gotten the answer through the spreadsheet, chances are you won't get the time or the resources to recreate the process in a more auditable, quality-conscious way. It's a real and spreading problem. But, what to do?

This week I had the opportunity to preview a new product called Lyza that's due to launch on Sept. 22. In fact, you can download it and play with it already. Scott Davis, the CEO of Lyzasoft Inc. explained that they had spent a lot of time investigating how business analysts, the power users of spreadsheets, actually work. This is usually a good idea, because you find out what the users really need, and which of your assumptions are right or wrong. It will probably come as no surprise that most analysts approach their work in a highly unstructured and iterative way, pulling bits of relevant data into Excel from a variety of known sources - both official data marts and reports as well as unofficial files, spreadsheets, etc. they happen to have created before or borrowed from trusted colleagues. And they do it in Excel, because that's the only way they can.

What Lyza does is to provide an easier, more intuitive way of pulling data together from diverse sources, combining and manipulating it and creating results and reports for distribution to the business. Well, that's all fine and dandy for the business analysts you may say, but how does it help the BI and data quality departments address the data contagion? The answer is that Lyza tracks and saves an audit trail of every action and every step of the analysis process that the user is building as well as enabling snapshots of the results to be cached and preserved for posterity. Now the data quality folks are beginning to smile. And the BI department? Well, they're less sure: they like the added traceability, but this is still outside their comfortable data mart zone.

However, we could look at it in a different way. We could imagine that Lyza provides a new type of data mart - a "playmart" - a sand box where power analysts can experiment with data and perform all sorts of analyses in a safe, well-managed environment. Now, if only we could evaluate the analysts' logic and productionalize those analyses and reports that are going to be reused and built upon in the future.

Scott's initial answer was that you can certainly do all this within Lyza itself. But a bit of further probing convinced me that the metadata that Lyza stores to describe the analysis processes is probably sufficient to enable the creation of ETL scripts for your ETL engine of choice. This would certainly require further investigation and automation, but it seems like the bones of the idea are there. In this case, the playmart could address a set of business analysts' needs that have been long ignored by the BI departments and by BI vendors as well.

The only real fly in the ointment is whether Lyza will be able to convince the spreadsheet jockeys to get off their current Excel rocking horse and jump on the bright new Lyza pony in the playmart! (And that sentence would work so much better if only Lyza had chosen a mustang for their logo rather than a gecko.)

Claudia and Colin have drawn a box on the left of their architecture picture that arises directly from the insight that operational BI really is a different beast from the traditional BI we've all known and loved over the past 20 years or so. When you deeply consider the implications of building an operational BI system, as Claudia and Colin clearly have, it becomes obvious that operational BI has many of the characteristics of traditional operational or transaction-processing systems. Therefore, from a systems architecture point of view, you put them in the same box, in this case called "Business process intelligence".

There are also some differences, of course. The most important is how the business users interact with these two related types of system. The value proposition of operational BI is that human decision-making skills can improve operational processes. How? Well, there are two very distinct threads here.

One is the proposition that we can apply advanced analytics technology automatically to parts of the operational process. Fraud detection is a good example. Applying advanced analytics on the fly to credit card transactions gives better detection of fraudulent transactions. Note that this type of operational BI is almost completely invisible to the business users: they see the results of more fraud detected or less false positives, but how that happened is both unknown and uninteresting.

The second thread brings users very directly into the loop. Here, the operational BI technology is made part of the users' visible process. Business users are presented with decision support technology that displays trends or exceptions in near real-time data, so that they can potentially choose a different course of action to that embedded in the normal flow. In effect, business users get to change the business process on the fly, rather than doing little more than data input as was previously the case.

Now, keeping this in mind, here's the million dollar question. What's the difference between an operational system and an informational system; how do you distinguish between an operational process and an informational process? In the good old days, it was easy! The operational side was nearly or actually real-time, dealt with individual transactions or data elements according to a predefined process where the users had minimal freedom to act intelligently. Informational systems, in contrast, were centered around users who were expected to make intelligent decisions based on historical data without any clear process to turn those decisions into action.

So, what is the answer today? When we in BI start building operational BI and the operational world starts implementing adaptive SOA-based systems, the distinction between operational and informational more-or-less disappears. This puts operational BI and operational systems together in one box of the architecture. But the deeper and probably longer-term implications of this bold step have not been explicitly called out. In fact, these implications are obscured by the naming of the new architecture as "Decision intelligence", because the top level of this architecture is no longer confined to the world that was formerly BI; it actually becomes the single, common process or interface through which all business users will interact with the underlying IT systems.

Is that scary? Absolutely! But it is a clear and logical consequence of the paths that BI and operational systems are currently on. It means that we in BI are no longer in total control of our destiny. But the same is true of the operational systems. And, although I've not covered it here, collaborative systems (e-mail, office support, etc) are also being drawn inexorably into the same converged path.

It's time we all started to talk to one another! And that does imply that decision intelligence may be too narrow a term for us all to agree on. May I propose again the "Highly Evolved Business"?

I fully agree. I've been using the terms "Highly Evolved Business" and "Business Insight" over the past year or so to express exactly the same thought. Indeed, Claudia, Colin and I have discussed this whole idea already at length and are very much on the same page. But I hadn't seen their architecture picture before, and it gives me the opportunity to discuss the whole topic from a higher perspective in this and the next post.

Under Decision intelligence, the architecture shows three vertical blocks called "Business process intelligence", "Business data intelligence" and "Business content intelligence". The meanings of these blocks are fairly obvious, but take a look at the linked article for a full explanation. My thought is that they are almost too obvious: they closely reflect our current arrangement of systems building blocks in the IT world.

Let's first examine the data and content blocks. Today, if you look at typical enterprise implementations, you will certainly see databases and separate content stores. You'll also notice independent systems built upon these separate stores. But, if you step back from the storage and processing issues, it's pretty difficult to distinguish between the two categories. Try explaining the difference to a business user!

Take an example of a clinician who's trying to make a treatment decision. She's looking at a chest x-ray - content in our terms. And she's also looking at the "structured data" that goes with it: this x-ray is of a 45 year old male, smoker of 20 cigarettes a day for the past 30 years who has been admitted with shortness of breath. Does she see unstructured content and structured data that must somehow be combined in her decision making? I'd argue not. She simply sees a set of information she's using.

And some of the old barriers between the storage of structured data and unstructured content are breaking down. Where is the EXIF data (structured metadata) of a photo stored? Yes, in the JPG file along with the unstructured content. Where do e-mail systems store the structured metadata about sender, subject, date sent, etc? Sure, in the database with the unstructured e-mail body content.

I could make a similar argument about the lack of distinction between real-time data (or operational) data and historical (data warehouse) data.

My point is that if we want to create a new vision for the future, we need to start seeing the world through non-IT eyes. It's all information. It's a single concept; a single category of "stuff". And we in IT need to start creating the tools and methods that allow us to create, manage and make available all information in a coherent and consistent way. At a conceptual level, that has to be the goal and that should be our first pictorial representation.

Keep that thought in mind. I'll come back to next time when I look at the process side of the picture.

Now I'm as fond of a shortcut as the next guy, but I've learned the the word "Instant" is not all goodness. When I've bought some instant Spaghetti Bolognese in the local supermarket I've found that the cost is a lot higher than the individual ingredients and the taste, well, leaves a lot to be desired. Sure, I saved some time when I got home, but did I get value for money? And did I end up with what I really wanted? So, why should I expect more from an Instant DW?

"Caveat emptor" as the Romans used to say. Here are a few contra-indications for when instant gratification should not be expected in your next BI (or SOA) project:

1. The business users are not quite sure what they want.
   Most BI projects start with a vague set of requirements from the potential users. It's going to take some time to hone these down to a usable definition of data and query needs. In the meantime, maybe it's best to let the users continue to play with their instant Excel spreadsheets and look over their shoulders to see what they're doing.
2. Somebody forgot to document the meanings of the data in the source applications.
   This is the oldest metadata problem. If your data sources have not been properly described, an Instant DW is likely to be instantly dismissed as misleading and inaccurate. Do you want to go there?
3. Garbage in, garbage out. Or worse...
   If your ingredients (data sources) are contaminated with erroneous data, you're going to end up with a very sick business on your hands if you just take the Instant DW approach. Understanding and fixing dirty data is time-consuming, but mandatory.

It's all about quality time... or quality vs. time. If I bring home my instant Spaghetti Bolognese, I may get it on the table within a few minutes. But, if the kids won't eat it or, worse, throw up that night, I'd argue I've made the wrong trade-off between time and quality. You need to consider the same balance in a BI or SOA project.

Now, I'm off to spend some quality time with my kids :-)

Information OnDemand. No, not the annual IBM Conference in Las Vegas, in October. But a rather low key effort from Business Objects with a website to allow companies to access market data and incorporate it into their BI efforts.

There's a definite growing interest these days in combining external data with the contents of the warehouse. But it does raise some concerns, not least about the reliability of the external data and how to create a valid semantic relationship between the two data sets. In the past, companies have addressed these concerns by obtaining key market and other external data from trusted sources like Dunn and Bradstreet, Reuters and others and then ensuring that such data entered the warehouse via a controlled feed designed by Information Architects who could match the two data sets correctly. After all, such external data is another information source for the warehouse and should be managed like any other.

This method works well for large enterprises with a centrally-controlled approach to the warehouse. And where the value-add derived from or risks incurred by using this data are significant, this method is probably still required. But what if you are a small or medium enterprise? Or what if you really only want to do a couple of once off analyses?

Shopping at the Information OnDemand website appears to be the answer! Here you can buy prebuilt, but customizable, reports combining your data with external market and financial data. You can buy one-time snapshots or subscribe for regular updates.

For larger companies, this could provide a safe and cost-effective way of dipping their toe in the big ocean of external data. For smaller companies, it could be all they need. Sounds like a useful idea to me!

The service has been available since September 2007, but I hadn't come across it before. Maybe there are some similar services I should know about, so please feel free to comment.

A Google-like search solution for business content? What exactly does that mean?

The phrase "Google-like search", of course, covers a multitude of marketing-speak, but let's assume that it includes the patented PageRank technology behind Google's Internet search success. Google itself describes PageRank as follows: "PageRank relies on the uniquely democratic nature of the web by using its vast link structure as an indicator of an individual page's value." (http://en.wikipedia.org/w/index.php?title=PageRank&oldid=230400158 as of Aug. 7, 2008). A number of questions arise for me: Does an enterprise intranet usually have a vast link structure? Would business executives really consider the "democratic vote" of the organization as a valid indicator of a document's importance? Indeed, how democratic is the link structure in an intranet?

Google, Wikipedia and many Web 2.0 systems have an underlying belief in James Surowiecki's concept of "the wisdom of crowds". Data warehousing, Business Intelligence and, indeed, all traditional IT development tend to put more faith in experts and their accumulated knowledge. In the BI world, I'm beginning to see some level of acceptance that the so-called experts do not have a monopoly on business knowledge. We see that there is a growing need to allow and, indeed, facilitate the feedback of knowledge that emerges on the fringes of the BI community (the front-line staff and first-line managers) back into the core of the warehouse for wider promulgation and reuse.

But, to what extent does Google and the Web 2.0 community recognize that some knowledge is inherently more useful or valuable (although not necessarily "right") simply based on the authority of its source? And within the tighter and more closed confines of an enterprise, that not all the requirements for wise crowds are met? If not, we may see the many years of careful effort by data modelers and administrators, and information stewards overturned in the rush to Web 2.0. This would not be in anybody's interest.

On the other hand, if I've made the wrong assumption about what "Google-like search" means... Anybody care to comment? Or maybe I'll find time to sign up for the webinar!

That's all very well and good, but, excuse me, have I missed something? Since when did data marts become data warehouses? I know that the appliance vendors tend to label themselves as data warehouse appliances, but I thought we all knew that was marketing. Of course, any appliance will be part of a data warehouse system in the broader sense. But when you look at the features and strengths that appliances have, you can see that they are really data marts. Data "hypermarts" perhaps, but marts nonetheless.

By definition, a data mart is a subset of the data in the enterprise data warehouse that has been optimized for use by a particular set of users. Such optimization includes selecting the data needed for some set of business purposes and structuring it to allow the fastest, most appropriate query access for users. It's all about how you get the data out! Sounds to me like exactly what the appliance vendors emphasize.

On the other hand, the data warehouse focus is on getting the data in. How to cleanse and reconcile the diverse data. How to ensure the cross-source timing is right. How to create a model that reflects the needs of the wider enterprise. And finally to make the consolidated view of the business available to the users - usually through data marts.

So, does the Microsoft acquisition disrupt the entire data warehouse market, sending the large players into a spin? I doubt it. Building a real data warehouse will continue to be as challenging as ever, requiring the same strong integration and project management skills as before as well as the deep database integration and manipulation technology that only the big relational databases possess as of now.

First, though, a quick gripe at the airlines. First, they failed to transfer my bag from Dublin through Heathrow onto my flight - arriving in Helsinki at 17:45 and following three more flights from London over the next few hours - finally delivered it to my hotel at 2am. Then they added insult to injury by telling me that the system couldn't accept a request to transfer my bag through Lhr on the way back (until they got exceptional management permission!) because Aer Lingus had left the Oneworld Alliance three weeks ago. Well done to BAA, British Airways, Finnair and Aer Lingus for making the trip such a pleasant experience!!!

Fortunately, the meeting was much more enjoyable.

You can read the official blog elsewhere, so what follows are my eye-openers.
"Democratisation of content" is changing the media business irrevocably. We may tend to think (as I did) of this phrase as meaning that Joe or Jill Bloggs can create brand new content such as text, images, audio or video far more cheaply than professionals can, and sometimes the quality can even be acceptable. But, in many cases, it's not brand new content; it's often a mash-up of content from existing material. This is a much more common form of creativity that has existed since time immemorial and can be seen in art forms such as collage. Calling this "pirating" just confuses the issue; the social acceptance of this behaviour and the ease with which it can now be done is a fact of the marketplace that content providers may just have to get used to. But what I think it means is that the sale of digital content "rights" (or "restrictions" as one of our participants phrased it) is fast becoming an untenable business model. The people who seem to be hurting most are the publishers, record companies and production / distribution organisations that have most profited from this model in the past; and, unsurprisingly, they are the most resistant to change.

One thought we had was that the emphasis in the Media industry needs to switch from the subset of users who are unwilling to pay for anything (estimated to be less than 20% of the market) to those who understand that there is no such thing as a free lunch, and are willing to pay a fair price for content and who often want to see that payment go in a larger share to the actual creative people who built the content. Unfortunately, at present there are not too many real and legal mechanisms to do this.

At the artist's end of the spectrum, the internet is creating a new opportunity for collaborative working that may further diminish the role of the publishers and distributors. Artists can create collaborative communities on the Web where they can meet people with the complementary skills they need to turn an idea into a marketable product. In the music industry, that might include a producer, a sound engineer, a graphic designer, a marketer and a few more. Collaborative tooling and a place on the web to meet and work is probably all they need to get it all together. In this case, what's the role of the record company in the future?

It certainly looks as if the traditional publishers and producers are in danger of being dis-intermediated.

The second topic that really caught my attention was a "digital identity" or "digital me" on the Internet. The technologists tend to see this identity as a cool way to enable a more free-flowing experience in the online world, where appropriate content in context ("the information you need and only the information you need") could be channelled directly to you. Privacy advocates see significant dangers in such a consolidated digital identity, greatly increasing the risk of abuse of power, and so on.

While some of our debate centred on the question of "should we have a digital identity", it struck me that we are acquiring a digital identity anyway by stealth - courtesy of data warehousing / business intelligence - so, the real question we need to ask is: "how can we, the consumers, take back control of our online identities?" What we need, therefore, is technology that makes it easier to create and manage our digital identities, as well a trustworthy institution or set of institutions who would hold these identities with maximum privacy and security.

The bottom line in all the topics - a new business model for content, communities of collaborating artists and digital identity - boils down to a single word: TRUST. Interestingly, trust has been the basis on which the open software industry has grown. It's at the core of e-Bay's business. And it looks like it's going to be key to Media and Digital Content in the future.

The increasing role of Joe or Jane Citizen in producing content is proving to be very disruptive to the traditional media businesses, and reflects a deep change in the relationship between "consumers" and "producers" in today's society. Who would have thought only a few years ago that people would choose to view personally-produced videos over professional ones. Yet that's what appears to be happening with the growing populatity of YouTube. The same shift can be seen in Wikipedia - an encyclopedia written by non-professionals, in Amazon with every Joe Bloggs writing book reviews; not to mention the explosion of blogging itself.

I'm only posing the question here, because I don't have the answer, but how is this power shift going to play out in other business areas, in govenrment and in education? What are the implications for the way software develops? And what's the knock-on effect on business intelligence when everybody wants to have their say on the business trends?

I'd say we're in for a sea-change. Hold tight to the rails. And follow the blog on the GIO website above to see how the thinking is developing.

In truth, Neil has got much closer to the new spirit of "2.0" as he recognises the impact of the changed paradigms of social computing, collaboration, mash-ups and other Web 2.0 approaches on traditional data warehousing. He's also spot-on when says that "better ties between analysis and action" will emerge.

But my question is - will it be Business Intelligence 2.0 or will it simply be "Business Intelligence to zero"? If everything a business user needs is going to be mashed-up into a user-defined portal interface, and analysis and action (aka informational and operational) are intimately linked, why have Business Intelligence as a separate entity at all?

Two forces are acting simultaneously to change the face of BI beyond recognition. At the back end, SOA is gradually redefining the way applications interact, driving more integrated data and allowing more pervasive access to that same data. Data modelling, cleansing, extraction and transformation - traditional preserves of the data warehousing IT department - will become part of the mainstream IT business. Out on the user-facing side, business people increasingly expect an integrated view of all their tasks - without having to think if they are analytical or action-oriented.

As an example, take a look at the IBM Workplace Dashboard Framework, where traditional BI user function is delivered through a standard portal. Now recall that traditional action-oriented, operational applications can be surface through the same portal; and the portlets can exchange information! Where will the boundary exist between informational and operational in such a portal?

Neil is absolutely right when he says in conclusion: "Prepare for the inevitable". It seems to me it's the inevitable re-integration of BI into the mainstream of IT.

First of all, let me welcome you all to my channel on BeyeNETWORK. It's a pleasure to be here and have the opportunity to talk to you all -- and most especially to hear your views, too.

This is going to be a short entry, since it's a holiday here in Ireland today, as in the rest of Europe. All the better to get over the hangovers, I guess!

I'd like to focus discussion here around the future of BI and how I see it evolving in the coming years. You can see that's where my interest lies from the contents of my first article and I hope it's something you are all interested in too. My belief is that we are going through quite a sea change in the BI world at present, driven by similar changes in the IT world in general. We have the opportunity now to define how we want to see BI evolve. If we don't participate, we will be swept along by the changes going on in the wider IT world. It would be a shame to be left behind ... so I hope you'll join me on this exciting journey.