Business Intelligence Network Business Intelligence Resources

Blog: Richard Hackathorn

« Becoming a Real-Time Enterprise: Is the Book worth It? | Main | Mixing Old Wine in a New Wine Skin »

The Black Swan of Business Intelligence

Sometimes you start reading a book with low expectations about its significance. But, the book surprises you and delivers a message of great significance. That has happened with a new book entitled The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb. He is a professor of the Sciences of Uncertainty (an odd title) at the University of Massachusetts. See his Wikipedia entry and a PBS podcast.

Let me start with the bottom line. I strongly recommend this book for all professionals in Business Intelligence (BI) who care about the means and results of our profession upon our clients.

I have this naïve belief that more information is better, assuming that the information is relevant to the business, properly cleansed, structured cross-functional, analyze appropriately, distributed to the right people and so on. This book totally negated that belief, instilling a humble attitude toward how much we can not know and shocking me about how much our current BI practices do damage to our clients.

And... I have just read the first few chapters. I am starting to be aware of the problems in general, confused about their implications to BI, and wondering whether there are any solutions. This is a book that will take several months to consume (because you read a few sentences, think ‘what?’ and then reread it several more times).

Let me give a small taste of Taleb’s argument. Before Australia was discovered, everyone knew that all swans were white, because all swans that were ever observed were white. Therefore, rule of nature was that all swans are white. Someone discovered a black swan in Australia. That one swan negated a belief held for a thousand years by all of mankind. Afterward, people concocted explanations as to why such a rare animal was perfectly normal and should have been expected. Taleb then extends this analogy to explain the events and aftermath of September 11, along with many other pivotal events in human history.

That is the Black Swan. It is a totally unexpected, but significant, rare event that seems plausible...afterwards. In Taleb’s words, the Black Swan is an event with three attributes: “First, it is an outlier as it lies outside the realm of regular expectations, because nothing in the past can convincingly point to its possibility. Second, it carries an extreme impact [changing our basic paradigms that explain the world]. Third, in spite of its outlier status, human nature makes us concoct explanations for its occurrence after the fact, making it explainable and predictable.”

I submit that we are unprepared to handle the Black Swan with current BI technology and practices. In fact, current BI does more harm than good, by giving us a false sense of reliability in what we think we know.

Help me with my struggle to understand the practical importance of the Black Swan. I would like to get a discussion established on Black Swan issues within the BI profession, along with joint publications with some of you. Is there anyone interested in this pilgrimage?

Posted by Richard Hackathorn on Friday, 31 August 2007 |

Comments

The goal of BI is to better predict what would happen if certain parameters change based on historic information. BI helps us “speculate” better and faster. I don’t think its intent is to drive deterministic findings about the subject matter but rather to identify patterns within empirical data. I don’t think the black swan discredits BI as a discipline; instead it highlights that when our assumptions are incorrect, BI helps accelerate making bad decisions and better yet, support them with numbers.
I guess in a nutshell, if you expect collated and correlated data (information) from BI, you are in good shape. If you expect wisdom – you may just be looking at the wrong place.

Here is another random question – is the black swan an outlier in the long tail?

Posted by: Sami Akbay | August 31, 2007 3:10 PM

I liked the book too (indeed I reviewed it here) and I took away both that the systems into which we embed our insight must be agile enough that we can change them fast (when we turn out to be wrong) and that there are some advantages to automation of decision-making that come from eliminating people and their tendency to interpret data to support pre-conceived notions.
A fun topic...
JT
The EDM blog
My ebizQ blog
Author of Smart (Enough) Systems

Posted by: James Taylor | August 31, 2007 4:04 PM

Very interesting... I fear that BI can never attack this problem. I’m sure people will try, and may even be successful in a few very narrow areas. Contemplate this case for a while:

Back in the all-swans-are-white days, there were also a number of other animals that we had conceptions about (in fact, all discovered species). For example, we thought that all frogs have only one heart, that all horses have four legs, etc. There are almost an infinite number of facts like that can be thought of. Now, with the discovery of Australia came new knowledge, not all swans are white, some animals jump around on two legs, etc. Similarly, in business, there are an almost infinite number of things that could affect a company. Knowing which to attend to is almost impossible, and until you know which factors to attend to and proof against, you can’t attempt to predict it (not to mention the price of proofing against too many things). Perhaps we can build a Nostradamus machine?

Do you have access to VC funding for said machine?

Posted by: Kai Larsen | August 31, 2007 4:13 PM

The Black Swan concept is quite fascinating. My take is that Black Swans occur due to the non-linear nature of the universe (Chaos theory) - small changes causes unpredictable and disruptive changes over a period of time.

I feel that this theory lot of relevance to the BI world.

My perspective is:

1)BI with its reporting / analytics paradigm is based on accumulated "history" of events, where each point is assigned the same weightage. With that, I don't think BI can predict future outliers at all. BI at best can provide a confidence interval of measures for future events.

2)BI has to embrace the simulation world more rigorously. Each data point in a report / analytical model has to be fed into a simulation model that mimics the organization processes. The simulation model might account for the non-linearity in processes.

Thanks for your informative blogs.

Posted by: Karthikeyan Sankaran | November 20, 2007 3:20 AM

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)