From Business Superstition to Business Intelligence

Originally published 17 February 2011

In recent history, each generation has seen great advances in medicine, transportation, technology, learning and understanding. Studies in mathematics and science have furthered our collective knowledge and allowed us to replace old superstitions with evidence-based approaches. If a doctor prescribes a medicine, I can be confident that this medication has been shown to be effective for my condition and I can be assured that common adverse effects will be listed. I can choose to take the medicine in the light of evidence already collected and scrutinized.

An evidence-based approach is at the core of good science and is the basis for all good decision making. Superstition, on the other hand, is “a credulous belief or notion, not based on reason or knowledge” [Wikipedia]. The trouble is that some business decisions are taken based on poor or irrelevant information, anecdotal evidence, untested hypotheses or weak assumptions—the original breeding ground for superstitions. So why does this happen and how do we elevate our decision-making processes to a more reasoned and informed level?

Anecdotal and Subjective Evidence

The tabloid newspapers would have a hard time livening up their articles without anecdotal evidence. It’s seductive, it’s engaging, it’s personal and it can be enormously misleading. It’s a survey with a sample size of one. Anecdotal and subjective evidence is frequently relied on when proper measurement is absent. A single opinion is no basis for good decision making and even though everyone knows this, anecdotal evidence still creeps in. Take employee performance, for instance: A good employee review process will canvass a range of opinions (e.g. 360 degree feedback) and will include some hard (non-subjective) performance measures, rather than rely on one person’s say-so.

Confirmation Bias

The confirmation bias is a tendency for individuals to notice the evidence that supports their existing opinion, whilst ignoring the evidence that contradicts it. The amount of evidence required to change an opinion is often far larger than that required to confirm it. For example, a company may choose to offer discounts to acquire new customers. The promotion may increase customer acquisition and could be interpreted as a success. However, if these customers are not retained, this may actually translate into reduced profitability. It is easiest to fall foul of the confirmation bias if information is collected on an ad hoc basis to justify decisions.

Perception Management

Many departments are responsible for reporting their own performance upwards and can even choose the metrics and the presentation of this information. The danger of allowing self-assessed performance is that departments and companies start cherry picking the information that they want to tell you. I describe this kind of situation as ‘perception management’—when a department is more interested in managing what you think of them than in actually improving their performance, for example. In the worst cases, business intelligence (BI) systems can be designed by departments as perception management tools, rather than for performance management.

Newspapers do this kind of thing all the time. The underlying statistics say one thing, but the narrative and presentation say another. For an example, see the Telegraph article “24-hour drinking leads to surge in violence”,  which claims “violent offences had soared since 24-hour licensing” but admits that “There was a 22 per cent leap in specific alcohol-fuelled crime between 3am and 6am, although averages across the whole day were slightly down”.  I’m sure shareholders would be delighted to see a sales report presented in this way: “New Internet promotion leads to surge in sales. There was a 22% leap in sales between 1 and 2 pm, but overall sales were down”. Here the flaw is obvious, but it’s much harder when the “but overall sales were down” is not said. To find the weaknesses in your business intelligence, look for what is not measured.

Measuring the Wrong Thing

In the early stages of collecting information for decision making, it’s easy to start with the information at hand. But is this the right information?  Does it lead to the outcomes you are looking for?  For example, if I want to increase my sales, I might not find much meaning in analyzing the number of miles travelled by each salesman. Where measures are used to drive a process, it is important that the desired outcomes are measured. This way, you can test whether salesmen who travel widely also sell more.

So How Do We Make It Right?

When Peter Drucker said, “If you can’t measure it, you can’t manage it”, he clearly described the need for evidence-based decision making. So how do we do this?  Well, first of all, make sure that you have a measurement system in place that links to your business strategy (see my earlier article, Strategic Business Intelligence). A BI system is only as good as the information it presents. Kaplan and Norton’s Balanced Scorecard gives detailed information on how to define a sensible balance of leading (forward looking) and lagging (historical) measures, encouraging organisations to focus on the drivers for growth as well as short-term financial performance.

In addition, Kaplan and Norton encourage the use of hypothesis testing—a very useful tool to help you guard against information superstition. Many metrics can be used to measure the current progress in implementation of a corporate strategy, but critically there must be measures (typically financial) to test that the implementation of your strategy has led to real improvements in your business.

Finally, the information must be presented clearly. Widely used BI tools provide elegant scorecards and dashboards that can be used to provide this information in a clear, readable and explorable format. Your organisation may have these tools already. For example, Microsoft SharePoint may already be in use in the organisation and can be used to provide interactive dashboards and scorecards.

By using a well-defined measurement system, appropriate hypothesis testing and clear data presentation, the traps of information superstition can be avoided and your BI system will be true to its name.

SOURCE: From Business Superstition to Business Intelligence

  • James AdmanJames Adman
    James Adman is an Information Management Consultant at IPL, a leading UK IT services company specialising in the delivery of intelligent business solutions. He has over 10 years of experience in helping a wide range of high profile clients exploit the full potential of their information.

    James has significant information management expertise gained across a variety of sectors, including manufacturing, finance, government, transport, emergency services and petrochemical.  James has a number of ongoing engagements in business intelligence including system procurement, specification and design.  He can be reached at

Recent articles by James Adman



Want to post a comment? Login or become a member today!

Be the first to comment!