I have now completed the second of my on-line courses on master data management for eLearning Curve. This one goes into considerable detail on how to evaluate an MDM vendor, based around an in-depth MDM functionality model which I have developed (and which has been through a significant review process by some serious MSM experts). The course also looks at the MDM market and talks about the current vendor Landscape in some depth, and finally goes through a suggested process for software procurement, including some tips and hints I have learnt by being on both sides of the negotiating fence.

The course can be accessed here:

http://ecm.elearningcurve.com/The_MDM_Market_How_to_Select_a_Vendor_p/mdm-03-a.htm

Its price is what seems to me almost absurdly cheap (eLearning Curve is new and they are trying to promote things), but as a reader of this blog you can take advantage of a special offer as well. When buying the course just quote the following voucher code:

AHDisc11R

and you will get a further discount of 20% off the already amusingly low list price. Seriously, this is a real bargain. Over five hours of chunky, in-depth material, to absorb at your leisure.

As the old Derek Bok saying goes, if you think education is expensive, try ignorance.

The data warehouse appliance market has become very crowded in the last couple of years, in the wake of the success of Netezza, which has drawn in plenty of venture money to new entrants. The awkwardly named Dataupia had been struggling for some time, with large-scale redundancies early in 2009, but now appears to have pretty much given up the ghost, with its assets being put up for sale by the investors.

If nothing else, this demonstrates that you need to have a clearly differentiated position in such a crowded market, and clearly in this case the sales and marketing execution could not match the promise of the technology. However it would be a mistake to thing that all is doom and gloom for appliance vendors, as the continuing recent commercial success of Vertica demonstrates.

To me, something that vendors should focus on is how to simplify migration off an existing relational platform. If you have an under-performing or costly data warehouse, then an appliance (which implies “plug and play”) sound appealing. However although appliance vendors support standard SQL, it is another thing to try and migrate a real-life database application, which may have masses of proprietary application logic locked up in stored procedures, triggers and the like. This would seem to me the thing that is likely to hold back buyers, but many vendors seem to focus entirely on their price/performance characteristics in their messaging. It actually does not matter if a new appliance has 10 times better price performance (let’s say, saving you half a million dollars a year) if it takes several times that to actually migrate the application. Of course there are always green-field applications, but if someone could devise a way of dramatically easing migration effort from an existing relational platform then it seems to me that they would have cracked the code on how to sell to end-users in large numbers. Ironically, this was just the kind of claim that Dataupia made, which suggests that there was a gap between its claims and its ability to convince the market that it was really that easy, despite accumulating a number of named customer testimonials on its web-site.

Even having the founder of Netezza (Foster Hinshaw) did not translate into commercial viability, despite the company attracting plenty of venture capital money. The company has no shortage of marketing collateral; indeed a number of industry experts who have authored glowing white-papers on the Dataupia website may be feeling a little sheepish right now. Sales execution appears to have been a tougher nut to crack. I never saw the technology in action, but history tells us that plenty of good technology can fail in the market (proud owners of Betamax video recorders can testify to that).

If anyone knows more about the inside story here then feel free to contact me privately or post a comment.

My company has now completed its latest assessment of the MDM market, which we represent via our “Landscape” diagram. The research is quite time-consuming, and involves surveying vendors for factual information (then sifting out their more blatantly optimistic assertions; I am a suspicious soul), looking at product demonstrations and asking pesky questions about features of the products, and carrying out a survey of reference customers. The results are amalgamated into a high level view of each vendor in the market in the dimensions of “market strength”, “technology “and “customer base”. Much more detail of the breakdown of the various elements that contribute to these scores are held in our vendor profiles.

You can see the diagram and some accompanying text on the front page of our web site.

I have spent some time recently in building up an MDM course. Normally such things are done at conferences, so unless you happen to be at some distant venue, they pass you by. However this one is different. It is an on-line course, marketed by a new company called eLearning (who have some well-known founders). The company reckons that it is harder and harder for people to justify trips to conferences for education, and this is certainly true at the moment from what I have observed and heard about technology conferences. Hence its model is to sell courses on-line.

The course is “MDM Fundamentals and Best Practice”, and you can see more about it here. It is actually quite a lot of work to put together such a course, but I am pleased with the result, and you can now get over three hours of my views on MDM for a very fair price indeed, all from the comfort of your desk or living room, and at a pace that you can control. Of course you do miss out on the trip to Las Vegas or similar, but you can’t have everything.

We have now completed our survey of data quality. Based on 193 responses from IT and business staff from around the world, there were some very interesting findings. Amongst these was that 81% of respondents felt that data quality was much more than just customer name and address, which is the focus of most of the vendors in the market. Moreover, customer name and address data ranked only third in the list of data domains which survey respondents found most important. Both product and financial data was felt to be more important, yet product data is the focus of barely a handful of vendors (Silver Creek, Inquera, Datactics) while of all the dozens of data quality vendors out there, few indeed focus on financial data. Name and address is of course a common issue and conveniently is well structured and has plenty of well-established algorithms out there to attack it. Yet surely the vendor community is missing something when customers rate other data types as higher in importance?

Another recurring theme is the lack of attention given to measuring the costs of poor data quality. Lots of respondents fail to make any effort to measure this at all, and then complain that it is hard to make a business case for data quality. “Well duh”, as Homer Simpson might say. Estimates given by survey respondents seemed very low when compared to our experience, and also to anecdotes given in the very same survey. One striking one was this: ”Poor data quality and consistency has led to the orphaning of $32 million in stock just sitting in the warehouse that can’t be sold since it’s lost in the system.” This company at least has no difficulty in justifiying a data quality initiative. The survey had plenty of other interesting insights too.

The full survey and analysis, all 33 pages of it, can be purchased from here.

As part of our ongoing research program, we are conducting a major survey into the state of data quality today. If you have a few minutes it would great if you could participate in this (all participants get a free summary of the survey results).

The survey can be found here:

In addition your e-mail address will be entered for a prize draw offering you the chance to win one of ten free annual subscriptions to The Information Difference website (worth USD $ 550).

Thanks in advance.

Most data quality vendors have their roots in name and address checking, even if their software can go beyond this. What is less well known is that the actual business of getting street level address data (to verify postal codes etc) is a tedious business that varies dramatically by country (the UK post office database covers almost every address in the UK, but Eire has no post code system, for example). Software vendors do not typically want to be in the business of updating street address databases, and there is a patchwork of local information providers that fill the gaps. If you have any international aspirations, though, just discovering who does what by country, and licensing the various data sources is in itself a non-trivial task, and so companies exist that do this. One was a UK company called Global Address, bought some time ago by Harte Hanks (who market Trillium), while the other was Address Doctor. Many data quality vendors use Address Doctor, including some that might superficially appear to compete. These include Dataflux, IBM, and even QAS. Some MDM platform vendors also use Address Doctor, who provide at least basic name and address data for 240 countries and territories.

The cat was put firmly among st the pigeons this week when Informatica bought Address Doctor. From their viewpoint this secures a key provider of address data, and follows their prior acquisitions of Similarity Systems and, more recently, Identity Systems. Informatica, via these purchases, has established itself as one of the major data quality vendors. Given its competitive position, the data quality vendors who use Address Doctor will, at the least, be feeling nervous. I spoke to an executive from Informatica this week and was told that Informatica intended to honour the existing arrangements, but who knows how long this state will last? As Woody Allen said, the lion may lay down with the lamb, but the lamb won’t get much sleep.

The problem for the other vendors is that there is no obvious place to go. Global Address is already in the hands of Harte Hanks, and while Uniserv in particular has its own name and address data, it is mainly strong in this area in Europe. Address Doctor was a convenient neutral player and is now in the hands of a major market competitor, and other vendors may have little choice but to look at building up their own networks of address data providers if they are to sleep easy. Of course it is not clear that they have to worry; for example Pitney Bowes Business Insight (who have what was Group 1 software) use Global Address, and this arrangement has continued without incident despite Harte Hanks Trillium’s ownership of them.

It will be interesting to see what measures the current Address Doctor users take, or whether they will just cross their fingers and hope Informatica plays nice.

We have recently completed a major survey into the deployment styles used in MDM implementations. My colleague Dave Waddington has recently posted a summary of the results here. As can be seen, MDM projects are turning out to be quite meaty in size, but encouragingly the sucess rates were higher than I was expecting.

There were several quite interesting results that came out, and we will be doing further research into this area. The full report can be purchased from our website.

I have recently been spending some time looking at the data quality market, and a few things seem to pop up time and again. The first thing is, in talking with customers, just how awful the quality of data really is within corporate systems. One major UK bank found 8,000 customers whose age was over 150 according to their systems. All seemingly academic (if you are taking money out of your account, who cares what your age is?) until some bright spark in marketing decided that selling life insurance to these customers would be a fine idea.

Story after story confirms some really shocking data errors that lurk beneath most operational systems. These are the same operational systems that are used to generate data for the end-year accounts which senior executives happily sign off on pain of jail-time these days. I hope no one shows these sames execs the data inside some of these systems, or they might start to get very nervous indeed.

Yet in a survey we did last year, only about a third of companies in the survey have invested in data quality tools at all! Does anyone else find this in any way scary? Do you have any entertaining data quality stories you can share?

There are a number of approaches of “styles” to tackling MDM within a company, at least in terms of what to do first. If your most pressing issues are improving the quality of business reporting you may opt for “analytical” MDM, or if your issues are mostly with the data in transaction systems you would go for “operational MDM”. Within such categories there are different degrees of invasiveness, from a “registry” style where you leave the master data intact within operational systems, to an extreme root and branch approach “transaction style” where you rip out the ability of operational systems to maintain master data and put in new MDM system(s) to do this and feed the rest of the enterprise, while “co existence” recognises the reality that it may be necessary to live with a mix of approaches. But just how many companies go with which style, and how successful are they?

To answer this The Information Difference is launching a major piece of market research. This survey, sponsored by Microsoft and with media sponsors DM Review and CIO Magazine, is an in-depth look at this topic. To take the survey please click here.

When the survey is complete and the analysis is done I will let you know. Participants will receive a summary of the finding.