Business Intelligence Network business intelligence resources

Blog: Richard Hackathorn

« September 2006 | Main | November 2006 »

Wednesday, 11 October 2006

What to do about Very Large Data Streams (VLDS)

In Colin White's article on Enterprise Data Mountain, Part I, he opens Pandora’s Box of various data store components for the new architecture of enterprise data integration, whose objective is to create an integrated and consistent view of your enterprise. The architecture is service-oriented using an Enterprise Service Bus to link the components.

Forget the old paradigm of OLTP-OLAP happily coexisting! It isn’t your father’s data warehouse! It’s a new ballgame, folks!

But... Colin forgot an important component... More and more data comes to the enterprise as a stream. You say, "So what? Just batch the stream!"

This is a BIG stream, as in megabytes per second coming from a virtual data store of many petabytes. This is a CONSTANTLY CHANGING stream in content and structure. Forget storing the data. Examine it on the fly and decide instantly what aspects are important to manage within the enterprise data mountain. This gives new meaning to the phrase "unstructured data". See the work of Jeff Jonas of IBM on perpetual analytics.

Let’s label this new component as a Very Large Data Stream (VLDS), as a bow to the critical contributions of the Very Large Data Base community over the past twenty years.

Is VLDS important? I would predict that such streams are and will be critical sources to the enterprise data mountain as it evolves of the coming decade. New media sources (established news channels, blogging, YouTube), near real-time satellite images, detailed weather data, point-of-sale data from all your stores plus samples of the rest of your industry, RFID data tracking your full supply chain globally, and on and on... That is Enterprise 2.0 of the future.

What do you think? Crazy ideas or right on?

Posted by Richard Hackathorn at 10:09 AM |

Saturday, 7 October 2006

Private Data Sold by India Call Center

One of the issues of off-shoring has been security of the information that U.S. companies share with their partners. Outsourcing firms, especially in India, are sensitive to this issue and go to great lengths to assure their corporate clients that their data security is excellent.

Well... As reported by Zonk in Slashdot, Channel 4 in the U.K. did a 12-month TV investigation showing how credit card and passport data on thousands of banking customers could be bought for as little as US$9.50 each.

The fallout from this investigation has been significant! Read the comments on this Slashdot posting over the coming month. There are several insightful comments, along with a diversity of perspectives.

Posted by Richard Hackathorn at 12:00 PM |