BeyeNETWORK UK Blogs

State of Chaos - 1

Tue, 9 Mar 2010 12:19:07 MST

As i look around at some of the leading companies across the globe, I find a state of chaos across these companies, especially in information management. What is interesting to note is that the problem transcends Fortune 100 to Small Business and is very similar. Where did things go wrong? all these companies have spent valuable dollars to improve information management.

Deeper investigation reveals several gaps and loopholes. The vendors that came to provide strategy and technology services addressed "point in time" problems and developed focused solutions. With the craziness of technology improvements, market conditions (m&a), business changes all coming together, the entire situation is like playing "poker" at the high stakes tables in Vegas.

How do we unravel from the state of chaos to a state of order. Hiring more strategy consultants is not the only answer. We need to do a holistic overview of the business situation and find out where issues have come from and how the current solution process will address today and the future.

In order to recover from a state of chaos to clarity, you need to look at the strength of the organization and its people, assess its technological prowess, its competition, business value, market value, strategic maturity and adoption of processes in the organization and more.

However doing all this in a short span requires a great deal of focus and more maturity from the vendor that will be hired for this exercise. Size does not matter in this spectrum.

A vendor rating from the Analyst community is a starting edge of this process, but that alone is not enough, we need to ask for organization maturity, delivery capability and much more to not repeat the mistakes to get to a state of chaos. Sometimes a few "niche" vendors may be needed to solve the larger problem.

To be continued.....

Visit of SQLStream

Tue, 9 Mar 2010 05:18:34 MST

Quite recently, I visited the SQLStream. As the majority of database server vendors, SQLStream is located in California; in San Francisco to be more precise. Of course, their primary product (also called SQLStream) supports the database language SQL, and they try to follow the SQL standard as much as possible. So far, nothing new under the sun. You would almost think that this is again one of many new vendors trying to dethrone Oracle, Microsoft, and IBM. However, that would be an incorrect assumption.

As the name implies, SQLStream is a so-called streaming database server, comparable to IBM InfoSphere Streams and StreamBase Server. The main difference between SQLStream on one hand and most other products on the other hand, is that the former is a pure SQL-based product. The statements to stream are according to the SQL standard. Most other streaming products use proprietary languages, such as Spade, or use extensions.

For those who haven't studied this topic in detail yet, a streaming database server allows us to formulate queries on streams of data. Examples of streams are log files of certain systems, messages that are entered, or web logs. Even before this data is stored in tables, we can already access them and analyze the data. Someone once explained streaming database servers as follows: queries executed in the context of a classic database server are like: how many fishes live in this pond, whilst queries executed in the context of a streaming database server is like: how many fishes swim by in a fast-flowing river during a certain period of time.

SQLStream offers all the features above. In addition, views are used to define streams, and this type of streaming views can serve as input for other streaming views. Through join and union operators, data of different sources can be integrated. In fact, SQLStream supports many of the features normally found in an ETL tool, except that SQLStream uses streams and SQL. Data streams are integrated live the moment they arrive. The result of an integrated stream can be send to an application or data warehouse. See the following link that contains an explanation on how SQLStream can be used together with SQL Power.

In short, SQLStream is absolutely worth studying.

Note: The owners of SQLStream are also the founders of Eigenbase.org. This organization supplies a toolset with which database servers can be developed. As can be expected, SQLStream is also developed with this toolset.

The Age of Columnar Databases

Mon, 8 Mar 2010 11:01:45 MST

We all know by now the word "columnar database" and some of the most popular providers - ParAccel, Vertica and Infobright. Though columnar databases have distinct performance advantages and applications that can be deployed, from a business perspective, there was a question mark and doubts of where to apply these solutions.

In my opinion, the unstructured data integration into the data warehouse is a very key area of applicability for the Columnar database. The reason for this statement, stems from the fact that the unstructured database is a very column oriented data store and if we are processing oodles of text, the reference of information will be more clustered in nature. This drives the need to store the data in a co-located manner, which then leads me to look at columnar databases more closely.

I do not imply that Oracle, SQL Server, DB2 or Teradata cannot support the unstructured database needs, but for those who want to adopt to the cloud, use more on demand scalability etc, columnar databases may be an in-house option. Additionally for those providers in cloud arena, columnar databases may be more optimal to adopt to.

I'm currently running tests on Infobright and will share the results in the next few days. I'm processing a large volume of semi-structured and unstructured data and will measure the throughput and performance against the standard RDBMS platforms.

In my technical opinion, choosing a columnar database for processing large textual data provides more business benefits, the final results can be migrated to a corporate environment, but operational aspects can be done in the columnar platform.

Evidence Based Medicine - Textual ETL Makes It Possible

Thu, 4 Mar 2010 09:05:54 MST

I have been researching on this subject for the past few months. It is becoming clear that to control errors in diagnosis and treatment of medical conditions, Evidence Based Medicine can be defined as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care of the individual patient. It means integrating individual clinical expertise with the best available external clinical evidence from systematic research." (Sackett D, 1996).

This requires research and processing of clinical information that can be leveraged by a Doctor. While search from Google, Oracle and others are supporting the search portions, Clinical data is heavily semi-structured - a combination of structured and unstructured data. The unstructured portions is where you have all the content and context of diagnosis, treatments, prognosis, conclusions etc. This is where Textual ETL and the unstructured database build will help us. The entire DW2.0 methodology can be directly applied to this system. We can classify, categorize, contextualize and capture information from the clinical data into a database.

In conducting the research, I have found that we can process years of clinical data into an unstructured database with relative ease and when combined with the statistical research data, the results have been an eye-opener. The possibilities of reducing errors and offering the right treatment means the cost of healthcare can be managed well. The quality of the treatment will be more rewarding and welcome.

I know that many Doctors and other medical professionals have different opinions on this subject, but that was when no technology was available to support EBM, now that we do, we should start paying attention to this area.

I will be writing my first article for this year on this subject, stay tuned for the article.

EDW and Columnar Databases

Wed, 3 Mar 2010 11:17:21 MST

Columnar databases, especially those with an MPP approach, have been notching up impressive query performance figures, showing gains of 100X and more on the traditional players. Such figures make great press releases, but they do place the emphasis on using these databases as data marts rather than the enterprise data warehouse (EDW) component of the data warehouse architecture. Focusing on data marts and very specific business intelligence applications makes a lot of sense for new market entrants and smaller players in the DW space, allowing quick wins and easily understood sales messages.

But, I have been convinced for some time now of the much greater potential such performance unleashes in the broader and more complex EDW environment. And the vendors have been fairly quiet about this part of the market so far, maybe preferring to leave such more technically and politically complex projects to the big guys. So, it was good to see Vertica's 4.0 announcement last week beginning to address the EDW market with its emphasis on "enterprise ready" and a number of interesting new features and expansions of old functions.

Robust workload and resource management for mixed workloads is a prerequisite for an EDW. Vertica's introduction of administrator-defined resource pools with memory-usage, priority and concurrency settings and the assignment of users to these pools is a big step in this direction. A rework of the optimizer in support of this and other features suggests that Vertica are serious about this support.

Also introduced in V4.0 is a newly optimized single record lookup on primary keys. While aimed at a particular financial analysis use case, this function shows that the database can do more than just crunch columns. Added to the FlexStore feature introduced in V3.5 where newly loaded data is kept in row format in memory for some period of time, I believe we're seeing the database's growing ability to handle the sort of record-level processing often needed in EDWs. The new time-series support in V4.0 also plays directly in EDW needs.

Time and customer experience will, of course, prove if I'm correct, but it seems to me that Vertica is beginning to test my assertion that columnar, MPP databases can be applied to EDWs. And further that their performance characteristics offer the possibility of re-architecting the EDW / data mart divide.

After 40 Years of Practice, Developing IT Systems Is Still Hard

Tue, 2 Mar 2010 05:09:20 MST

Last week, my family and I visited a basketball game in Phoenix. The Phoenix Suns were playing against the Philadelphia 76ers. It was a great game, the Suns won with 106-95. However, before I was able to get into the stadium I had the following experience.

A few days prior to this game, I bought the tickets through Ticketmaster.com. To buy those tickets, I had to enter my credit card information. Normally, this doesn't cause any problems, because credit companies operate internationally and know that some of their customers are based outside the US and they know those addresses might have different formats and structures.

What I had to enter was, as you might expect, my name, credit card number, expiration date, and a security number. In addition, I had to enter my address information so that they can verify a few things. So, dutifully I entered my address including the zip code. Entering the address components went well until I got to the zip code. The zip code was not accepted because the system expects five digits and I tried to enter four digits and two letters, which is the format of the Dutch zip code. But it didn't accept the letters. They had probably switched on a simple check: digits only please.

Now, this caused a problem, because for getting through the verification process I had to enter the correct zip code, but for buying a ticket I had to enter an incorrect zip code. Eventually I made the decision to enter the zip code of the hotel I was staying at. And, to my surprise it worked. I got my tickets and printed them.

Unfortunately, Ticketmaster.com had accepted my address information, however the credit card company's IT system had not. I discovered that when I entered the stadium and showed them my tickets. They were not accepted. Guess why? The zip code didn't match the rest of the address and it didn't correspond to the correct address.

How is it possible that in the year 2010 we still have problems with this simple type of data entry. Didn't they get the right definition of zip code from the credit card companies, or don't they check whether the zip code matches the rest of the address? Is their system not aware that the formats of zip codes can be different in other countries? And how is it possible that they first inform me that the credit company has accepted the credit card information, and later on they indicate they haven't. We have about forty years of experience in developing IT systems, and we still make errors such as this.

In the end, I did get in, I just bought new tickets at the ticket sales, and guess what, I got exactly the same seats. I still wonder if I had also entered the full address of the hotel, whether it would have been accepted.

Lyza Commons: how to integrate BI and Enterprise 2.0

Thu, 25 Feb 2010 14:58:11 MST

Having worked with CEO, Scott Davis of Lyzasoft and produced a white paper on Collaborative Analytics in the first half of 2009, it came as no surprise to me that version 2.0 of Lyza had a major emphasis in the same area. What did surprise me, however, was how far they have advanced the concepts and implementation in such a short timeframe!

Successful collaboration between decision makers requires an environment that facilitates a free-flowing but well-managed conversation about ongoing analyses as they evolve from initial ideas to full-fledged solutions to business problems. Consider a common scenario. The first analyst gathers data she considers relevant and creates an initial set of assumptions, data manipulations and results. She shares this via e-mail with her peers for confirmation, and she receives suggestions for improvement, some of which she incorporates in a new version. Her manager reviews the work personally and makes further suggestions; a new version emerges. She also shared the intermediate solution with a second department, and the analyst there created another solution based on the original. Meanwhile, the first analyst finds an error in her logic buried deep in cell Sheet3!AB102...

We all know the problems with multiple unmanaged copies, rework, silently propagated errors and so on in the usual spreadsheet- and e-mail-based business analysis environment. Lyza and Lyza Commons together address these issues by creating a comprehensive tracking and auditing mechanism for every step of an analysis and providing an integrated environment for sharing and discussing work among collaborators. Integral metadata links all copies derived from an initial analysis. Twitter-like conversations (called Blurbs) about an analysis are linked to the referenced object creating a comprehensive context for the conversation and the underlying analysis. The folks at Lyzasoft have also come up with a security concept for sharing analyses they call Mesh Trust that should make sense in most enterprise collaboration environments.

My bottom line? Lyza and Lyza Commons 2.0 provide a seamless blending of analytic function, managed and controlled access to information resources and enterprise-adapted social networking around analytic results and their provenance. This is precisely the type of function needed by businesses who want to regain control of spreadmarts that have run amok. This is the right conceptual foundation for real, meaningful business insight and innovation going forward.

Irrational Behavior? Any ideas

Fri, 19 Feb 2010 18:34:52 MST

I'm stumped by irrational behavior from large technology corporations, especially ones that feel that they have big muscles and have earned the right to such behavior. They go to any lengths to be winning, and do not realize that what they think is collateral damage from their perspective, can be a serious business factor in the future.

What do these companies think? do they even think before they get on an assault charade? do they care for ethics and values? if they do not, do we also start questioning their ethics and values?

The end result of such negative behavior is the loss of trust in the long course from their own customers and marketplace. Sadly their focus is on today's gain and not the overall future perspective probably, which to an extent explains such behaviors. One does not know how many times these companies have lost business due to these behaviors.

I hope that at some point in time, these corporations and their management do realize the value of earning the trust of their customers and the market is very important for their continued success and it is not just technology or financial glories that carry an impact.

Data Warehouses and Solid State Disks (SSD)

Wed, 17 Feb 2010 14:34:45 MST

As mentioned in my last post, ParAccel had a really interesting announcement coming out this week. I was talking about their partnering with Fusion-io to attach SSD technology in their Paraccel Analytic Appliance for even faster query performance. ParAccel are not alone in their use of SSD; Teradata's 4555 and Oracle's Exadata 2 also include the technology. For me, it's not even about faster query results for users. It's about the implications for the entire Data Warehouse architecture.

Over the past couple of years, we've seen dramatic improvements in database performance due to hardware and software advances such as in-memory databases, columnar storage, massively parallel processing, compression, and so on as described in my white paper from April 2009. SSD, in one sense, is just another piece of accelerating technology. However, add it to the existing list, and you begin to see the possibility of revisiting old assumptions about what is possible within a single database. Here are a few ideas to play with:

Do you still need that Data Mart? With so much faster performance, maybe the queries you now run in the Mart could run directly on the EDW. Reducing data duplication has enormous benefits, on storage volumes, but principally in reducing maintenance of ETL to the Marts.

Where to do operational BI? It was once considered necessary to install a separate ODS to support closer to real-time access to consolidated atomic data. But with such a fast database, couldn't you just trickle feed the data and do it all in the Warehouse itself. One less copy of data and one less set of ETL can't be all bad!

ETL or ELT? Extract, transform and load has been the traditional way of loading a Warehouse, with a special engine to do the transform step. Well, with a faster and more powerful database engine, you have the option to try extract, load and transform and let the Warehouse database do the transform work.

Although ParAccel, like all the smaller vendors are focusing more on selling to the "bigger, faster, more complex analytics applications" market at present, I'm pretty sure that the work ParAccel is doing under the covers on query optimization, workload management, loading and updating features will pave the way for a sea change in how we do data warehousing in the next few years.

Want Broader BI Usage? Crystal Founders Offer Indicee

Wed, 17 Feb 2010 10:01:44 MST

Mark Cunningham has reunited some of the team that built Crystal Reports (now part of SAP Business Objects) and launched Indicee, a SaaS-based BI reporting play that is pointed squarely at the continuing difficulty of extending BI beyond its seemingly permanent minority usage model.

It's commonly understood that users continue to fend for themselves manually, moving data to spreadsheets for analytic manipulation because IT is unable to respond quickly enough to their needs. Indicee tackles this by re-using existing report and spreadsheet content (not surprisingly, Crystal reports lead the source list), moving it to the cloud for data mart-based interaction, and innovating a different approach to user interaction. It's worth a look, and a free download for trial use sweetens the deal.

more...

Social Media and Business Impacts

Tue, 16 Feb 2010 14:28:43 MST

In the past few days I have been seeing a flurry of corporate activity in Twitter, Facebook etc from large corporations like Schwab, Starbucks, Victoria's Secret etc. Not only have they got the corporate websites, catalog advertisements, direct mail, email, now they are connected in the "digital world".

The reason these companies and others have the "digital" presence, is because today you need to be connected to the customer to create a positive business impact for the business. How do you stay connected? only if you keep listening. With the advent of social media, the world has shifted from a passive to an active participant in providing businesses feedback. Not only is the feedback public, it is also creating disruptive impacts.

Then there is also the question of competitive threats and also collaborative opportunities. Due to all these business impacts, Social Media is becoming a very essential business requirement for businesses today

There is a TDWI night school that I'm holding on this topic next week for two days. If you are attending, we will see you there.

Textual ETL - Your Key to Success In Web2.0/3.0 Large Data Management

Fri, 12 Feb 2010 14:22:45 MST

As you look at the Web2.0 and Web 3.0 world, it is becoming clear that information management on the internet with respect to data will be a catalyst to your success or failure in the new world order. Traditionally when you build applications in the Web, you do not look at it being very data savy and largely transactional in nature. But looking at the way Flickr, Facebook, Twitter and Social networks have changed the game, we understand that no longer are we looking at a transactional silo, but rather need to react to a "long tail".

In the new world, you will need to be "big" but "nimble", large and flexible. Wow that's a mouthful to keep saying. The reason for this thinking is we need to look at the structured and the unstructured data to understand the customer and their needs, and react quickly to address those needs. When you talk of unstructured data, in a Web world you cannot afford to load Gigabytes and Megabytes of data, most of which is noise. You need to get the intelligence extracted and linked, but leave the content and the context outside. How do you accomplish this? there are some companies addressing this need, but we need a nimble and strong ETL engine to do this process.

This is where you need to look at Textual ETL and understand how to build the unstructured database. The traditional vendors are doing their part, but the end result has left a lot to your imagination.

Textual ETL is complex and deals with data which has minimal structure and completely 180 degrees opposite of transactional data. As we move towards the Web 2.0 --> Web 3.0 world, we will encounter this hurdle and I hope there are tools that will handle this Large data management to integrate the transactional and textual data.

Initiate Acquisition, InfoSphere Business Information Monitor Intro Punctuate IBM's Information Governance Launch

Fri, 12 Feb 2010 12:00:58 MST

IBM chose a regular meeting of its Data Governance Council to launch a series of now-renamed Information Governance services and products and announce a definitive agreement to acquire Initiate Systems. A privately held software vendor based in Chicago, Illinois, Initiate stands as IBM's 30th acquisition in the information and analytics sector. Initiate's data integrity software for information sharing is designed to help health care clients work more intelligently and efficiently by supporting timely access to patient and clinical data. It also enables governments to share client information across multiple agencies to better serve citizens.

This post was co-authored by Merv Adrian and Charles King of PUND-IT. It's a little more narrative than my usual posts (and longer), and is reproduced here in the form it appeared in the PUND-IT newsletter.

John Schwarz was never going to be CEO of SAP

Thu, 11 Feb 2010 14:01:00 MST

If you are interested in business intelligence or follow the doings of SAP, you probably already know that the former BOBJ CEO John Schwarz has left the company. The temptation is to delve into the details of all the post-takeover events at SAP and try to puzzle out the proximate cause of this move. Then you could do a little Kremlinology, trying to guess what is going on in the minds of various execs.

But I prefer to stick to a few general observations.

When business intelligence companies get taken over, the senior management almost always leaves after a few years.
There has been some discussion that Schwarz was shooting for the CEO job, but that hardly seems likely to me. Remember SAP had about ten times the revenues of BOBJ. Also, as SAP has learned the hard way, selling business intelligence software is a very different business than selling ERP.
The main thing that keeps an executive of the acquired company on board is a clause in the contract. As soon as the clause expires, the executive jumps ship.

That's about it, really.

OK, like Oscar Wilde, I can resist anything but temptation. Look at the press release. Schwarz says "I strongly believe that SAP BusinessObjects will play a vital role in SAP's future success" and that he is "confident about the future direction of the company". It's hardly a ringing endorsement of SAP's ERP business, is it?

EDW: What's not working?

Tue, 9 Feb 2010 06:52:56 MST

Kim Stanick and Rick Glick of ParAccel were at the Boulder BI Brain Trust (BBBT) last Friday. They have an exciting announcement coming soon, and much of what was discussed was under NDA, so I can't give details here. But about half-way through their presentation, they threw up a slide saying simply "EDW: What's not working?"

Well, that's a negative question! And, anyway, I believe most of us have some good ideas about what's not working--from project scoping and delivery issues to problems of complexity of feeds and bottlenecks in timely data availability. So, let me re-frame the question: "Where next for EDW?"

I wrote a BI Thought Leader for ParAccel last April called "Analytic Databases in the World of the Data Warehouse" that began to address that question, and as the world of BI has evolved since, I want to revisit that question briefly. Back then, I wrote:

"Specialized analytic databases using [advanced] technologies ... now offer significantly improved performance for typical BI applications, enable previously impossible analyses and often lower cost implementation. They also have the potential to challenge the current physically layered Data Warehouse architecture. This paper ... argues that analytical databases may enable a move to a simpler non-layered architecture with significant benefits in terms of lower costs of implementation, maintenance, and use."

In brief, it's our old friend, the paradigm shift, enabled by a dramatic shift in the price-performance characteristics of data warehouses driven by a new generation of technology. The possibility I saw then was a return to a physically simpler, more singular implementation of the EDW. And indeed that may still be a first step.

My thinking has evolved further since then, and I'm really beginning to envisage a much larger problem space that we need to address--how to integrate the entire enterprise information set, operational, informational and collaborative. I call that Business Integrated Insight (BI2), described in a more recent white paper. The discussion at BBBT last Friday led by a number of physical database technology experts gave rise to some new insights into how BI2 could be physically instantiated.

Virtualization at every level of the environment--servers, applications, data and particularly databases--linked closely with advances in the technology (as opposed to the hype) of cloud computing is widely discussed today as a way to reduce IT capital and operating costs, consolidate infrastructure, simplify resource management and so on. However, database virtualization offers new possibilities in the physical implementation of an enterprise data architecture that spans all data types and processing needs. Chief among these are flexibility of implementation, adaptability, mediated access to and use of data across multiple database types, significant reductions in data duplication and the gradual construction of overarching models that describe the entire business information resource. I'm sure there's much more to be said on this topic, but I'd love to hear the views of some experts in the field.