Pattern Description

This pattern uses data virtualization to provide on-demand integrated data to applications, reporting tools, processes and portals via a web services user interface. Structured and semi-structured data sources are supported including RDBMS, any web service (internal or external), web syndication feeds, flat files, XML, packaged applications and non-relational databases.

Pattern Diagram

Pattern Example Use Case

A company needs to different kinds of information services targeted at different role-based user communities for access via their enterprise portal.  These services include:

·         Internal operational and analytical information services

·         Information services that integrate structured and semi-structured information including internal and external syndicated web feeds

·         Information as a Service (IaaS) services that  render information in various XML formats (e.g. XBRL) for consumption by external users and applications

Reasons For Using It

Rapid development of re-usable information services for consumption by portals, composite applications, processes and reporting tools.

Pattern Description

This pattern uses data virtualization to provide one or more on-demand integrated views of master data entities such as customer, product, asset, employee etc. even though the master data is fractured across multiple underlying systems. Applications, processes, portals, reporting tools and data integration workflows needing master data can acquire it on-demand via a web service interface or via a query interface such as SQL.

Pattern Diagram

Pattern Example Use Case

A manufacturer needs to make sure that changes to its customer data are made available to its marketing, e-commerce, finance and distribution systems as well as its business intelligence systems to keep business operations, reporting and analysis running smoothly. A shipping group of companies needs to perform a routine maintenance upgrade on a particular type of asset. However, its assets are managed by different systems in multiple lines of business. In order to budget for this upgrade it needs to have a single view of assets to fully understand maintenance costs. 

Reasons For Using It

To obtain a single integrated views of master data for consistency across business operations quickly at a relatively low cost.

Pattern Description

This pattern uses virtual views of federated data to create virtual data source components for use in ETL processing. The purpose of this pattern is twofold. Firstly to protect ETL workflows from structural changes to operational data sources. Secondly to create re-usable virtual data source 'components' for accessing disintegrated master and transactional data. The virtual data source pattern effectively 'ring fences' just the data associated with a customer, or a product for example, meaning that ETL workflows can be built for customer data, product data, asset data, order data etc.  This helps ETL designers to create ETL jobs dedicated to a particular type of data e.g. the customer ETL job, the product ETL job, the orders ETL job. Simplistic design of data consolidation workflows dedicated to a type of data allows these jobs to be re-used if the same data is needed elsewhere, e.g. customer data needed in two different data marts. It also guarantees that the same data is made available again and again via the same virtual data source  

Pattern Diagram

Pattern Example Use Case

Merger and acquisitions and new system releases often cause changes to operational systems data structures. This pattern can be used to shield ETL jobs that populate data warehouses and master data hubs from structural changes to source systems simply by changing the mappings in the virtual source views.

Reasons For Using It

Reasons for using this pattern include the ability to manage change more easily, lower ETL development and maintenance costs and modular design of data integration workflows associated with consolidating data.

Artesian are are focused on monitoring media news, competitors intelligence and market intelligence that can be fed into front-office processes - in particular to sales force automation. Integration with SalesForce.com is provided as is delivery to mobile devices for mobile salespeople on the road. Their intention on media intelligence for example is to track coverage across all media channels contextually matched to commercial triggers or specific areas of interest.  What I like about Artesian is the fact that they have looked at how to drive revenue from intelligence derived from web content by plugging it into front-office processes. Also by adopting social software attached to front-office systems like SalesForce.com's new Chatter offering it becomes possible to collaborate over this intelligence. I would like to see this solution integrate with Microsoft SharePoint and IBM Lotus Connections for more use in large enterprises. However, seeing the need to focus attention on content that has real value in the front office is a real strength of this young startup.  

Lixto has a integrated development environment that allows you to build analytic applications pulling data from web sites such as competitor price information, new competitor marketing campaign data and other information that can be loaded into their customisable analytic applications to monitor competitors for example. 

Extracting insight from external data is definately on the increase with YellowBrix and Mark Logic also in on the act. IBM jumped into the market back in October with their announcement of IBM Cognos Content Analytics. This market is heating up. It seems to me that the start-ups are out there with competitive offerings.

Looking at the trend here, it is clear that companies need to look seriously at moving from separate data management tools from many different suppliers, each with its own metadata, to single platforms with integrated shared metadata

• IBM's InfoSphere Information Server platform,
• InfoSphere Streams event processing,
• Change data captue,
• In-memory data in SolidDB and Cognos TM1
• Cognos Performance Management and Analytics,
• SPSS Predictive Analytics,
• Automated decisions via iLog rules engine and other technologies such as WebSphere Business Events and Cognos Now!
• Collaborative decision making via Lotus Connections
• Process Optimisation using the WebSphere BPM technologies and ESB/message Broker.

On top of this IBM will deliver solutions (both crosss industry and vertical . We are entering an era of business automation to get business optimisation whereby BI is integrated into processes and event driven automated decision making and action taking keep the business running optimally at all points of operation.

In addition, ILT has 4000 IBM consultants already in place to chase business.  Time will tell how successful this initiative is. It is very ambitious but real-time use of intelligence and predictive analytics on an event-driven and on-demand basis is definately the right direction. The challenge here is bringing all these technologies together and getting IT groups to play ball. In addition many businesses need to learn how to optimise their business. Trusted data (via Enterprise Data Governance and MDM) will be fundamental to that as will the need for companies to make an inventory of their business events. Unless companies learn what to look for in different parts of their business they will not be able to maximise the benefits of business optimization. 

Pattern Description

This pattern uses data virtualization to query structured data held in multiple underlying core operational and analytical databases and file systems to answer business questions.  It uses a search like user interface that can return results as to where data associated with items being searched on can be found e.g. a search could be done on a customer name, an order and a sales representative name. The data discovery pattern allows users to query the virtual views of a data held in multiple systems via the data virtualization server. Through this mechanism users can find relationships between different data items across systems, view the data as if in a single system to discover answers to business questions.

Pattern Example Use Case

Call centres are receiving a lot of enquiries as to why their orders are not being fulfilled. Data can be queried using a customer name, products ordered and the sales representative who took the order. Results returned show all occurrences of data about orders, the customer and the sales representative across multiple systems. Using the virtual views, this data can be analysed across systems to see what the reason is for delays in deliveries are e.g. order exceeds credit limit or order cannot be fulfilled due to inventory levels being too low. 

Reasons For Using It

This pattern has the affect of broadening access to enterprise data from a much larger user base who are confident in using a search box interface but who are not aware of where the data they need is located and who do not have the time and/or skills to use BI tools.

This brings back memories of Business Process Outsourcing adoption several years back and what it did to companies that had no business process integration in place before they outsourced some process activities. The result of that strategy was that it fractured processes even more in many cases and sent some of the data outside the enterprise making it more difficult to get at. As applications go off premise there is a real danger MDM could get out of reach. It requires MDM to start to get implemented to get control over data. SalesForce.com data is already coming inside the enterprise via ETL tools into DWs. Several ETL vendors support this. I just don't think that there has been many bringing it back in to populate MDM. Siperian has some case studies of their MDM customer working with cloud applications - in particuar SalesForce.com. What it does say, is that pursuing a cloud computing strategy on external cloud based virtualized servers without a data governance strategy, could very well wreak havoc on any enterprise.

With virtualization being high on the agenda of many CIOs, I would suggest that they should also keep an eye on risk management and compliance otherwise they could well cause make it harder to achieve trusted data. Without MDM, a clound computing deployment strategy certainly puts an Enterprise Data Quality Firewall and data integration services high up the agenda priority list!

·         Do you know what data exists in your enterprise?

·         Do you have an inventory of data items in use?

·         How many names have you got for the same data item?

·         How many metrics with same name but with different formulae?

·         Do your Excel metrics formulae, DBMS metrics formulae, BI tool metrics formulae, ETL tool calculations, ... all agree?

If the answer is no to any of these questions, what chance do you stand of remaining compliant or of trusting your data? If you don't know how many different variations of a data item exist in your enterprise how can you govern your data?

Some other questions to ask here from a business perspective are:

·         How many times do your core processes break because of dirty data?

·         Have your company ever messed up an order and angered a customer because of dirty data?

·         In terms of compliance, do you trust your data enough to tell it to a judge?

In my opinion what companies actually need is an interactive data map so that you can press a button and see where your customer data is or where your order's data is.  In order to be able to do this you need to have common data definitions for your customer data attributes and for your order data attributes etc. In fact you need to have a common set of enterprise wide definitions for all core entity data, transaction data and metrics. 

Having established this, the next step is to discover where you data actually is. Therefore Data Discovery technology (albeit a new area in data management) is critical help do this. In fact I would go as far as to say that without data discovery technology it is very difficult to get data under control. Increasingly therefore we are seeing vendors acquire or build this kind of software.  Once you data is located you need to map disparate data definitions for the same data to common enterprise wide definitions to be able to see where data is. Physical column names, data models, BI tool semantic layers, reports, SPREADSHEETS, files, XML schema, Access databases... If you can't tie all these to the same corporate definitions how to you govern data?  This is where data dictionaries/ business glossaries are key.  Lineage matter.

Ultimately the objective is to get consistency across the enterprise. You have to unravel your spaghetti ball.  

This means you need to get organised correctly, get the right technologies in place and get the right processes in place for enterprise data governance. 

Master data is also part of the program. You need to find out where your master data is maintained. How is it synchronised? What screens on what applications are used to update it? Do you know? MDMis not as simple as it looks. It is often a multi-year investment. So ask yourself do you want to start with read-only or read write? If  you buy an MDM system and you start updating master data centrally, what happens if you are still updating it also in other applications? Companies need a well thought out strategy for MDM as part of the Data Governance program. I will be addressing this in further blogs in the near future.

For now though, let me salute my fellow Irish countryman. Arthur Guiness. If you are stressed out on Data Governance at the end of a hard week there is nothing like a decent pint to help you unwind. Cheers!

In a recent paper by Jeremy Hope on Transforming Performance Management he states that "Most organizations want to adapt rapidly to changing events, but find that they are handicapped because of fixed budgets and poor forecasts. Adaptive organizations are able to respond more rapidly by switching resources dynamically to meet new threats and opportunities... ". 

In order to do this there is no doubt that companies have to deliver information more rapidly irrespective of whether or not the data is in a BI system. What many cannot afford is going through a formal time consuming process of data warehouse change to bring in all information necessary. Data federation is capable of sourcing data from several places one of which would of course have to be a data warehouse or data mart. But with increasing amounts of valuable information residing outside BI systems (especially on the internet) it seems that data federation has a role to play as a delivery platform rather than having to change data models, ETL processes and creating another cube or relational data mart. My expectation is that over the coming year we will see an increase in demend for data federation software. 

Several BI tools have been shipping data federation software as part of the BI tool bundle for some time to allow quick delivery of integrated information. Note that we are not talking about virtual data warehouses - on the contrary, data federation sofware rapidly integrates historical DW data with other data sources (operational data, internet feeds etc.) to deliver higher value information more rapidly. Therefore I see data federation software as complementary to BI systems.   If you would like more information on data federation please see this article on how it works. There is also a white paper on Maximizing Business Value from Data Virtualization  that talks about patterns and best practices to get the most out of this software. If you have alreay purchased data federation software or are considering it, let me know.  

When it comes navigating such designs to develop queries (or even generate them) it is often the case that IT professionals developing reports for the business can figure out how to use retrieve the information required  (although even IT developers can struggle). However when it comes to business users developing their own ad hoc queries and reports I frequently see these users really struggling to navigate the 'flexible' design first trying to figure out what measures mean, if the measure(s) is/are additive and whatnot. More often than not I see this resulting in real frustration among business users who end up getting aggregations in reports wrong and then start to lose faith in their new BI system.  Of course IT steps in to rescue the situation by building more snapshot tables, more materialised views etc. burying generic 'complexity' to make the job easier for the user.  More often than not these users also often resort to switching back to Excel to hold data outside any data mart so that they can look at data in a form they understand.

Have you seen this in your organisation?  If so I want your feedback. Is it the case that so called  'flexible' design techniques are rope for end users to hang themselves with?  My question is this. What is the best way that you see to design fact tables so that business users become productive and can easily understand how to get at the data when building their own reports? I am not so sure that being so generic is of business value.  Sure it is flexible. But is it usable? What use is flexible design if a business user cannot understand it and make use of all that valuable data? Is it not better to have multiple metric attributes in a fact table (if multiple metrics are needed) with each attribute name saying what the measure actually is?  Let's have your input!