Following on from my last blog on data federation, the next data federation pattern I would like to discuss is a On-Demand Information Services Pattern. This is as follows:

Pattern Description

This pattern uses data virtualization to provide on-demand integrated data to applications, reporting tools, processes and portals via a web services user interface. Structured and semi-structured data sources are supported including RDBMS, any web service (internal or external), web syndication feeds, flat files, XML, packaged applications and non-relational databases.

Pattern Diagram

Pattern Example Use Case

A company needs to different kinds of information services targeted at different role-based user communities for access via their enterprise portal.  These services include:

·         Internal operational and analytical information services

·         Information services that integrate structured and semi-structured information including internal and external syndicated web feeds

·         Information as a Service (IaaS) services that  render information in various XML formats (e.g. XBRL) for consumption by external users and applications

Reasons For Using It

Rapid development of re-usable information services for consumption by portals, composite applications, processes and reporting tools.

Following on from my last blog on data federation, the next data federation pattern I would like to discuss is a Master Data Virtual MDM pattern. This is as follows:

Pattern Description

This pattern uses data virtualization to provide one or more on-demand integrated views of master data entities such as customer, product, asset, employee etc. even though the master data is fractured across multiple underlying systems. Applications, processes, portals, reporting tools and data integration workflows needing master data can acquire it on-demand via a web service interface or via a query interface such as SQL.

Pattern Diagram

Pattern Example Use Case

A manufacturer needs to make sure that changes to its customer data are made available to its marketing, e-commerce, finance and distribution systems as well as its business intelligence systems to keep business operations, reporting and analysis running smoothly. A shipping group of companies needs to perform a routine maintenance upgrade on a particular type of asset. However, its assets are managed by different systems in multiple lines of business. In order to budget for this upgrade it needs to have a single view of assets to fully understand maintenance costs. 

Reasons For Using It

To obtain a single integrated views of master data for consistency across business operations quickly at a relatively low cost.

Following on from my last blog on data federation, the next data federation pattern I would like to discuss is a Data Warehouse Virtual Data Source pattern. This is as follows:

Pattern Description

This pattern uses virtual views of federated data to create virtual data source components for use in ETL processing. The purpose of this pattern is twofold. Firstly to protect ETL workflows from structural changes to operational data sources. Secondly to create re-usable virtual data source 'components' for accessing disintegrated master and transactional data. The virtual data source pattern effectively 'ring fences' just the data associated with a customer, or a product for example, meaning that ETL workflows can be built for customer data, product data, asset data, order data etc.  This helps ETL designers to create ETL jobs dedicated to a particular type of data e.g. the customer ETL job, the product ETL job, the orders ETL job. Simplistic design of data consolidation workflows dedicated to a type of data allows these jobs to be re-used if the same data is needed elsewhere, e.g. customer data needed in two different data marts. It also guarantees that the same data is made available again and again via the same virtual data source  

Pattern Diagram

Pattern Example Use Case

Merger and acquisitions and new system releases often cause changes to operational systems data structures. This pattern can be used to shield ETL jobs that populate data warehouses and master data hubs from structural changes to source systems simply by changing the mappings in the virtual source views.

Reasons For Using It

Reasons for using this pattern include the ability to manage change more easily, lower ETL development and maintenance costs and modular design of data integration workflows associated with consolidating data.

Everywhere I look at the moment I see my clients talking about needing to benchmark themselves against the market, to understand customer and prospect sentiment on social networking sites and to understand competitors in much more detail. It is not just me that has recognised this need. It also seems that new young startup companies have also seen this gap in the market. Over the last few days I have spent some time talking to Andrew Yates, CEO of Artesian and Christian Koestler, CEO of Lixto about their solutions in this area.

Artesian are are focused on monitoring media news, competitors intelligence and market intelligence that can be fed into front-office processes - in particular to sales force automation. Integration with SalesForce.com is provided as is delivery to mobile devices for mobile salespeople on the road. Their intention on media intelligence for example is to track coverage across all media channels contextually matched to commercial triggers or specific areas of interest.  What I like about Artesian is the fact that they have looked at how to drive revenue from intelligence derived from web content by plugging it into front-office processes. Also by adopting social software attached to front-office systems like SalesForce.com's new Chatter offering it becomes possible to collaborate over this intelligence. I would like to see this solution integrate with Microsoft SharePoint and IBM Lotus Connections for more use in large enterprises. However, seeing the need to focus attention on content that has real value in the front office is a real strength of this young startup.  

Lixto has a integrated development environment that allows you to build analytic applications pulling data from web sites such as competitor price information, new competitor marketing campaign data and other information that can be loaded into their customisable analytic applications to monitor competitors for example. 

Extracting insight from external data is definately on the increase with YellowBrix and Mark Logic also in on the act. IBM jumped into the market back in October with their announcement of IBM Cognos Content Analytics. This market is heating up. It seems to me that the start-ups are out there with competitive offerings.

So here I am in Las Vegas at IBM Information On-Demand - IBM's global information management conference. The up-coming theme that will be launched here is IBM's new Information Led Transformation (ILT) initiative which opens up IBM's major play in the Business Optimization market.  IBM is pouring enourmous amounts of money into this space, stating that this market is growing twice as fast as any other initiative including Business Automation. Their estimation on market size is $105Bn.  The objective of Information Led Transformation is micro-optimisation whereby every business optimization is carried out in real-time (or should I say right time) at all points of impact. That means optimising all decisions and process activities based on the current situation as it happens by leveraging event processing, predictive analytics, rules engines for automated action management based on a base of trusted information delived on-demand and in-context where it is needed and when it is needed. IBM ILT will leverage

• IBM's InfoSphere Information Server platform,
• InfoSphere Streams event processing,
• Change data captue,
• In-memory data in SolidDB and Cognos TM1
• Cognos Performance Management and Analytics,
• SPSS Predictive Analytics,
• Automated decisions via iLog rules engine and other technologies such as WebSphere Business Events and Cognos Now!
• Collaborative decision making via Lotus Connections
• Process Optimisation using the WebSphere BPM technologies and ESB/message Broker.

On top of this IBM will deliver solutions (both crosss industry and vertical . We are entering an era of business automation to get business optimisation whereby BI is integrated into processes and event driven automated decision making and action taking keep the business running optimally at all points of operation.

In addition, ILT has 4000 IBM consultants already in place to chase business.  Time will tell how successful this initiative is. It is very ambitious but real-time use of intelligence and predictive analytics on an event-driven and on-demand basis is definately the right direction. The challenge here is bringing all these technologies together and getting IT groups to play ball. In addition many businesses need to learn how to optimise their business. Trusted data (via Enterprise Data Governance and MDM) will be fundamental to that as will the need for companies to make an inventory of their business events. Unless companies learn what to look for in different parts of their business they will not be able to maximise the benefits of business optimization. 

Following on from my last blog, the next data federation I would like to discuss is the Data Discovery pattern. This is as follows.

Pattern Description

This pattern uses data virtualization to query structured data held in multiple underlying core operational and analytical databases and file systems to answer business questions.  It uses a search like user interface that can return results as to where data associated with items being searched on can be found e.g. a search could be done on a customer name, an order and a sales representative name. The data discovery pattern allows users to query the virtual views of a data held in multiple systems via the data virtualization server. Through this mechanism users can find relationships between different data items across systems, view the data as if in a single system to discover answers to business questions.

Pattern Example Use Case

Call centres are receiving a lot of enquiries as to why their orders are not being fulfilled. Data can be queried using a customer name, products ordered and the sales representative who took the order. Results returned show all occurrences of data about orders, the customer and the sales representative across multiple systems. Using the virtual views, this data can be analysed across systems to see what the reason is for delays in deliveries are e.g. order exceeds credit limit or order cannot be fulfilled due to inventory levels being too low. 

Reasons For Using It

This pattern has the affect of broadening access to enterprise data from a much larger user base who are confident in using a search box interface but who are not aware of where the data they need is located and who do not have the time and/or skills to use BI tools.

Having seen a lot of increase in demand from my clients to start a program to create information services, I thought it might be useful to look at one way of doing that through the use of data federation software. Then I realised that it would be better to look at all the popular ways of using this technology. On that basis, this blog starts a series of blogs from me on popular patterns that companies can use to get maximum value out of the data federation software.