As web 2.0 edges towards becoming mainstream in the BI market (probably 2009 timeframe) expect to see more adoption of richer interaction in user interfaces and more collaboration capability to share BI for joint decision making. If you are already using Web 2.0 in BI applications and tools in your organisation please share your experiences on what works and what doesn't.

While Excel can never be ignored in any organisation, the increasing demand to analyse Excel data raises questions as to whether or not that data can be trusted especially if you have been sent this data in your email. It brings back the issue that has plagued many companies for years when it comes to Excel. Do you know where the data in the spreadsheet came from? How do you know you have the right version of the spreadsheet? Are spreadsheets managed? Is there other server side data sources that can be accessed from the dashboard tool that would give you more confidence in trusting the data?

With Office Excel 2007 increasing the maximum limit on the number of rows in a spreadsheet from 64000 to 1 million, my concern is that the increasing demand for dashboards will raise the likelihood of million row "spreadmarts" being created all over the organisation by business users rather than pointing dashboards at server side data in a BI system. Only time will tell however policy is clearly needed around spreadsheet management and dashboard development if we are to remain in control of the data and have confidence in it. I would be interested in hearing from many of you out there who are encountering this problem and what you are doing to manage it

1. Consolidation in the BI market will continue now that we have seen the software giants make their moves in 2007. Oracle bought Hyperion, SAP bought Business Objects and IBM announced acqusition of Cognos. In order to compete with this, other vendors will consolidate to try to offer and alternative. So expect to see more mergers in 2008

2. The cost of the BI platform will continue to drop amid pressure from the software giants (Microsoft in particular), and open source alternatives (e.g. Pentaho, Jaspersoft, Talend et al). The money to be made is in Performance Management and Data Management.

3. Both Performance Management and Data Management technologies will separate from BI platforms (if they haven't done so already) and become suites of tools in their own right

4. The growth in the size of the data management market is set to continue as companies try to standardise of a suite of tools for enterprise data management (an enterprise data management platform) which includes an end-user business vocabulary tool, data modelling tool, data discovery and mapping tool, data quality profiling, data cleansing, data integration (consolidation, federation and synchronisation). This data management platform will be used for data replication, data warehousing, data migration, master data management, data synchronisation and on-demand data management services published in a service registry and available on an enterprise service bus (ESB) in a service oriented architecture (SOA)

5. Complex Event Processing (CEP) will become mainstream in 2008 as companies try to analyse data and the business impact of events well before that data arrives in any kind of data warehouse or data mart. This is also known as business activity monitoring (BAM) except we are going to see monitoring of complex events (on the lookout for several events happening before triggering action) and not just single ones

6. 2008 will be the year of massive growth in memory exploitation. We will see parallel query execution continue to run across multiple shared nothing nodes in MPP systems with multiple processors, and multiple disks (as is the case today in many parallel relational DBMSs). However the difference here is that we will see this happening against in-memory data on a massively parallel scale in 2008 and beyond. With the volumes of data about to climb higher, and demand for CEP on the increase, we need to access data in memory to respond more rapidly and keep performance optimal. Massively parallel memory is therefore inevitable and will arrive on the scene this year whether that memory be in a single cluster server or deployed over a grid in a virtual memory configuration

7. Performance Management is set to grow with BAM, process management, scorecards, dashboards, budgeting and planning and Business Intelligence all being integrated into a commpon performance management suite (enterprise performance management platform). Performance Management platforms will sit on top of BI platforms but will also integrate with other enterprise infrastructure software such as business process management, portals, enterprise content management systems and live collaboration tools.

8. Web 2.0 collaboration will push its way into Performance Management. In particular, socially networked performance management will start to appear so that end users can tag metrics, graphs and reports in order to organise BI and PM content. This user defined categorising of content via tagging is known as Folksonomies and is already heavily used on the public internet on sites like Facebook, MySpace, de.licio.us, Digg, Flickr, Jotspot etc. Now it is coming inside the enterprise and will be applied to BI and PM content as well as other unstructured content. This means that users can see other users' profiles and the tags that they have used to annotate BI and PM content. From here it means that BI and PM 'tag clouds' will form showing popular BI and PM tems that lead to popular BI and PM content and metrics. Also by following BI and PM tags we will see the dynamic formation of BI social networks consisting of people within the enterprise that have similar interests in acting on BI to improve performance. People will also be able to share reports and collaborate with others (in real time - e.g. IM, threaded discussions etc.) in Web 2.0 collaborative workspaces. Wikis (group publishing) will also come together with BI so as to fuel rapidly forming BI and PM workspaces that will be of exceptional value to the business.

9. Seach and BI are set to explode into popular use in 2008 as seach opens the doors to mass access to BI content from a userbase that is not comfortable with BI tools

10. BI reports will be capable of being published in document management and records management systems

11. Master data management market size will continue to grow as companies try to wrestle with the complexity of their data and get it under control. Information and data architects will continue to be in demand with demand for such professionals potentially outstripping supply

12. Companies will have to invest again in data modelling and data modelling skills. There is no doubt that standards here are dropping, many companies still have no data modelling tools at all and also too few people are skilled in good data modelling practices.

13. Data management professionals will start to come together into integration competency centres so that people with skills in data cleansing, data integration, data modelling, master data management, enterprise content management, metadata management and ESB XSLT XML data translation are all co-located and can work together to solve the problem of enterprise data management

14. Metadata manegement will become a mission critical issue if it is not already. Business users need access to business metadata to unnderstand what data means and where it came from. Holding this metadata in spreadsheets is no longer acceptable. It must me made available to bith end users and shared across multiple technologies. 2008 will see companies looking to act to solve this problem.

Well that's all I have for now. Let me know your thoughts. I would be most grateful for your comments on any of this. Best wishes for a happy and prosperous New Year!

The Informatica 8.5 announcement includes the following:
• Power Exchange Real-time change data capture
• Integration of data quality services with SAP operational applications for on-demand data quality as you use the SAP applications (currently this is for name and address data only)
• An overhaul of the Power Center Metadata Manager to provide search, filtering and personalisation capabilities. This also includes the ability for users to annotate metadata
• A data masking option to keep secure data safe when generating test data
• Re-entrant data services and parallelised data quality for more scalability (this adds to the grid computing and the push down optimisation support added in release 8)
• A Data Quality Assistant to allow Data Stewards to participate in data integration workflows so as to review and edit poor quality data records
• Web based data quality reports and dashboards
• Pre-built Data Migration tool on top of the Informatica platform to address this kind of data integration problem

Involving data stewards in the data quality process through the new Data Quality Assistant and the Pre-built Data Migration tool certainly stand out as differentiators. The latter of these is certainly the beginning of a trend among data management vendors in that it introduces the first of potentially multiple patterns on top of the Informatica data management platform.

Not to be undone, IBM responded with the following enhancements to Information Server
• A new look WebSphere Business Glossary
• A new product WebSphere Busines Glossary Anywhere to access business metadata from your mobile device
• Integration of Information Server and WebSphere Customer Center Master Data Management
• A New Multi-Domain Master Data Management Server with pre-built integration with Information Server
• WebSphere QualityStage pre-built integrayion with SAP and Siebel
• Information Server Fast Track which automatically generation of DataStage ETL jobs (this adds to IBM’s ability to automatically generate EII virtual views and mappings for their WebSphere Federation Server)
• Metadata “Asset Interchange” to move metadata between different instances of Information Server (e.g. Development, Test and Production)
• Data Masking for data security
• Column level impact analysis
• Real-time Change Data Capture and Replication (from the acquisition of DataMirror)
• Pre-built integration of Information Server with WebSphere Integration Developer (WID) and WebSphere Portlet Factory
• Enhancements to Information Services Director so as to invoke information services via multiple interfaces including web services, EJB, JMS, REST (XML/JSON) and RSS
• More connectivity
• The Introduction of the Information Server Blade i.e. Information Server pre-installed with Tivoli Workload Schedule LoadLeveler on an IBM HS21 BladeCenter hardware blade.

It seems the two vendors in particular are fighting it out at the head of the pack. Oracle, SAP and Microsoft will have to move fast if they are to keep pace

ParAccel's new Massively Parallel Analytic Database runs on commonity hardware (first available release is Sun but other releases will follow on Dell, HP and IBM hardware) and comes in two flavours. These are:

ParAccel MAVERICK™ - a stand alone analytic platform using ANSI SQL
ParAccel AMIGO™ - a drop-in acceleration platform with query routing, synch, and syntax coverage for existing Mcrosoft SQL Server and Oracle instances

Interestingly this product works on columns during its parallel query processing rather than rows which raises some observations. For a start, in many of the DW reviews that I have conducted over the years I have seen thewell known practice of "Mini Dimensions" introduced to isolate popular columns in a large dimension (e.g. Customer) into their own separate Mini-dimension table. This is a performance 'trick' that can often speed up joins between large dimensions and large fact tables whereby if popular columns in the large dimension are selected in a query to qualify metrics in the fact table, then the join occurs between the mini dimension and the fact table instead of the much larger 'real' dimension and the fact table. As a result, join processing is faster.

The only problem with mini dimensions is that they complicate the key structures of fact tables, all BI tool business views (E.g. SAS Information Maps, MS Report Models, Business Objects Universes etc.) need to know about them in order to generate the right join SQL and tracking history across changes to columns in the main dimension and mini dimensions can be complex. The column approach to parallel query processing taken by ParAccel may well negate the need for mini dimensions as a performance tuning mechanism in many star schemas thereby simplifying design. If you are in the process of selecting a DW Appliance product it would certainly be worth investigating this point and worth a benchmark test.

In addition SQL advanced analytic aggregate functions are all column based. Once again therefore it would be worth benchmarking this in ParAccel Vs other DW Appliance products as there may well be a boost in performance here too for parallel processing at the column level. I might add that I have NOT personally benchmarked this but simply looking at the way this product is architected it would certainly warrant a look for specific analytic applications. Benchmarking is recommended however.

The ParAccel Analytic Database software is available now. ParAccel offers two licensing options
$1,000 per gigabyte for all-in-memory systems beginning at 100GB,
or $40,000 per node plus $10,000 per terabyte for disk-based systems beginning at 5 nodes.
Subscription licensing is also available starting as low as $5,000 per month.

So you can expect IBM to immediately want to integrate Information Server with the Cognos product line. I also would expect that the WebSphere Metadata Server (bundled with Information Server) will be integrated with Cognos Framework Manager so that Cognos metadata ends up in the Metadata Server itself at some future release point. IBM also has had several joint development initiatives with Cognos and so there is a lot of integration already.

The only question mark will be in the area of performance management where Cognos has had a strong position for some time. IBM's own Performance Management product IBM ActiveInsight will now have to come together with the Cognos Performance Management products. If this does not happen then the future of ActiveInsight as a product may well be in jeapordy. Time will tell.

So, the independent BI vendors are falling like nine pins as the software giants finally move in on the market. With Hyperion already falling to Oracle and Business Objects to SAP, it leaves only one large independent BI vendor left in the market - SAS. Of course SAS is privately owned and so will continue on that way until such time as its CEO (Dr Jim Goodnight) decides otherwise. MicroStrategy also remains independent also along with other smaller BI vendors. Expect further mergers and acquisitions to follow as the rest of the market consolidates to compete with the giants.

Now that Oracle and SAP have declared their hand the question is what happens next? I would be staggered if IBM continue to sit by and watch. Surely they must move as they have a real hole in their product portfolio on BI tools. This announcement only leaves two large independent BI vendors in the market - namely Cognos and SAS. With the latter of these the largest of the BI independents and also privately owned it seems inevitable that the spotlight will shine on Cognos as potentially the next candidate with IBM or Microsoft as a suitor.

It will be interesting to watch.

Of course SalesForce.com itself is no doubt keen on the BIaaS market and is already active in offering added value in terms of BI to existing clients.

Nevertheless, while simplicity, 'point your browser and go' and cool pre-built reports and graphs are the obvious attraction, there are implications when adopting BI as a Service in any business. The most important of these is that companies may need to supply their data to BI SaaS providers for upload to BIaaS sites so that 'instant BI' can be made available back to them via hosted web enabled BI tools and pre-built reports. There are also privacy regulations that have to be adhered to in this kind of situation not least the UK Data Protection Act. Companies need assurances on data protection as well as data security and should consider the implications of this in terms of giving BIaaS providers their precius operational data to be managed off-site. This is after all, the crown jewels of any business and there is no doubt that BIaaS providers would jump at the chance to know much more about their clients and would be sitting on a potential old mine with all that data. Reliability of such a service is also paramount so that BI is available when you need it. Companies considering this option should also think about what happens if they need their data back in-house and how easy is it to get it back from a BI SaaS provider. It would be madness to subscribe to such as service and overlook this requirement.

I have wondered about the potential size of the BIaaS market but it was not until I was looking at iGoogle a while back that I realised the real potential of BIaaS. Google have been steadily adding more and more services to their portfolio and are now making these services available over the internet for you to personalise with your own portal via iGoogle. All you have to do is click "Add Stuff" on iGoogle to see the huge number of instant services you can add to your portal. So what am I driving at here?

My question is this. How long before Google enters the enterprise SaaS market with avengence? Both SaleForce itself and a BI vendor could easily be a target to this Internet search giant. If you could get at hosted enterprise services in a SaaS offering just by using iGoogle "Add Stuff" to add it to your portal then how many SMBs would do it? That is a huge question. My guess however is that if it is as easy as Google are making it to add stuff on iGoogle today then the uptake by SMBs could be enormous. All Google have to do is solve the data upload problem and deliver vertical data marts for the industry of your choosing and they would no doubt get the attention of SMBs. So for those watching the BI market for mergers and acquisitions, I would not exclude Google from the mix. We may well see a very big splash if Google decides to move on the BI market to open up its stall as a BIaaS provider to SMBs. An iGoogle for Business offering would certainly do it. Who knows - I'm certainly watching with interest.

Not to be upstaged, Information Builders are spicing up reports with their Active Reports capability whereby the use of Javascript once again allows a much richer report to be made available to the user whereby much more report functionality and processing can be done client side in the browser . In addition Information Builders have also made it possible for data to be included in Active Reports so that users can conveniently take reports on the road with them in disconnected mode and carry on analysing. This they refer to as Active Reports with Quick Data.

Both of these are examples of how vendors are exploiting the power of Javascript and AJAX in BI technologies. I expect a lot more of this in the next 12 months. We may even see BI frameworks appearing whereby users can rapidly "compose" BI applications by leveraging pre-built visual components available in BI frameworks. The only hint of caution here is that if you are looking at this please don't assume all browsers are the same. There are differences across IE, Firefox, Opera and Safari when it comes to processing scripting languages. So what works on one may not on another. Also don't assume interoperability between the visual components if the components are all built using different tools.

Just like on the web today where there are hundreds of AJAX frameworks are available in abundance for rapid development there is a strong likelihood that several BI frameworks may well appear inviting end users to leverage BI "composition" tools to rapidly assemble visual BI and non-BI components into custom built rich interactive analytic applications (mashups). If multiple BI frameworks do appear we may well get caught up in the buzz of new technology and trailblaze building BI visual components galore without concern for the bigger picture of how these components all fit together on a page, how they are uniquely identified, how you deal with security when it comes to access them. or how they are managed. If you are building a personal dashboard then it may be OK because you are dealing with one technology but what about building sharable dashboards for the enterprise . In my opinion it depends where the visual compnents are aggregated into the page and what does this. A portal does this on the server by aggregating portlets to construct a page before serving up the page to the browser. Portlet interoperability is either proprietary to the portal product or standard via JSR 286 (successor to JSR 168 which does not support portlet interoperability) standard portlet interoperability.

However if BI vendors select Javascript as the mechanism to assemble (aggregate) BI visual components on to a page by using Javascript on the client (i.e. in the browser) to do this then I see no standard for visual component interoperability on the web page or between desktop widgets - it would have to be proprietary.

If you're company has multiple BI tools (which is often the norm especially in large enterprises) you may well find business analysts composing BI dashboards without IT even knowing about it. The problem is how to bring it all under the wing of the enterprise when a business user wants to mix and match BI and non-BI visual components that are assembled client side when there are no standards. The portal is certainly not dead as it can handle this. Also AJAX is not an alternative to portals - AJAX components can in fact be integrated into portals as AJAX portlets. For this reason, portal vendors are bundling industry standard (JSR 168/286) portlet containers for free with application servers and not just as part of their portal server products so that you turn AJAX BI compoenents into portlets without the need for a portlet development and still allow rich user interface interactions with portlet interoperability. As far as stand-alone AJAX interoperability is concerned, my suggestion is to watch what happens at the openAJAX consortium. Interesting that apart from the four software giants (IBM, Microsoft, Oracle and SAP) not a single independent BI vendor is a member! Seems to me its about time they joined.

That problem is one of multiple integration stacks. As I consult on a Europe wide basis I find this problem everywhere. Companies want to standardise on a single common set of infrastructure software which includes common BI infrastructure (gradual evolution to common BI platform) and common Business Integration infrastructure (portal, process management, ESB, ESR, single sign-on etc.). Yet as they buy packaged applications particularly from SAP and Oracle and upgrade to Office 2007 only to find SharePoint coming in though another door, they find themselves with multiple software infrastructure stacks in the enterprise and additional complexity they had not planned for. Plugging together multiple stacks was not on the original agenda. Do you recognise this problem?

It seems strange that the attraction of SOA is simplicity and flexibility and yet as you try to get closer to that you end up with more complexity. BI and data integration services in a SOA are much sought after but I found myself recently having to deal with questions like "Which enterprise servive registry should be use for managing BI services?" and also "should we define our business processes on our IBM WebSphere BPM software or the SAP Netweaver BPM software or the Oracle Fusion one?".

There are many more of these kinds of questions that companies are struggling with at present. It seems that vendors are forcing additional infrastructure on enteprises rather than offering flexibility to run on the infrastructure of choice that the customer wants. Yet the business case to executives is that SOA is needed because it facilitates consolidatation of IT infrastructure to reduce complexity and total cost of ownership. As one executive said to me recently "I am struggling to see the benefit". I must admit that I found it difficult to disagree with him

The trend here is clear and that is that data management tools such as business vocabulary tools (aimed at business users), metadata discovery and mapping tools, data modelling, data profiling and cleansing as well as data integration are all converging into a common platform of tools that have been integrated on a common metadata repository. Why is this needed? The reason is of course that enterprises are pushing for a common toolset for any kind of data management whether it be data consildation for data warehousing and master data management, data federation for on-demand reporting, data replication or data synchronisation. It is a natural thing to expand the use of these tools beyond popular areas like data warehousing into other applicational uses across the enterprise. Even unstructured data integration is now possible. The world of XML data integration is also on the increase including RSS Feeds as a data source. All of this is demanded at lower and lower latency. Data and metadata services in a SOA is a trend but the challenge for most of use is to identify and prioritise business areas that need to exploit these services in order to improve data supply and data flow throughout the enterprise and between enterprises

Also as companies increasingly invest in software as a service (SaaS) offerings it means that access to corporate data housed outside the enterprise is on the increase. Data in SaaS applications needs to be kept consistent with data inside the enterprise and brought back inside the enterprise to integrate with internal operational data if you have an internally hosted data warehouse.