Blog: Barney Finucane http://www.b-eye-network.co.uk/blogs/finucane/ Welcome to my BeyeNETWORK blog. My main goal here is to address hype issues that come up in the Internet, not to provide any overview of the BI market itself. I look forward to any questions or comments you may have. Copyright 2019 Thu, 04 Jan 2018 08:33:00 +0000 http://www.movabletype.org/?v=4.261 http://blogs.law.harvard.edu/tech/rss Negative energy prices and artificial intelligence Since renewable energy has started to become popular, an odd problem has appeared in wholesale energy markets: negative prices.

In other words, energy plants sometimes pay their customers to take energy off their hands. Usually older, less flexible plants that can't shut down without incurring costs are affected.
One solution to this problem is batteries. The idea is to store the energy when it is overabundant, and use it later when it is expensive. This is sometimes called "peak shaving".
Batteries are a great idea, but not the only solution. Another is to simply find an application that is energy hungry and can be run intermittently.

One possible application for soaking up excess energy is desalination. For example, a desert region near an ocean could build solar plants to desalinate water during the day only. The question is whether building a desalination plant that only runs 12 hours a day is worth the savings in energy. 
Another way to make use of energy that might go to waste is using it to power computers that perform analytics. The energy demand of data centers is growing quickly.

One source of energy needs is Bitcoin. Bitcoin mining consumes huge amounts of energy, so it is a great example of a use for negative energy prices. In fact there are already a lot of bitcoin miners in Western China, where solar and wind installations have outstripped grid upgrades. In these areas renewable energy is often curtailed because the grid can't keep up. So the energy is basically free to the miners.

Extremely cheap bitcoin mining arguably undermines the whole concept, but here is a more productive idea: Training artificial intelligence. For example, have a look at this link to gcp leela, a clone of Google Deepmind Alphago zero:
https://github.com/gcp/leela-zero
The entire source code is free, and it's not a lot of code. But that free code is just the learning model, and its based on well known principles. It's probably just as good as Deepmind Alphago Zero when trained, but they figure it would take them 1700 years to train -- unless of course they could harness other resources. This is partly because they don't have access to Google's specialized TPU hardware. Whatever the reason, training it is going to burn through a lot of energy.
This would be a great application for negatively priced energy. Game playing is more a stunt than a commercial application, but when they are paying you to use the energy, why not? And as time passes, more useful AI apps will need training.
So it gets down to whether the business model of peak shaving with batteries makes more economic sense than banks of custom chips for training neural networks for AI in batches. The advantage of batteries is that you can sell the energy later for more, but it's not terribly efficient, and using it directly is a better idea. Cheap computer hardware and a growing demand for AI may fit this niche very well.
This puts a whole new twist on the idea that big tech companies are investing in renewables. These companies make extensive used of AI, which is trained in batch processes. 
]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2018/01/negative_energy_prices_and_art.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2018/01/negative_energy_prices_and_art.php Thu, 04 Jan 2018 08:33:00 +0000
Understanding Artificial Neural Networks Artificial neural networks are computer programs that learn a subject matter of their own accord. So an artificial neural network is a method of machine learning. Most software is created by programmers painstakingly detailing exactly how the program is expected to behave. But in machine learning systems, the programmers create a learning algorithm and feed it sample data, allowing the software to learn to solve a specific problem by itself.

Artificial neural networks were inspired by animal brains. They are a network of interconnected nodes that represent neurons, and the thinking is spread throughout the network. 

But information doesn't fly around in all directions in the network. Instead it flows in one direction through multiple layers of nodes from an input layer to an output layer. Each layer gets inputs from the previous layer and then sends calculation results to the next layer. In an image classification system, the initial input would be the pixels of the image, and the final output would be the list of classes.

The processing in each layer is simple: Each node get numbers from multiple nodes in the previous layer, and adds them up. If the sum is big enough, it sends a signal to the nodes in the layer below it. Otherwise it does nothing. But there is a trick: The connections between the nodes are weighted. So if node A sends a 1 to nodes B and C, it might arrive at B as 0.5, and a C as 3, depending on the weights in the connections. 

The system learns by adjusting the weights of the connections between the nodes. To stay with visual classification, it gets a picture and guesses which class it belongs to, for example "cat" or "fire truck". If it guesses wrong, the weights are adjusted.This is repeated until the system can identify pictures.

To make all this work, the programmer has to design the network correctly. This is more an art than a science, and in many cases, copying someone else's design and tweaking it is the best bet.

In practice, neural network calculations boil down to lots and lots of matrix math operations as well at the threshold operation the neurons use to decide whether to fire. It's fairly easy to imagine all this as a bunch of interconnected nodes sending each other signals, but fairly painful to implement in code. 

The reason it is so hard is that there can be many layers that are hard to tell apart, making it easy to get confused about which is doing what. The programmer also has to keep in mind how to orient the matrices the right way to make the math work, and other technical details. 

It is possible to do all this from scratch in a programming language like Python, and recommended for beginner systems. But fortunately there is a better way to do advanced systems: In recent years a number of libraries such as Tensorflow have become available that greatly simplify the task. These libraries take a bit of fiddling to understand at first, and learning how to deal with them is key to learning how to create neural networks. But they are a huge improvement over hand coded systems. Not only do they greatly reduce programming effort, they also provide better performance.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2018/01/understanding_artificial_neura.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2018/01/understanding_artificial_neura.php Wed, 03 Jan 2018 15:21:00 +0000
Alphago probably isn't learning from Lee Sedol There has been quite a bit of discussion about whether Alphago can learn from the games it plays against Lee Sedol. I think not. At least, not directly. 
The heart of the program is the ”policy network” a convolutional neural network (CNN) that was designed for image processing. CNNs return a probability that a given image belongs to each of a predefined set of classifications, like ”cat”, ”horse”, etc. CNNs work astonishingly well, but have the weakness that they can only be used with a fix size image to estimate a fixed set of classifications.
The policy network views go positions as 19í—19 images and returns probabilities that human players would make one of 361 possible moves. This probability drives with the Monte Carlo tree search for good moves that has been used for some time in go computers.The policy network is trained on 30 million positions (or moves) initially. 
CNN (aka ”deep learning”) behavior is pretty well understood. The number of samples required for learning depends on the complexity of the model. A model of this complexity probably requires tes of thousands of example positions before it changes much. 
The number of samples required to train any machine learning program depends on the complexity of the strategy, not on the number of possible positions. For example, Gomoku ("five in a row", also called goban) on a 19í—19 board would take many fewer examples to train than go would, even though the number of possible positions is also very large.
Another point: Any machine learning algorithm will eventually hit a training limit, after which it won’t be able to improve itself by more training. After that, a new algorithm based on a new model of game play would be required to improve the play. It is interesting that the Alphago team seems to be actively seeking ideas in this area. Maybe that is because they are starting to  hit a limit, but maybe it's just because they are looking into the future.
So Alphago probably can’t improve its play measurably by playing any single player five times, no matter how strong. That would be ”overfitting”. The team will be learning from the comments of the pro players and modifying the program to improve it instead.
Interesting tidbit: Alphago said the chances of a human playing move 37 in game 2 was 1 in 10,000. So the policy network doesn’t decide everything.
]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2016/03/alphago_probably_isnt_learning.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2016/03/alphago_probably_isnt_learning.php Sun, 13 Mar 2016 13:13:00 +0000
Alphago is a learning machine more than a go machine The key part of Alphago is a convolutional neural network. These are usually used for recognizing cat pictures and other visual tasks, and progress in the last five years has been incredible.
Alphago went from the level of a novice pro last October to world champion level for this match. It did so by playing itself over and over again.
Chess programs are well understood because they are programmed by humans. Alphago is uses an algorithm to pick a winning move in a given go position. But the heart of the program is a learning program to find that algorithm, not the algorithm itself.
Go programs made steady progress for a decade with improved tree pruning methods, which reduce the total number of positions the program has to evaluate. The cleverest method is Monte Carlo pruning, which simply prunes at random. 
]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2016/03/alphago_is_a_learning_machine.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2016/03/alphago_is_a_learning_machine.php Sun, 13 Mar 2016 06:00:00 +0000
Actian acquires Paraccel
Paraccel was founded in 2005 by former Netezza executives as an alternative to Netezza's DW appliance idea. Paraccel runs on a commodity platform. It got lots of VC money (I've heard estimates as high as $90m) but has comparatively few customers. 

Recently Paraccel apparently hoped to generate a little revenue with hosting deals, which are always low priced. Or maybe they were just in it for the publicity. The technology is used for MicroStrategy Wisdom and I doubt MicroStrategy paid them very much. Shortly thereafter, they entered a deal with Amazon to license their technology for Redshift, which Amazon is reselling at very low rates.

Paraccel's struggles aren’t very surprising, since ALL the other vendors in the space except Teradata lost their independence in 2010/2011. Sybase, Netezza, Kickfire, Greenplum, Vertica and Aster Data were all acquired. Also HP killed Neoview in the same time frame. 

Actian is the newish name for Ingres, and controls the open source database by the same name. It also has several other databases including VectorWise, and there seems to me to be a good deal of overlap between the products. It is not a high profile company but it will be interesting to see what strategy they adopt to squeeze cash out of this highly funded and presumably expensive product.
]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/04/actian_acquires_paraccel.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/04/actian_acquires_paraccel.php Sat, 27 Apr 2013 13:42:00 +0000
How headlines misrepresent data this TIME article. The headline is "Americans Are Eating Fewer Calories, So Why Are We Still Obese?".

It goes on to cite two studies. One study shows that old people eat less fast food then young adults, and that in general food consumption is declining. This comes from the aging population I guess. The other shows a small decline (about 7%) in the calorie consumption of children. 

Neither study showed a decrease in calorie consumption by adult Americans, as far as I can tell. The naive assumption would be that consuming fewer calories would result in less obesity, but no data on the topic is presented. The headline is a complete misdirection.

The moral of the story is you should limit your observations to what the data actually says.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/03/how_headlines_misrepresent_dat_1.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/03/how_headlines_misrepresent_dat_1.php Wed, 27 Mar 2013 13:56:00 +0000
BI Trends for 2013

Here are some trend ideas that are floating around the internet. There are basically three kinds:

New data sources: This mostly means Big Data, whatever that means. (Personally, I like Amazon's definition, which is "too much data to handle on a single server". Other definitions strive to include appliances like Teradata and HANA.)

New applications: Sentiment analysis, predictive analytics, collaboration

New UIs and functions: Better dashboards and visualizations, more self service and agility, voice interfaces, mobile.

New Platforms: This is mostly in memory, cloud and SaaS.

My take on all this is simple:

New data sources: I think big data is still mostly for online and mobile providers. It's true that manufacturers and retailers are trying to figure out how to make better use of the huge amounts of data their business directly and indirectly generates. But this business is still heavily dependent on boutique providers that bring a lot of domain knowledge and deep understanding of statistics with into the deal. I do not think it will have much impact on existing BI business. It's something different.

New applications: The same remarks apply to sentiment analysis as to big data. Predictive analytics is a more interesting market, but to find large, non-specialist audiences, vendors need to prove that their "black box" predictions are as reliable as the expertise of business users without explaining the math behind them.

New user interfaces and functions continue to appear, but I believe that as long as most BI companies stick to solving the easy problems, like making software look cool in a demo, and ignore the harder problems, like user-friendly data governance, there will be no big changes here. Mobile has surprised me, but it still hasn't made a big difference in the BI business.

New platforms. It is good to remember that business users don't care what platform is used, and that the most successful projects are controlled by business users. A platform is only good in the sense it delivers things like speed and convenience. It doesn't add any value per se.
]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/03/bi_trends_for_2013.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/03/bi_trends_for_2013.php Thu, 14 Mar 2013 09:13:00 +0000
How headlines misrepresent data this TIME article. The headline is "Americans Are Eating Fewer Calories, So Why Are We Still Obese?".

It goes on to cite two studies. One study shows that old people eat less fast food then young adults, and that in general food food consumption is declining. This comes from the aging population I guess. The other shows a small decline (about 7%) in the calorie consumption of children. 

Neither study showed a decrease in calorie consumption by adult Americans, as far as I can tell. The naive assumption would be that consuming fewer calories would result in less obesity, but no data on the topic is presented. The headline is a complete misdirection.

The moral of the story is you should limit your observations to what the data actually says.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/02/how_headlines_misrepresent_dat.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2013/02/how_headlines_misrepresent_dat.php Fri, 22 Feb 2013 09:20:00 +0000
Everything is a computer
But smart phones aren't really phones at all, they are just palmtop computers that include an interface for cellular networks and a phone app. They are called smart phones for marketing reasons -- because the phone companies use them to lock consumers into overpriced network contracts. If photography were the killer app, they would be called smart cameras, which is just as appropriate. The only thing that seems to be keeping the entire electronics industry from being swallowed up by the black hole of Moore's Law is this kind of marketing wheeze and the (rapidly falling) price of screens.

I'm old enough to remember when Radio Shack was a national treasure. Now it's just a place to buy batteries while being harassed by a hard selling salesman from some telecoms oligopolist best known for hating its customers. And in this age of smart hearing aids, robotic assembly lines and fly-by-wire airplanes, its not just consumer electronics that is being computerized.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/12/everything_is_a_computer.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/12/everything_is_a_computer.php Fri, 30 Dec 2011 14:17:00 +0000
Mobile business intelligence expectations here, together with some opnions by Howard Dresner.

The interview reflects my opinion pretty well. I see mobile BI as a way to find new types of customers for BI more than as a way to replace existing installations. Too bad I didn't mention salespeople on the road, I think they are an important potential market as well.

Another point is that I think most people who said they expected mobile BI to be in use within 12 months were being too optimistic. The BI Survey 11 will address this question.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/12/mobile_business_intelligence_e.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/12/mobile_business_intelligence_e.php Wed, 14 Dec 2011 13:24:00 +0000
Chasing new trends
What is interesting is that they are also presenting a social media tool called Tibbr (wonder how they came up with that name!) and a cloud version. There's nothing wrong with this of course, but it seems to me that it doesn't fit their bus and/or ROLAP approach very well.

Their justification for the investment is that some analyst or other is predicting fast growth in this area. This reminds me of what an important ole analysts play in the market. Thanks to the analysts, a lot of BI vendors are jumping on the cloud bandwagon, even though the cloud sales channel is very different from what most BI vendors are accustomed to, and that the idea of mving data off site and then back onsite adds complexity.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/11/chasing_new_trends.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/11/chasing_new_trends.php Thu, 17 Nov 2011 03:11:00 +0000
Enterprise BI and agility post.

QlikTech has a what they call a "land and expand" policy, which means getting a single department on the tool and expanding from there. Actually, all BI companies that can deliver departmental solutions have something similar. The reason for this is simple: The cost of sales for selling to a company that is already using the tool is much lower than for completely new customers. In fact, a lot of BI tools spread through companies from department to department this way.

Now QlikTech is concentrating more on enterprise accounts. So it's interesting to see that the company is moving away from the previous claim that the tool is a replacement for a data warehouse. I think that any attempt on their part to compete as an enterprise solution would just distract them from their end users.

A lot of BI companies go through a similar life cycle as they grow. Most start out as ways to create departmental solutions, which tend to be faster, more agile projects. As they get bigger management tends to concentrate on larger accounts, which means making sure they are acceptable to the IT department. But IT is more interested in keeping processes running than in agile development. As a result, the products tend to become more complex and less suitable to agile solutions.

This is a big issue for QlikView right now because they have grown so quickly in recent years. But currently the company seems to backing away from radical changes in the tool. But it applies to and BI tool that is growing.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/08/enterprise_bi_and_agility.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/08/enterprise_bi_and_agility.php Thu, 18 Aug 2011 03:37:00 +0000
Why short projects are part of agile business intelligence
Delivery working software with two months may sound a bit extreme, but there is good evidence that short projects are more successful than long projects. In fact, our research in the BI Survey shows that the application should be rolled out to the users less than six months after the product has been selected. We have found the same result year after year in the ten year history of the Survey. Amazingly, project benefits start to fall as early as a month after the product is selected, and continue to fall thereafter. And of the many project parameters we study, none shows as clear an effect on project success as project length.

These results from the BI Survey provide clear empirical support for the idea of using agile methods in business intelligence projects. The results have remained also consistent since we started the Survey ten years ago, long before the idea of agile development or agile business intelligence became popular.

But why do short projects work so much better? Our research shows that the main problems that arise in longer projects are organizational, not technical. Needs change over the course of time, and end users lose interest in the project. Disagreements over the project goals arise. Lack of interest and disappointed end users are a major issue in business intelligence.

And needs certainly do change quickly in business intelligence. For example, another study we carried out shows that three quarters of European companies modify their planning processes one or more times a year. In an environment like this, a project that takes a year to implement is quite likely to be obsolete before it is finished. Even a six month wait can push potential users to look around for a more agile solution.

The problem this creates is that not all business intelligence projects can be carried out within a few months. This is especially true when major data management issues need to be addressed. The agile solution to this is to find way of splitting large projects into smaller ones. The usual argument against this approach is that it creates the risk of reducing efficiency in the long term. But the agile methodology is to measure success in terms of working software delivered in the short term, instead of attempting to meet nebulous long-term goals.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/07/why_short_projects_are_part_of.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/07/why_short_projects_are_part_of.php Thu, 28 Jul 2011 14:44:00 +0000
Business Intelligence, the semantic Web and Alltop
According to the site 'The purpose of Alltop is to help you answer the question, ”What’s happening?”' Alltop is a 1990's style Web registry maintained by human editors.

But Alltop's business intelligence page has several problems that make it less useful than it could be. The page has fallen victim to a semantic Web style gotcha. Like many other phrases, business intelligence means different things to different people. If you don't disambiguate somehow, a web registry based on a phrase may make no sense.

There are three distinct meanings of the phrase "business intelligence". The first is something about software for analyzing business data -- like my blog. The second is news about businesses, which is interesting but unrelated. These are some of those blogs:

MEED NEWS, EMARKETS.DE, B2B TRADE, DEALBOOK, ARUNDEL BUSINESS NEWS, FOREX TRADING INFO, THE FINANCE KID, ARBITRAGE MAGAZINE, SMALL BUSINESS SUPPORT

The third meaning is based on a completely different meaning of intelligence -- intelligence as in IQ, as opposed to intelligence as in information for analysis. In the sense business intelligence just means being smart about business, which could men just about anything.

So Alltop's business intelligence page contain sites that are not at all related to the #businessintelligence tag on Twitter. A lot of these seem to be advice about sales and entrepreneurship or general management consulting blogs. A few are just political blogs, or blogs about general market or marketing trends. They're fine in their way, I guess, just misplaced. Here's a list:

STATSPOTTING!, WSJ: THE NUMBERS GUY, MANAGE BY WALKING AROUND, BUILDING BUSINESS VALUE, CORPORATE MANAGEMENT STRATEGIES, KNOWLEDGE WORKS, HUSTLEKNOCKIN', THE THINK HERE BLOG, LEAD VIEWS, THE SOLOPRENEUR LIFE, SMEDIO, INTERCALL BLOG, GLOBAL INSTITUTE FOR INSPIRATION, SMALL BUSINESS SUPPORT, RED WING SOFTWARE BLOG, FRED67 B2B REFERRAL CLUB, RESULTS.COM BUSINESS GROWTH TIPS

I'm not criticizing any of these guys, just saying they seem to be improperly categorized.

Also Alltop is syndicating advertising material thinly disguised as blogs. Of course, maybe they're getting paid, what do I know? If not they should be. The following are BI vendors, which may or may not be problem:

RED WING SOFTWARE BLOG, LOGIXML, BIME - SAAS BUSINESS INTELLIGENCE (BI), BLOG.JINFONET.COM, MICROSTRATEGY 101, PANORAMA BUSINESS INTELLIGENCE

In addition, there are several aggregators -- Yahoo, and Beyenetwork twice. These guys can be seen as comptetitors, I guess.

In the end I think that the lack of careful stewardship reduces the usefulness of the site. The problem is that business intelligence is a vague term and needs a semantic Web to be useful. Manual editing in a web registry is a workaround, but it is not being used here to much effect.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/07/business_intelligence_the_sema.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/07/business_intelligence_the_sema.php Sun, 17 Jul 2011 09:50:00 +0000
Discovering hierarchies in columnar databases columnar and Wayne Eckerson asked me for a clearer explanation of what I mean by columnar databases "discovering hierarchies".

For example consider the approach of two well known products, IBM Cognos TM1, which is multidimensional, and QlikView, which is columnar.

My definition of a data model is a structure that is informed by an administrator, or set down in the master data. To me this is different to a structure derived from analyzing the transactions. In the following simple example, let's say I have two sales teams, one for dental hygiene products and one for soap.

If I were designing a data model in TM1, then I could create a hierarchy, which is a set of parent child relationships between the departments and the products they sell. If the soap people cross-sold some toothpaste, it would have no effect on the hierarchy, because it is predetermined by my idea of how my company is supposed to work.

If I were to import the same data in QlikView I could create a report that showed me the relationship between the sales teams and the products without defining the model. Once the data is imported, QlikView recognizes the relationships automatically.

When the soap guys cross-sell toothpaste, QlikView discovrs the new relationship, but the hierarchies stay the same in TM1, because that's how I defined the model. To me this is the key difference. On the one hand the structures are coming directly from the actuals, and on the other hand they reflect my predefined perception (or "model") of what is going on.

So columnar databases typically discover the relationships automatically, and multidimensional databases allows you to define the relationships as you want them. Another way to look at this is that the transactional data drives the master data structure in a colunmar database, but those structures are wired into the multidimensional model.

So which approach is better? It depends on the application.]]>
http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/07/discovering_hierarchies_in_col.php http://www.b-eye-network.co.uk/blogs/finucane/archives/2011/07/discovering_hierarchies_in_col.php Sat, 09 Jul 2011 11:47:00 +0000