With all the hype surrounding predictive analytics (PA), one may be forgiven for mistaking it with fortune telling. In spite of the promise of predictive analytics, it is dangerous to treat it as a crystal ball. Using it has several challenges and certainly will dispel any such illusions.
Recent history serves up a cautionary tale. Business intelligence (BI) which preceded PA was pretty much similarly welcomed. Bill Cabiro, a BI expert estimated recently that less than 5% of employees in organizations that have BI tools actually use them! Before wondering if this is the fate of PA in the years to come, let us look at some basic differences between business intelligence and predictive analytics.
Business intelligence is about generating alerts and providing dashboards which light up depending upon which key performance indicator (KPI) is triggering it. Thus BI covers breadth of all the problems a business may encounter and is a sort of rear view mirror. Business Analytics on the other hand is about getting deep into the data that business generates and extracting crucial nuggets of information, to make “predictions”. The process of performing business analytics is also called data mining.
So if you simply want to know where the problem is coming from, use your business intelligence dashboard. But to really understand whats driving the problem, you need to perform data mining. Which means that business analytics requires statisticians, managers to run the operations, and “translators” to bring the message back to the busy or short-sighted (or both) executives.
The complexity of a PA undertaking and the challenges that confront it during adoption virtually ensure that it is no crystal ball when it comes to ease of use!
Clearly the first key challenge is handling the deluge of data. HorizonWatching estimates that 15 petabytes of data are created everyday which is equivalent to adding the material in 8 US libraries per day! This requires tools that must analyze, correlate and potentially take action nearly 200 times a second if PA were to be incorporated into a BI style dashboard!
The second challenge is cultural. Experian says that companies still use gut feeling to make decisions rather than facts or data. Changing this corporate culture would speed up adoption and real benefits from PA will only follow after this.
A final challenge is deployment of a PA model. After reducing a data set to a few key variables, imputing missing values and addressing outliers properly, the model that is built will be so highly “personalized” that, overfitting may become a problem. Not withstanding this problem, deploying the results so that a new data point can be classified or predicted is currently not an easy task and is not embedded in applications.
Democratizing predictive analytics will require not just technical training for new analysts, but cultural education of executives. Analysts will benefit from having access to real time discussions and training material. Executives will benefit from its jargon-lite approach to problem solving.
Originally posted on Thu, Jun 23, 2011 @ 10:10 AM