Key Performance Indicators: effect of wrong data or wrong questions
The purpose of using analytics is to help make sound decisions using data as opposed to making shoot-from-the-hip decisions using instinct or gut-feel. This is all well and good, however there is one pitfall to watch out for before starting on the analytics journey: wrong data or wrong questions will derail the best efforts. What do we mean by this?
Wrong data happens when you know what question you want to have answered, but you don't start with appropriate data. A client of ours had gathered hundreds of gigabytes of data from experiments. This data reflected physical characteristics of a product they had designed and they were testing how it performs under varying operating conditions and environmental effects. So the data simply reflected how their design would perform. But the questions they really wanted to answer were these: how can we reduce the time it takes to run each experiment, and what experimental settings influence how quickly the experiment can successfully conclude. In addition to knowing when to stop the experiments and where to take their measures, they also needed to understand the confidence intervals around these numbers. Clearly the focus of analytics was on trying to reduce the costs associated with running expensive experiments while at the same time ensuring that the runs complete successfully.
But the data they initially started with did not have any information about what constituted a successful run. Nor did it select other critical information such as experimental run times which is what they needed to optimize! So when they decided to use KeyConnect to identify key performance indicators, the essential question was which parameters are key to drive experiment optimization, but they had the wrong set of parameters going in. Using KeyConnect to identify KPIs is of course easy with any data. But the problem they were facing was not with the choice of analytics technique, but a more fundamental garbage in - garbage out issue.
Fortunately, fixing this problem was quite easy. They did not have to run any new experiments (thankfully). All the data needed was already available, but just not properly extracted. As a next step, they needed to reformulate the business question as follows: Which experimental set up parameters have the strongest influence on the stability of the measures over the 5000+ iterations? Clearly the sooner the measures (in this case, average forces) become stable, the earlier they can stop the experiment. The follow up to this question would be how do we select/tune the parameters so that we can achieve shorter experimental run times.
Wrong questions: Sometimes the converse problem will appear. We start out with all the data we could possibly obtain, but end up asking incorrect questions. According to this new article (requires sign-up) on the Harvard Business Review on Big Data and role of management, the first question a data-driven organization should ask itself is not “What do we think?” but “What do we know?” Correctly, they say that this requires a move away from acting solely on hunches and instinct. It also requires breaking what the identify as a bad habit: pretending to be more data-driven than companies actually are. "Too often, we saw executives who spiced up their reports with lots of data that supported decisions they had already made using the traditional HiPPO [highest paid person's opinion] approach. Only afterward were underlings dispatched to find the numbers that would justify the decision."
Deploying analytics successfully will require 5 key steps. The first and possibly the most important one is to spend sufficient time formulating the questions for which we need the answers to. Then comes the second step: take inventory of what data is currently available and understand if the correct data is indeed available. As we saw in the example, obtaining correct data may not be very difficult. The final steps involving modeling, and deploying the model are also equally important, but they are downstream problems.
Do you need to deploy analytics affordably in your organization? Find out more by participating in our pilot program by clicking below.