machine learning process

Machine learning is at the heart of all analytics built on unstructured data such as text, image or video analytics. The cost of sensors – such as the gyros used in a smartphone – has been coming down dramatically over the last 2 decades. The average price of a motion sensor has dropped from $10 in the 1990s to about $0.10 today. Additionally the size of these devices has also shrunk in accordance with Moore’s law. What this means is that it is easier and cheaper than ever before to capture all sorts of data. 

Another device which also falls under the sensor category is the digital camera. The CMOS chip inside each digital camera has also been following the same trajectory of commoditization. Clearly this is one of the primary reasons for the explosion of online digital image content. 

All this exponential growth in data accumulation means two things: most of this data is never really used and identifying data that is fit and meaningful for business use gets harder and harder. This is particularly so for image data and this is where machine learning can come in. Machine learning can help with “shrinking” the data haystack when it comes to unstructured data such as image data. 

As an example consider a business that develops designs for paper and fabric: if they have been around for a few years, they might have several thousand design patterns which are sourced for production. But it would be very valuable if they knew for instance which patterns or designs have a tendency to sell better. Additionally, by correlating the sales data to design patterns, they can also determine which designs sell better in different geographies and during different seasons. But for doing this correlation, they would first have to identify “features” of their various designs. For example, they may classify their designs as one of three categories: geometric, floral and novelty.

feature engineering for machine learning
Feature engineering of unstructured data

The problem is that among the thousands of designs not all of them would fall neatly under one of these categories. Some may have more elements of floral than geometric while others may have more novelty elements than floral. Using human judgment to segment these is not only impractical, but also highly subjective. 

Anyone who is familiar with predictive analytics will immediately see that this is a “classic” classification problem and machine learning can be used to help here. There are a variety of tools that can be used for prediction, but first we need to convert the unstructured image data into a standard training data set that is needed. To that end, one must use image processing techniques. B-Designer is a toolkit that is available not only as a standalone product but also comes as an extension to RapidMiner. If you are familiar with RapidMiner, it is a breeze to quickly put together a process which first converts the images into a nice structured data set using the more than 150 pre-built operators which B-Designer provides. 

machine learning process

In a future article we will detail the steps involved and show that we can quickly generate 80-90% accuracy on the type of design categorization. As the volume of unstructured data continues to explode, the sorting, segmenting or categorizing of such data becomes a first step in image mining to help us in decision making.

Originally posted on Tue, May 12, 2015 @ 08:49 AM

No responses yet

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.