| March 28, 2012
In this O’reilly Radar article, Edd Dumbill, defines big data as:
Data that becomes large enough that it cannot be processed using conventional methods.
Needless to say, there are many people happy to apply new tools and analyses to beat big data into submission, and many companies employ their services. And there is no doubt you can do some cool stuff with big data. But is the promise of big data overstated? History suggests the answer may be yes.
The wealth of data created by e-commerce sites, social media and customer relationship management systems, promises wealth of another sort for companies and individuals, which has traditional market researchers looking over their shoulders. With massive databases comprising petabytes of data, it is all too easy to believe that the findings derived must be the truth. Data-driven discovery and decision making promises to reduce costs, boost sales and increase profits. As a result, companies are eager to mine their data for gold.
But like the gold rushes of the past, big data requires substantial investment before it gives up its riches. There is prospecting to be done: figuring out how to configure and handle the data, exploring the data to look for patterns, and writing the code to automate the process. Only then can you mine data for findings.
Making sense of big data is expensive. It requires not just expertise, but also creativity and understanding. Analysts need to know what to look for, to enable them to read the geography of the data in the same way the old-timers used to read the land.
You might think that the application of new techniques, such as machine-based learning, would ensure impartiality. But in reality, this type of analysis still only provides us with a set of generalizations that are open to interpretation. The analyst no longer defines how to solve the problem, but they still need to interpret the results and decide what to do. And therein lays the challenge. Any analysis subject to reasoning is far from unbiased. Instead, it is subject to all the biases and beliefs of the decision maker.
To see what might happen if we fail to use big data wisely, step back a couple of decades to when scanners were first introduced to grocery stores. The ready access to information on what people bought and how much they paid promised a revolution in consumer packaged goods marketing.
Fast forward to 2007 when Len Lodish and Carl Mela, wrote their paper, "If brands are built over years, why are they managed over quarters?" which found that instead of helping marketers to build strong profitable brands, scanned sales had instead led to an undue reliance on unprofitable price promotions (read more here). The paper refers to the "myopia" which results from managing brands by the data you have, not the data you need, and taking business decisions based only on short-term sales data.
In my next post on this subject, I will dig a little deeper into why the promise of big data is so alluring, and suggest what we might do to reduce the risk of marketing myopia. Meanwhile, what do you think of the opportunities and threats posed by big data? Please share your thoughts.