The big risk of big data

by User Not Found | March 28, 2012

In this O’reilly Radar article, Edd Dumbill, defines big data as:

Data that becomes large enough that it cannot be processed using conventional methods. 

Needless to say, there are many people happy to apply new tools and analyses to beat big data into submission, and many companies employ their services. And there is no doubt you can do some cool stuff with big data. But is the promise of big data overstated? History suggests the answer may be yes.

The wealth of data created by e-commerce sites, social media and customer relationship management systems, promises wealth of another sort for companies and individuals, which has traditional market researchers looking over their shoulders. With massive databases comprising petabytes of data, it is all too easy to believe that the findings derived must be the truth. Data-driven discovery and decision making promises to reduce costs, boost sales and increase profits. As a result, companies are eager to mine their data for gold.

But like the gold rushes of the past, big data requires substantial investment before it gives up its riches. There is prospecting to be done: figuring out how to configure and handle the data, exploring the data to look for patterns, and writing the code to automate the process. Only then can you mine data for findings. 

Making sense of big data is expensive. It requires not just expertise, but also creativity and understanding. Analysts need to know what to look for, to enable them to read the geography of the data in the same way the old-timers used to read the land. 

You might think that the application of new techniques, such as machine-based learning, would ensure impartiality. But in reality, this type of analysis still only provides us with a set of generalizations that are open to interpretation. The analyst no longer defines how to solve the problem, but they still need to interpret the results and decide what to do. And therein lays the challenge. Any analysis subject to reasoning is far from unbiased. Instead, it is subject to all the biases and beliefs of the decision maker.

To see what might happen if we fail to use big data wisely, step back a couple of decades to when scanners were first introduced to grocery stores. The ready access to information on what people bought and how much they paid promised a revolution in consumer packaged goods marketing. 

Fast forward to 2007 when Len Lodish and Carl Mela, wrote their paper, "If brands are built over years, why are they managed over quarters?" which found that instead of helping marketers to build strong profitable brands, scanned sales had instead led to an undue reliance on unprofitable price promotions (read more here). The paper refers to the "myopia" which results from managing brands by the data you have, not the data you need, and taking business decisions based only on short-term sales data.

In my next post on this subject, I will dig a little deeper into why the promise of big data is so alluring, and suggest what we might do to reduce the risk of marketing myopia. Meanwhile, what do you think of the opportunities and threats posed by big data? Please share your thoughts.


Leave a comment
  1. Le BrandDoctor, July 29, 2013
    BIG Data is one of the strategic stakes behind the Publicis-OMD merger. Like all Analytics, BIG data only has the intelligence/added value that your strategic vision brings to it....BIG data , the after shock of the Digital revolution, is the next big shift/game changer for companies, marketers & strategists who understand its incredible potential & will learn how to empower and drive customer centric transformation with highest return & performance...
  2. trevor attridge, April 27, 2012
    It is better to be 
    vaguely right than exactly wrong.
  3. Ferdie Jorge, April 03, 2012
    Many companies, especially the big global ones, have loads of data at their disposal - but they don't even use them smartly. I would argue that the future will transition to small, but meaningful and targeted information - which is more actionable and not overwhelming!!
    It's important to take stock of the data that you already have, formulate a working hypotheses, and zoom in to few information areas that would make the most impact to the client. A bit of pre-work up front, but well worth the effort.
  4. Nigel, April 02, 2012
    Thanks Mats, totally agree with your comment. 
  5. Mats, March 30, 2012
    As always, data is not the same as information and definitely not the same as insights. Consequently, the big fallacy with big data is that it gives the impression of providing big insights. It will help us show what happened in more detail, but (usually) not why - and "why do people do what they do" is still the big question for anyone who wants to influence consumer behaviour.
  6. Nigel, March 29, 2012
    Thanks for the comments. Believe me, Jason, I prefer your second ending!
  7. Jason M. Sherman, March 28, 2012
    Big data... Since history repeats, maybe this movie has two alternative endings?:

    1) Big data supplants other sources of knowledge. The data are crunched, munched and the answers, voila!, are spewed out. (Remember the 2000 prequel, "CRM Voila?")

    2) Big data are considered, in combination with other methods, by people trained, knowledgeable and smart about what customer insights they can generate economically and reliably from every source.

    -Jason M. Sherman
    Whyze Group
    Qualitative, CX, UX Research and Innovation
  8. phil herr, March 28, 2012
    I think the prospect of big data is seductive. Just think -- no longer need to worry about attitudes and other "messy" elements of the human mind? No cares about emotions -- just focus on behavior and follow the money.  Oh, wait a minute, isn't that what got us to this state before? never mind.

    My flippancy aside, I believe there is a role for big data. It will enabale us as researchers to better understand consumer behavior. But not as a surrogate, but really as an adjunct. It will become a marvelous source of hypotheses and in many cases the starting point for exploration. But it won't replace the necessary research that tells us why, and explains that behavior.

    Leave a comment