What is your data not telling you?

by Nigel Hollis | March 29, 2017

Today we are drowning in a sea of data. Companies the world over – including Kantar Millward Brown – are using algorithms and cloud computing to beat that data into submission and force it to cough up its secrets. But what if the data is not measuring the most important things you need to know?

In discussing the topic of infobesity at the Next Forum in the King Abdullah Economic City I was reminded that  my colleague Phil Herr circulated a link to this article on Brand Strategy Insider which addresses precisely this issue. I will let you read Derrick Daye’s take, but I will briefly recount the central story here because it is a salutary reminder that we need to ensure we have the data we need not simply use the metrics that are most easily available.

Abraham Wald was a mathematician during the Second World War who was tasked with figuring out where to add armor to long distance bombers to better protect them over hostile territory. In an attempt to answer the question Wald started to analyze the places where the returning bombers were most damaged, but then realized that the data not going to help answer the real question. The planes he was studying had made it back in spite of their damage. The real question was where had the planes not been hit?

sea646x366

Today marketers and researchers risk mistaking the data that is easily available to them as defining the problems they need to solve, rather than taking a step back and figuring out what the real question might be and then figuring out whether the existing data holds the solution or not. I suspect that in many cases huge opportunities go unseen simply because the available data has set limits to our thinking.

This is nothing new, of course, data has always defined how we see the world. One of the abiding problems with data generalizations is that people come to accept them as always true. I have spent over thirty years trying to figure out how brand attitudes and advertising response affect people’s behavior and then systematizing that learning to facilitate research and guide future actions. But that does not mean I assume that these generalizations cover every eventuality.

The benefits of generalization and systematization in terms of efficiency and effectiveness are huge. However, there is always the risk that a difference in circumstances will produce a different result. The risk may be low but any model – conceptual or statistical – will produce the wrong answer if the context changes. But there is a much bigger risk to not having a generalized framework on which to base your analysis and judge the results. As the economist Ronald Coase states,

“If you torture the data long enough, it will confess.”

To what it will confess depends on whether you have the right data to start with, and how the findings fit with your existing understanding of how things work. But what do you think? How big is the risk that the available data defines what we believe is important rather than seeking out real insight? Please share your thoughts.

6 comments

Leave a comment
  1. Gonzalo Fuentes, April 05, 2017
    Nice post Nigel. Has anyone heard of Click Troughs? Likes? "Engagement metrics"?...obviously a rhetoric question. Most advertising campaign reports from most (if not all) media agencies (and many publishers) use this data as default...why? Because it exists and it is "free". But what does it mean? We live in a crazy world where the "measurement" has become more important than the outcome and we find ourselves optimizing and doing all type of recommendations and changes to drive those measures and not to drive an outcome.
  2. Mike Berendes, March 30, 2017

    I spent many (too many) years torturing point-of-sale and 3rd-party data to get at an insight.  

    I found that when I visited the store and asked random shoppers about the topic at hand that their answers would provuide the missing ingrediant to better structure the data to get at the true, unbiased insight.

    This people-first approach got me thinking: are marketers and analysts focused on the wrong center?  People should trump impersonal data every time, at least if you're looking to tell a compelling insight story...

  3. Per Kristensen , March 30, 2017
    If you're interested in good answers, never let data come in the way for the question. 
  4. Marcelo Gomes, March 30, 2017

    Making poetry

    Data is like love, sometimes blinds us and does not allow us to see the problems under our nose.

  5. John Cripps, March 29, 2017
    I love when the data confesses. I use Proc Waterboard in SAS most of the time.
  6. Ed C, March 29, 2017
    If all you have is a hammer, then every problem looks like a nail. I agree, we often drown when not focused on the real issues.

    Leave a comment

    Share