Why researchers need to ask less and impute more

by Nigel Hollis | September 07, 2015

Time is short for all of us. Survey respondents do not want to answer lengthy questionnaires any longer (if they ever did). But contrary to the most optimistic beliefs of the big data fanboys, sometimes you just need to ask people a few questions.

If you combine attitudinal data with behavioral data from the same people, it quickly becomes apparent that attitudes do matter. People, a lot of the time, do act in line with their beliefs, particularly when it comes to paying a price premium for a brand. Look at that data over time, and you come to realize that brands which people believe are meaningfully different are the ones that return good financial results over the long-term.

Taking a survey

So attitudes do matter, but collecting those attitudes in a survey has typically been a lengthy process. Even today some surveys still exceed 20 minutes, with huge consequences for completion rates and the cost to the client. To quote one survey respondent,

“If the survey is too long, you will click anything just to finish. This makes the survey worthless. You would think they would know this.”

How can we fix this problem? There are at least three ways that Millward Brown has risen to this challenge.

First, do not ask the question if that data can be collected elsewhere, for instance, from sales panels,search queries or social media. But you have to be confident that the replacement data is valid. One reason to continue asking one or two duplicative questions in your survey is to make sure the different data sources are telling you the same thing, or to help explain why when they do not.

Second, focus in on real key performance indicators; the ones that predict and, better yet, lead behavior. Millward Brown’s Meaningfully Different Framework has reduced the number of questions you need to ask to monitor attitudinal brand health to five, at the same time improving the link with sales, price paid and potential to grow.

Third, do not ask everyone about every brand. Adaptive Brand Sets is a technique that allows a questionnaire to serve brand lists dynamically. A stratified sampling algorithm enables us to both cover a whole category and also reduce the brands asked to a list that is relevant, representative, and easily answerable.  The backend imputation then allows us to do the respondent-level analyses as well as removing any ‘brand fan’ bias. Testing of this approach has proven that respondents not only like the survey process more, they provide richer information with stronger discrimination between brands.

Like it or not I suspect that surveys are going to be with us for a long time yet. They may be served dynamically, in micro segments and across different devices, but if you want to get insight into what is going on in people’s heads sometimes you really do need to ask. So what do you think is the future of surveys? Please share your thoughts. 

6 comments

Leave a comment
  1. Jamie Lord, September 09, 2015
    Making sense once again
  2. Benjamin N Mutua, September 09, 2015
    Sample survey questionnaires shouldn't include nice to have questions but rather targeted questions addressing clients' business and marketing challenges.
  3. Mauro Schneider, September 08, 2015
    Surveys should always be another way to add value, not only get information, no matter how important they might be. 
  4. Mohtashim Abbasi, September 08, 2015
    Shorter, more focused surveys are the future. Consumers these have less time and more distractions making the quality of long form surveys even harder to guarantee. Perhaps the way forward is to have a population representative survey panels similar to Household Panels and Retail Panel, allowing researchers to skip the whole recruitment and category/brand usage understanding part of surveys and sticking to the must have questions. Also over time this will create a much more richer understanding of consumer groups & individual as learning's across surveys can be looked at holistically to develop better understanding of what drives consumer understanding.
  5. Faizan Mukaddam, September 08, 2015

    I feel Surveys have to be more engaging, there should be a benefit for the respondent (not a gift voucher) who fills in the survey. Most importantly they should be so easy that they could be self administered. Surveys should have stories to tell (relevant to respondent not just the brand/client)

    I think creating surveys should move up a step ahead and should be treated as importantly as a 'marketing tactic' (that too an effective one). We should think about a central Idea around the survey which can make it engaging

  6. Abhinav Chauhan, September 08, 2015
    Until and unless we won't receive monotonous responses from respondents we should not stop questioning clients. This also support us to re-structure interview guide (semi-structured / structured) and it also strengthen the power of survey and opens the dialogue with stakeholders. 

    Leave a comment

    Share