Do your competitors know something that you don’t? What makes your customers tick? Should you launch a 50 per cent discount offer or a buy-one-get-one-free promotion?

In which ways is your brand relevant to millennials? You may think you already have the answers to these questions because research relying on “claimed behaviours” guides your marketing strategy.

But isn’t this why so much market research goes wrong? In the mid-1980s, Coca-Cola lost millions because flawed research suggested it ditch the original taste for a sweeter recipe based on a sample of 200,000 people who were asked to compare the two products without being told that New Coke would replace the old soda product.

In more recent and rather more significant examples, the Brexit and Trump campaigns confounded predictions, revealing the many frailties associated with traditional opinion poll practices that try to gauge decision-making.

Eliminating bias from decision-making or when stating preferences requires a magic wand, according to Nobel-prize winner Daniel Kahneman, who is credited with the creation of behavioural economics, a method of analysis that applies psychological insights into human behaviour to explain purchase decisions.

The research industry has indeed relied far too much on claimed versus actual behaviour, including too much reliance on past data rather than current and predictive data. That has taken a toll on credibility and impacting its share of the budget in the marketing mix, sometimes even failing to be a part.

The good news is that thanks to the confluence of data mining, machine learning and predictive models, marketers in today’s heavily digitised environment can optimise the efficacy of their strategies. This can be achieved because research firms can measure actual behaviour, including physiological and psychometric reactions, instead of mere claimed behaviours.

By unlocking the full value of data, companies can make their customers happier, improve sales, target cost savings or improve profitability and pricing performance. This is already happening and research specialists such as UK-headquartered Future Thinking are targeting brands in the Middle East with sophisticated models designed to inject steroids into marketing and turn today’s CMOs into future CEOs.

There is more to come in terms of innovation to help support marketing decision-making as research enters a bold new era, thanks to the availability of data generated by transformative businesses in the digital and sharing economy spaces. Data that unveil details about what people are consuming, doing and absorbing, all in real-time.

Using processes that integrate data from multiple and disparate sources to help make comprehension easier, marketers can draw dramatically improved insights to grow their businesses.

Behavioural economics has seamlessly integrated into mainstream survey design with items such as TER (Timed Emotional Response). Brands can now measure non-conscious attitudes rather than just the purely banal. Tools like Galvanic Skin Response (GSR) — using lie detector technology principles — are also deployed to understand consumers’ physiological reactions to stimuli of advertising and marketing communications.

But even these highly advanced solutions may not be enough in a future where the “quantified self” becomes plain vanilla in market research circles.

Also known as life-logging, the quantified self movement will incorporate technology into data acquisition on aspects of a person’s daily life. This would relate to inputs like food consumed, quality of surrounding air, mood state, blood oxygen levels, and performance — whether mental or physical.

This is where the body itself becomes a rich source of data, whether that’s obtained via smart watches, fit bands, smart glasses, etc.

What does this all mean for traditional research? Is it still relevant? The answer is a resounding yes, for the time being that is. Particularly in sectors where there is a real necessity to get in front of respondents at the moment of truth, as when customers leave a store.

“Whoever understands the customer best, wins,” goes a famous quote about research. Marketers today have the opportunity to understand their customers better than any other time.

Those willing to embrace new-age research practices are a step closer to the coveted boardroom; those who don’t put their own relevance at risk.

The writer is head of PR and Social Media at Al-Futtaim and author of ‘Back to the Future of Marketing — PRovolve or Perish’.