Big data: are we data rich and knowledge poor?
26 March 2013

I sat through two sessions at this week’s MRS Conference, ‘Shock of the New’ devoted to ‘Big Data’ – one to discuss the strategic issues, and one on tactics (how to do it). Interesting though they both were, I remain confused about what constitutes ‘Big Data’ and what this means both strategically and tactically. I bet I’m not alone.

Yes, we all know that the amount of data that might be of value in potentially creating additional competitive advantage is, like the universe, constantly expanding and at great speed (‘infobesity’?), but this is not a new issue.

Anyone in the data game has been challenged by this since the application of computers in marketing created a new sector build round customer databases and marketing analytics in the 1980s. Readers Digest in the UK were the leading pioneer in this field as far back as the late 1960s, and for many years the epitome of best practice with astoundingly successful direct marketing campaigns, especially for their range of coffee table books.

This success was underpinned by a process for developing new products that integrated market research with their database knowledge, leading to extremely precise forecasts of likely demand based on propensity models and tightly targeted marketing, including ‘hold-out’ control samples. It was a highly disciplined process. I know this was replicable, because we applied similar methods at the AA in testing and launching new products and services with similar success.

But despite this illustrious heritage in knowledge management and application, RD in the UK is now a pale shadow of its former self, shedding the once highly profitable books and merchandise activities, to concentrate on keeping the magazine alive. A wheel that appears to have gone full circle.
RD’s strategy was underpinned by the management of data at strategic and tactical levels focused on achieving the business goals.

Listening to the two panels, I was left wondering whether organisations today are increasingly data rich, but remaining knowledge poor, struggling to know how to manage the mountain of information available to them today and extract value that leads to real, and sustained, competitive advantage.

Is data management a real strategic priority within organisations; are the disciplined, science based methods, as utilised by Readers Digest, being applied in executing the strategy; is data management firmly aligned with the business goals?

To be honest, the panels didn’t throw too much real light on this, but I detected some glimmers. For example, in the strategic level panel, Clive Humby, a leading innovator in the field of data management and an MRS Patron, argued that market research and marketing science needed to be brought together for success in harnessing the potential from ‘big data’.

However, despite the undoubted success of the data led Clubcard strategy, Tesco isn’t quite the shining star it once was. Humby’s view echoed a similar call for action by Rachel Kennedy at the IJMR Research Methods Forum last November.

Mark Risely (Google) in the second session also advocated this marriage, and argued the need for developing hypotheses and testing these through experiments – a disciplined ‘test and learn’ process, that was at the heart of RD’s methods. Remember, Google is the organisation whose Chief Economist, Hal Varien advocated statistics as a savvy career move.

For Risely, market research methods remain essential in addressing the ‘why’ questions of human behaviour to provide context within ‘big data’, but I know that others believe that if you can build predictive models with sufficient precision, the need to know the ‘whys’ goes out of the window! ‘In the world of big data, this kind of relationship has a huge amount of power: once we know two things are associated, we don’t need to know why’ (‘From guessing game to tyranny’, James Ball, Guardian 12th March).

I believe you dismiss the ‘why’ at your peril!
As the RD and Tesco examples show, despite expertise in data management, the going can still get very tough – whether this is due to other factors (and maybe every dog has its day), or that the data management strategy is no longer the focus it might, I don’t know.

Someone said (I don’t recall who) that the lessons of the past shape the future. Maybe we can reduce the impact, or fear, of the ‘Shock of the New’, at least for dealing with ‘big data’, by looking more closely at the experiences and practices of those pioneers who ploughed the original furrow.