Is market research invited to the big data party?
22 October 2014

The Association of Survey Computing (ASC) held their second one day conference of 2014 at Imperial College, London, on September 26th on the theme of ‘Making connections: unleashing the power of data’.

As usual, a very interesting day, but I want to focus on just one presentation, entitled ‘Market research at the big data party’, given by Mike Page (Blueocean). Page discussed the extent to which market research is viewed as a key input in the world of big data, and what might be the challenges in doing so.

Sadly, I felt it was like going back a quarter of a century, to when I was attempting to merge market research with data from a customer database and there was a major debate on whether market researchers should be involved in handling data from sources other than surveys. 

I’ve written on this in several blogs and editorials before, but progress in widening the research net seems to be painfully slow and as a result the possibility of market research as we know (knew ?) it becoming marginalised in favour of other sources becomes more of a reality.

Readers Digest in the UK could do this in the late 1960s, but it is still apparently a rare occurrence. For example, the ‘First T’ method of matching survey data with customer data, developed by BMRB/Dunn Humby (and by other research agencies, such as TNS) in the late 1990s never took off (a model you will find in the Information Commissioner’s recent data anonymisation code of practice). Such models enabled sophisticated segmentations to be developed, and propensity models to be built. 

Page argued that market research data missed out on the key enterprise data warehouse (EDW) phase of a decade ago. This in itself identifies how behind the times we are, as the EDW trend started to gain momentum in the late 1990s (I have the slides and texts from that era to prove it). However, whatever the time line MR missed this boat, meaning that when client companies built their EDW structures, MR data was not catered for in the design. Why? 

Page identified three important factors: companies not viewing MR data as a key asset; datasets remaining with the agency rather than being accessed by the client – EDW’s are primarily populated with data flowing within the company; no common data formats in survey design. 

I would argue two other key reasons. Firstly, the market research function would not commonly be involved in the development team for the EDW, and possibly not even consulted. Marketing and IT were not commonly in touch in those days, nor would the research team. 

Secondly, the data numbers would be peanuts in volume compared to the data flowing from other sources within the organisation.

But as Page also discussed, the technical architecture required to cater for research data, at participant level, would possibly require differences in EDW design that would not be warranted due to the marginal volumes of data. So, where do we go from here? 

Page discussed three approaches. Firstly, linking survey data to business intelligence systems and tools; secondly, building respondent level architecture; thirdly, a combined approach. 

I think that the first option is the most tenable. However, as Page warned, all of this is made more difficult by not have common formats in survey data, and the need to build in hooks in surveys to link with other data sources. Is this happening? Would it constrain the research input to the organisation by imposing too much regimentation in survey design?

However, Page believed that there is some real progress emerging in ‘setting the data free’ within organisations. He advocated a ‘system of engagement’ strategy that would facilitate the democratisation of the data by enabling interfaces with other tools and systems. This would require improved documentation of survey data structures to create a more open access situation and require researchers and IT to agree a methodology, and possibly a business case to justify resources and investment.

Maybe in 20 years hence we might have cracked the problem, or will there still be demarcations drawn between research and other data flows? If so, then I think that research will occupy a niche. As I described in the latest ‘Landmark Paper’ blog, Bowles & Blyth in their 1985 MRS Conference Paper had envisaged a future where data integration had become the norm. However, in 2014 it still seems to be somewhere in the future.

I have also written a follow-on blog post looking at the Data Documentation Initiative, which have recently issued a call for action to develop standards for the reuse of metadata in surveys and questionnaires, something of which Page believed was lacking. Click here to read it