Integration: turning data into knowledge
31 March 2014

This is the big challenge for so many organisations, in the public and private sectors, as the stream some of us lived with in the 1980s became a river as we moved into the 21^st century, and is now a raging torrent with the advent of social media and mobile technologies. It’s been called ‘infobesity’ for good reason.

Wrestling with just one of the tributaries provides data analysts (or are they now data scientists?) with enough headaches, but integrating these tributaries into one flow of data, and making sense of it all, represents the supreme challenge. 

A recent IBM international survey of 4,000 senior executives (‘The Customer Activated Enterprise’, IBM 2013) identified that whilst 82% saw ‘integrate information across the enterprise’ as a very important need within their organisation, only 24% felt this was being effectively achieved – a massive gap of 58%!

So, the new report from the Institute of Practitioners in Advertising in the UK, ‘Data Integration Explained’ (see www.ipa.co.uk for details), is very timely. It is written by Steve Wilcox, and Edited by Ken Baker – who both have a wealth of experience in this field – and draws on the integrations that underpin the IPA TouchPoints Channel Planner dataset. 

What they describe is of course much more complex than simply developing matchkeys to combine data about a customer from various sources to create a ‘single customer view’ as is common in direct marketing. 

Coming from the IPA, an emphasis on media planning is to be expected, but a chapter covering wider examples, under the heading of big data, widens the value of the report.

Overall, the report seeks to provide readers with a practical guide to the key methodologies, and how to avoid some of the pitfalls. At its heart is an overview of the eight main data integration techniques, including clearly written summaries of the main pros and cons for each of method. 

The methods start with the most simplistic, cross analysing a single demographic group in each independent survey, and finishes with hub surveys for multi-media fusions (as in TouchPoints). This is followed by a more in-depth description of the main modelling techniques and fusion techniques. Writing a brief and selecting an appropriate integrator are covered, with comprehensive checklists for initiating an integration project, plus a guide to evaluating the outputs. 

Thorough planning and post evaluation are vital to success. A summary is included of how the TouchPoints Channel Planner is created, using 10 sources, including the IPA TouchPoints Hub Survey at its core. 

Finally, the report contains a very useful bibliography, which underlines the role played over the years by IJMR papers on this field of research, including ‘Data fusion: an appraisal and experimental evaluation’, Ken Baker, Paul Harris & John O’Brien (JMRS Vol. 30, No.1, 1997) based on the fusion of two TGI datasets funded by the Market Research Development Fund and conducted in 1989. 

Re-reading this paper, and comparing it with the TouchPoints Channel Planner, demonstrates how far integration methods have evolved over the intervening years.

Despite this being a very practically orientated guide, as Baker states, ‘Data integration is a complex subject’, and I’m tempted to warn ‘don’t try this at home’! 

The report is packed with good advice, and is extremely helpful in providing a layman’s guide to the topic, but then hand it over to your favourite data scientist (who, of course, will be a Fox – see earlier blog).