IJMR Landmark Paper: 'Managerial consequences of inadequately validating cluster analysis solutions'
15 April 2019

'Managerial consequences of inadequately validating cluster analysis solutions', Paul Bottomley and Agnes Nairn, IJMR Vol. 46 Q2, 2004

I’ve chosen this as the latest Landmark Paper, one that is rather newer than others selected in the past, for two main reasons.

Firstly, segmentations lie at the core of many submissions to IJMR, but simply because a cluster analysis has produced a number of discrete groups of consumers, that does not mean it provides a valid interpretation of the market. Also, should a segmentation be based solely on customers of the organisation, or on the market that the organisation inhabits to be of most value in developing future strategy? A market segmentation is generally of much more value as it provides a much more holistic picture of all relevant consumers, and a context for an organisation’s customers.  Secondly, segmentations need to facilitate action, but often the clusters are somewhat meaningless if it is not possible to use the output in a practical way, for example, to develop a marketing strategy that can communicate differentiated messages to the target groups. In addition, I’m very pleased to say that we’ve recently welcomed Agnes Nain (now Professor of Marketing, University of Bristol) back to the IJMR Executive Editorial Board. 

As the authors discuss, segmentations need to be validated if they are to reassure users that the results can be trusted. This was an issue that academics were beginning to explore at that time to address rising concerns about the stability, or even the reality, of segmentations. This is especially concerning as the techniques used require a high level of subjective judgement, the ‘p value’ being missing from most projects. The paper was also written at a time when the range and volume of data that could be incorporated into a segmentation project was increasing and multivariate methods were being incorporated into the analytical suites within the new customer relational management (CRM) systems, leading to outputs derived via ‘black-box’ technologies, with users being unaware of possible limitations. However, with segmentation becoming an increasingly important method to describe markets, the main contribution from academics had been to help users address the second point I described – ensuring that the resulting outputs could be applied within marketing strategy. However, the authors contend that validity in terms of whether the proffered solution was based on firm statistical foundations was being overlooked, a meta-analysis of cluster analysis studies showing over half of the studies included no assessment of reliability; nearly two thirds failed to assess validity, and a third included neither of these tests. Of much more concern was the finding in a paper published by the authors the year before in IJMR 45/2 (see reference in the Landmark Paper) that ‘cluster solutions derived from random data can appear to be as reliable and valid as those derived from real data if not comprehensively tested’ – this latter point being a focus within the later paper.

The basis of the paper is therefore to identify how managers perceive the usefulness of cluster solutions, and to explore whether users have either the knowledge, or expert support, that would enable them to distinguish between segments based on real or random data. Their aim is to ‘demonstrate that the consequences of inadequate validation are indeed real and serious.’

The authors used the two datasets developed for the earlier paper, producing three outputs from the real and random sets: profile of each cluster centroid based on responses of 14 statements about direct marketing; demographic profile of participants; responses to statements measuring attitudes, opinions and beliefs about shopping. Their three hypotheses covered perception of usefulness of both solutions; that the segmentation based on real data is perceived of being of more value; the segmentation based on random data is perceived of greater value – these hypotheses also exploring the impact of limited versus enhanced information about the validity of the outputs. Participant’s (142 managers on a marketing programme at a leading UK business school) level of expertise was also measured.

As might be expected, the experiment clearly showed that many users are not equipped to differentiate good from bad and would be happy to build a marketing strategy around a segmentation built from random data that was devoid of any meaningful structure. Novices and experts mainly differed on the ‘magnitude’, rather than the ‘nature’ of the interactions shown in the type and amount of information they were given. Information overload was also shown to create challenges in identifying good from bad, but sufficient nonsensical information could lead to doubt. The authors recommend that managers should be cautious and adopt a healthy scepticism when interpreting the outputs from a segmentation, especially as what they are likely to be confronted with in real life will be much more nuanced than in their experiment. As they conclude: ‘regardless of knowledge, all of us can be susceptible to the allure of random data’.

Since 2004, the opportunities to create segmentations based on an increasingly complex range of data sources have mushroomed. As I’ve said before in my Editorials, the trend towards data science and analytics creates even greater challenges to researchers and clients when confronted with segmentations and assessing their validity before committing to incorporating the outputs into strategy. Training has limitations, as I described in my last Editor’s blog. What is needed is advice and guidance from experts who can communicate the issues and limitations in plain language to ensure that the allure described by the authors is resisted.

(IJMR Editor blog 03 2019 PM 08-4-19)   

Access the paper for free

This paper is freely available to anyone via Sage Journals.