Classic paper: Cathie Marsh, 1990
03 April 2013

As I’m sure many of you know, Cathie Marsh was a leading social scientist, who died aged 41 in 1993. 

Her contribution as a quantitative sociologist was so important that there is the Cathie Marsh Centre for Census and Survey Research at Manchester University, where she latterly worked, and she is annually remembered by the Royal Statistical Society and the Social Research Association through the Cathie Marsh Memorial Lecture, held at the RSS each November. 

I always try and attend this event, and report it in my Editorial. However, pressure on space means that I did not cover the open-data themed lecture in 2012, but I thought that the introduction of the quarterly ‘Classic Paper’ feature here would offer an ideal opportunity to reflect on her work.

I feel very privileged to be the Editor of a journal that includes in its archives contributions by the likes of Cathie Marsh. In its previous guise as the Journal of the Market Research Society, two papers by Marsh were published in the early 1990s, and it is one of those that I’ve selected for the second ‘Classic Paper’ entitled: 

• Testing Nine Hypotheses About Quota Sampling

However, I’ve not just selected this particular paper just to mark her immense contribution to research. There are two very important reasons why this paper is, in my view, as relevant today as in 1990. 

Firstly, it is because the market research sector does not debate sampling issues today as often as I believe we should, considering that the need for fit-for-purpose representative sampling methods remain as vitally important as ever in providing clients with robust research findings. 

Secondly, too many papers submitted to IJMR pay lip-service to statistically based sampling methods, any-data-will-do seems the mantra in some cases. This is a major reason why papers are rejected at submission, as the ‘sampling plans’ are not fit for purpose.

The paper is based on an experiment conducted by the two authors in 1985 where half the interviews in each of two localities were based on respondents selected by quota sampling, and half via random sampling. At the heart of the paper is the survey design description, covering nearly five pages in the journal – in itself worthy of being termed ‘classic’ in the detail provided. 

Survey design is often far from being at the heart of some submissions to IJMR today, receiving no more than a passing mention, covered in as little as two sentences in some papers we receive! 

Obviously, you will read the detail for yourself, but one point I will make is that the fieldwork was conducted by NOP (now part of GfK), reflecting a collaboration on research methodology between the market and social research communities. As the title of the paper clearly signposts, the reason for the experiment was to test a number of hypotheses about quota sampling. 

The findings show clear differences between the two sub-sampling methods, some of which were not apparent in the literature up to that point in time, or easily explained. 

The results also demonstrate that it can be difficult to set quotas in small geographic areas, if relevant population data is not available. I don’t think this issue even crosses the minds of some authors today when sampling. 

One hypothesis looks at bias in quota samples against workers in manufacturing, and I wonder if this would be chosen today in such an experiment, reflecting the shift in employment in the intervening quarter of a century. Their overall conclusion is that ‘random and quota samples are different, but not entirely in the way predicted by the literature’. 

The authors’ key recommendation is that researchers should ‘use surveys to investigate methodological questions alongside their substantive concerns. Such questions cannot be left to methodologists; rather, a concern with methodology needs to be a regular feature of survey design’. 

I’m pleased to report that this still surfaces, occasionally, in submissions today, but is this recommendation one regularly applied today? 

They cite the reason for this as providing lower cost, ‘real life’, opportunities to conduct tests rather than developing more costly ad hoc experiments. I particularly like ‘real life’ as a factor – many submissions today are based on samples and situations that are far from ‘real life’, and therefore, if published, would provide little value to many readers.

I commend this paper to you as an example of rigorous experimental design to explore key methodological issues that underpin market and social research. 

The authors conclude: ‘Hopefully, however we have shown that these matters of methodology are not to be visited occasionally when sampling decisions have to be made. Rather, they need to be on the routine research agenda of practising survey researchers’. 

I hope this is the case; but is it? Finally, the End notes contain an interesting insight into research costs and costings at the time.