IJMR Landmark paper - ‘Validity and reliability in qualitative market research: a review of the literature’
11 November 2015

Wendy Sykes, JMRS Vol. 32 No. 3 July 1990 

When you receive your copy of the next issue of IJMR, 57/6, sometime in late November, you will see that it contains three interestingly different perspectives on qualitative research – two papers and the Viewpoint, plus my own thoughts in Editorial based on these contributions.

The papers and Viewpoint in 57/6 focus on the state of qualitative today, and the challenges that the authors feel need to be addressed, if qualitative is to play a full and innovatory role in research today. So, in selecting the latest ‘Landmark Paper’ I thought it might be both useful and interesting to find a perspective that provided an in-depth assessment of qualitative methods from an earlier era, and Sykes’ paper perfectly fits this requirement.

As you will see, the paper was commissioned by the Market Research Development Fund (MRDF), that for many years, in association with the MRS Technical and Development Committee, provided a research and development input on methodological issues to the UK market research sector. The paper was linked to a one-day seminar on qualitative research methods held in 1988.

As Barnham and Stening et al in their current papers discuss, validity and reliability remain core challenges today. Jumping straight to Sykes’ final conclusions, Barnham would I think agree with her that: ‘it is clear that there is scope for much more research in this area, particularly theoretically supported empirical work’. Barnham in particular discusses the issue of interpretation and the need for an accepted theoretical basis for this, to re-assure clients that this vital stage in the research process adheres to some commonly accepted scientifically devised principles, rather than simply a subjective view based on the researcher’s experience and perspective. Whilst Barnham argues that this is the case for the analysis of quantitative data, Sykes argues that quantitative researchers also face reliability challenges.

As the title of the paper implies, Sykes discusses the themes of validity and reliability in some detail, and she quotes a paper from 1986 that describes a procedure to help provide re-assurance, ‘transparency’ being a key attribute. A further suggested approach she describes is replicability. However, quoting Sue Robson, Sykes also points out the difficulties of disentangling the complexity of interaction ‘between problem analysis, data generation and interpretation’, and, ‘researcher effects’ from ‘method effects’.

In Sykes view, whether it is quantitative or qualitative, it is the meaning that researchers extract from the data that creates the concerns about reliability and validity, drawing on Martin Collins’ view that: ‘the researcher examines the data, attaches meaning to them, and draws inferences and conclusions, all against the background of the pervasive prior models and expectations’ – a perspective that Barnham argues can be overcome.

Wynberg in her Viewpoint in 57/6 argues the case for ‘authorship’, to bring greater ownership, accountability, creativity and a sense of pride to qualitative research. She argues that ‘authorship’ is personal – the value added by the experience and bravery of the researcher in venturing beyond the initial, often narrow, focus of the client’s brief. Sykes makes a similar point, quoting from a 1986 paper that in effect argues that the researcher must be open and honest in how they have undertaken all steps in the research process – taking transparent ownership of what they’ve done, and how they’ve arrived at their conclusions (or using today’s terminology, how their insights were derived).

In the paper Sykes discusses at length qualitative research best practice processes. At a time when, as she says, there was a ‘dearth of widely agreed rules for conducting qualitative research’, Sykes develops from a synthesis of available evidence a summary of the factors that contribute towards the notion of a theoretical basis for the process.

Literature reviews serve three key purposes. Firstly, they provide researchers, whether academics or practitioners, with a useful summary of where to find the detailed evidence from previous research on a topic. Secondly, they provide readers with a ‘state of the nation’ perspective on the field. Thirdly, they identify gaps in current knowledge which should be addressed to enhance the knowledge in this field. I feel that Sykes’ review ticks all these boxes, but the evidence within the contributions published in 57/6 suggests that key gaps remain that still need to be addressed, especially if validity and reliability continue to questioned by clients.

Qualitative research may be moving in a new direction with social media providing qualitative data on a quantitative scale, where tools such as discourse analysis and semiotics can play important roles in making sense of what’s there. As long ago as 2007 we published a paper based on one presented at the previous year’s MRS conference, ‘The web of insights: the art and practice of webnography’ by Anjali Puri (IJMR Vol. 49 Issue 3, 2007), who also presented her views at an IJMR Research Methods Forum a few years later, describing the then new world of data emerging from these channels. Does such a shift in direction provide opportunities to demonstrate that qualitative methods deliver validity and reliability, or are their still credibility gaps that remain to be addressed?

Finally, In my Editorial I refer to an interesting, and highly relevant, article by Joris Luyendijk (‘Don’t let the Nobel fool you: Economics is not a science’, The Guardian, 12th October, 2015) who questions why only economics is singled out from the social sciences to receive a Nobel prize, implying that economics has scientific based certainties that the other branches lack. The author argues that there is plenty of recent evidence to dispel any notions that this is the case, including anthropological, fieldwork based qualitative research, ‘which economists like to discard as “soft” and unscientific’, conducted by Karen Ho, and published in her book ‘Liquidity’, that identified boardroom behaviour within the financial sector which was fostering a culture that was not in the best long term interests of the organisations, leading to their collapse and the recent world-wide economic crisis. Whilst the author discusses the limitations of qualitative research discussed by Sykes, he claims that unlike economists, social science researchers are transparent about the limitations of their methods. And, he argues, even if imperfect, these observations play a vitally important role in understanding human behaviour. Researchers have humility; economists do not. It was qualitative research that helped expose the emperor’s new clothes, but the warnings were ignored, or dismissed.

There is still work to do to in cementing the credibility of qualitative research.

Read the landmark paper here