IJMR Landmark Paper: 'The scope for reducing refusals in household surveys: an investigation based on transcripts of tape recorded doorstep interactions'
03 October 2017

IJMR Landmark Paper, October 2017

The scope for reducing refusals in household surveys: an investigation based on transcripts of tape recorded doorstep interactions

Patrick Sturgis and Pamela Campanelli, Survey Methods at SCPR

JMRS Vol. 40 Number 2, April 1998

Today, Patrick Sturgis is Professor of Research at the University of Southampton and Director of the ESRC National Centre for Research Methods and Pamela Campanelli runs a survey training organisation, The Survey Coach. SCPR morphed into NatCen some years after this paper was published in JMRS. However, in recent times Sturgis is also known for leading the BPC/MRS inquiry into why the polls failed to predict the Conservative party win in the 2015 UK general election. So, why pick this particular paper as the latest in our quarterly Landmark series? The reason is that the MRS, the Campaign for Social Science and Social Research Association (SRA) jointly hosted a roundtable in May to discuss research design, data collection and the challenges of random probability research and secondly SRA followed up this roundtable event with a half day conference, ‘Where now for the random probability survey?’, held on June 29th with Sturgis providing a review of research on issues facing this method. I provided a review of this second event in my IJMR Editorial in 59/5. The next IJMR Lecture, being held on November 6^th at the MRS, ‘What is the future for random probability sampling?’, will be delivered by three of the speakers from the second SRA event, providing a summary of the challenges facing RPS, and innovative developments in methodologies using new technologies.

At the May MRS, Campaign for Social Science and SRA round table, response rates were shown to have declined by between 4-16% for three national studies. The main strategy to increase response appears to be instigating more call-backs, although evidence suggests that the cost-effectiveness in terms of increasing quality peaks at around 5 call-backs.

The JMRS paper focusses on one important source of non-response in RPS surveys - refusals, and whether strategies could be developed to reduce the then current levels. The authors describe the findings from their experiment based on 353 tape recorded doorstep interactions, from two surveys conducted by SCPR (Family Resources Survey - FRS) and NOP (Political Tracking survey - PTS), using 32 field interviewers in total. The PTS study was based on an electoral roll sample with pre-named individuals and ‘paper and pencil’ data collection, conducted in January 1996. The FRS study was conducted in March 1996 based on a sample drawn from the Postcode Address File with all adult members of the household being asked to participate. Interviewers were asked to record their initial conversations with potential participants to identify the strategies used by interviewers in attempting to obtain an interview. If potential participants objected to being taped, interviewers were instructed to allow them to erase the tape. Participation in the taped interactions was 61% (PTS) and 72% (FRS). In total, 848 interactions were taped, but 495 were unusable. The method used to analyse the taped data is described in detail, with six categories of interaction identified: Interview – no persuasion necessary (FRS 41%; PTS 38%); interview – little persuasion necessary (FRS 17%; PTS 5%) interview – reluctant, but eventually persuaded (FRS 17%; PTS 9%); refusal – interviewer did all they could (FRS 14%; PTS 18%); refusal – some shortfall in interviewer tactics (FRS 3%; PTS 8%); refusal – no opportunity to persuade (FRS/PTS 2%). The authors discuss the differences shown for the two surveys, mainly considered by the authors to be due to survey design issues, and that more could be done to decrease refusal rates for the FRS than for the PTS.

The authors conclude with a comment from an interviewer: ‘You have to sell it to us first, before we can sell it to others’, a point as relevant today as in 1996. As you will see in my Editorial in 59/6 (November), recent Greenbook GRIT surveys show that researchers put participant experience at the bottom of the list of survey design factors, and that this is reflected in participants’ views of their experience. Of course, many of these experiences (likely to be market research surveys) today will have no interviewer to ‘sell’ participation, or to provide feedback from the coalface on any design limitations.

Whilst the issues discussed by Sturgis and Campanelli were not apparently raised by speakers either in May, or at the June event, it was also clear that in today’s economic climate, with a greater range of flexible work opportunities, it is much tougher to recruit and retain interviewers than in the mid 1990s. Also, 86% of interviewers in the MRS field-force survey felt that gaining participation was a major challenge, but experienced interviewers also rated RPS projects to be of more interest (i.e. more motivational), addressing the comment above reported in the 1998 paper. The economic impact on interviewer’s earnings is more of issue as increasing refusal rates lead to reduced earnings where remuneration is based on completed interviews.

The JMRS paper describes an interesting experiment into tackling one important aspect of addressing refusal rates. The recent events demonstrate the tougher climate of today. Interview based surveys are less common meaning that panels of interviewer are smaller and turnover rates are higher due to a competitive employment market, and refusal rates continue to climb. Whilst current research on RPS methods is focussed on exploring the meaning of quality, how it should be measured, and devising innovative solutions to deliver it, we still need to ensure that interviewers are motivated by the research experience, if they are to be successful in persuading citizens to participate in RPS studies.

(IJMR Landmark blog 06 2017, PM 02-10-17)