IJMR Landmark paper - ‘Non-probability sampling for WWW surveys: a comparison of methods’
21 April 2015

A paper by Niels Schillewaert, Fred Langerak and Tim Duhamel, Ghent University, JMRS Vol. 40 No 4, October 1

In the next issue of IJMR, to be published in May, we will be publishing the full coverage of the IJMR hosted debate on online sampling held at the MRS 2015 annual conference last month. That issue will also include a paper covering an experiment comparing access panel with face to face research, conducted in Spain.

I therefore thought it might be interesting to comb through back issues of JMRS (as it was titled pre-2000) to see how we covered the early days of what has become the internet revolution in research. ESOMAR started running internet research related conferences and seminars in, I believe, January 1998, held in Paris. IJMR published the first papers on internet research in 1995, both comparing response rates in email versus mail self completion surveys (Mehta & Sivadas, Vol.37 No. 2 and Tse in Vol.37 No. 4). And in 1999, there was a complete issue devoted to Research on the Internet (Vol. 41 No. 4, October), which also published ESOMAR’s first guideline on conducting internet research.

However, the paper that was the first in our journal to explore sampling issues was the one I’m featuring as this quarter’s Landmark paper, by Schillewaert (now at InSites) et al. In particular, the authors introduced what has become a recognised feature of internet research, the use of non- probability sampling methods. This was of course before the development of online access panels, so their comparison was between samples collected from a press release, newsgroup postings, hyperlinks from other websites and emailing.

The authors identify internet users at the time of writing their paper as over 40m. worldwide, doubling every year. Interestingly, they identify a relative lack of interest at that point in the internet as a data collection method by academics, whilst practitioners such as Nielsen are putting their toes in the water by 1995, the year that JMRS first published studies testing such methods.

In the paper, the authors discuss the challenge of external validation, in an era where internet populations were likely to be very unrepresentative of national populations. The authors developed and tested three hypotheses covering completeness; response distributions across methods are similar; and, that the distribution of attitudinal variables is similar. The methodology is described in detail, plus the findings.

In discussing their findings, the authors point out that: ‘random sampling methods may prove inconvenient and financially unrealisable due to the low Internet penetration and the potential self-selection bias’. We still worry about the first factor – by country and particular population groups, but the second has become the basis for a high proportion of online research today. They also pose three issues that need further research:

• Studies should judge the effectiveness of various non-probability methods for internet surveys, and identify the practical and theoretical strengths and weaknesses;
• Compare probability and non-probability methods for the internet and other methods;
• Identify where non-probability methods may provide an economic alternative, and the most appropriate applications for such methods.

As they conclude, ‘future studies should be aimed at developing a comprehensive framework for describing when to use and when not to use the various sampling methods for WWW surveys’. The AAPOR Task Force report on non-probability sampling provides the closest we’ve come to developing such a framework, but the IJMR debate, and other recent IJMR papers, Conference Notes and reports in my Editorial shows that we still have a lot to learn.

Finally, what the authors couldn’t foresee at the time was how self-selected access panels would rapidly become such a dominant force in online quantitative research.