Accurate, Focused Research on Law, Technology and Knowledge Discovery Since 2002

Pew – Evaluating Online Nonprobability Surveys

“As the costs and nonresponse rates of traditional, probability-based surveys seem to grow each year, the advantages of online surveys are obvious – they are fast and cheap, and the technology is pervasive. There is, however, one fundamental problem: There is no comprehensive sampling frame for the internet, no way to draw a national sample for which virtually everyone has a chance of being selected. The absence of such a frame has led to lingering concerns about whether the fraction of the population covered by nonprobability approaches can be made to look representative of the entire population. For roughly 15 years, independent studies suggested that the answer to that question was generally “no” if the goal was to make accurate population estimates. Over time, though, researchers and sample vendors have developed technologies and statistical techniques aimed at improving the representativeness of online nonprobability surveys. Several recent case studies suggest a future (some would argue a present) in which researchers need not have an expensive, probability-based sample to make accurate population estimates…To better understand the current landscape of commercially available online nonprobability samples, Pew Research Center conducted a study in which an identical 56-item questionnaire was administered to nine samples supplied by eight different vendors…The study finds, as a starting point, that the methods used to create online nonprobability samples are highly variable. The vendors differ substantially in how they recruit participants, select samples and field surveys. They also differ in whether and how they weight their data. These design differences appear to manifest in the samples’ rankings on various data quality metrics. In general, samples with more elaborate sampling and weighting procedures and longer field periods produced more accurate results. That said, our data come from just nine samples, so the effects of these factors are not well isolated, making these particular conclusions preliminary at best…”

Sorry, comments are closed for this post.