Quota Controls in Survey Research: A Test of Accuracy And Intersource Reliability in Online Samples

In 2010, the Advertising Research Foundation (ARF) began a ground-breaking study-Foundations of Quality 2 (FoQ 2)-designed around several initiatives that investigated new questions about the quality of online samples. In addition, this research has examined how new sampling methodologies and technologies have evolved. The latest FoQ 2 initiative in 2013 explored the effectiveness of quota controls in improving the quality of online survey data. The researchers investigated the use of traditional demographic quotas to balance samples to resemble probability (randomly selected) samples. The researchers also explored new model-based ways of selecting samples that used additional variables other than demographics. But the variability of results of one of the model categories (D) sparked differences among FoQ 2 committee members about how to interpret this variability, leading some to discourage including these results in this report. In the end, the FoQ 2 committee agreed that publishing the work in its entirety was in the best interest of generating expanded research on this topic. Moreover, additional data will be needed to pursue a better understanding of which models will improve the accuracy of results using nonprobability samples, generally, and for specific topic areas.