Behavioral Lab 3.0: Towards the next generation of online behavioral research

After decades of using mostly college students, behavioral researchers gradually moved online to more cost-effective and flexible audiences, first to samples of convenience and then to sophisticated markets like Amazon’s Mechanical Turk (MTurk). Now, advanced platforms (such as CloudResearch and Prolific) promise researchers higher data quality using ex-ante vetting and controls on their participants’ pool. We systematically examine the advantages of these ex-ante controls, and their effects on various measures of data quality, and compare them to standard ex-post approaches of attention checks (in randomized positions). We find that ex-ante controls outperform ex-post checks on process measures of attention, honesty, and reliability, as well as on outcome measures of representation and replicability. Additionally, while samples from platforms with ex-ante controls show key differences in representation from the general population, they are still more representative than a major university lab pool. We discuss implications for researchers desiring high data quality, reviewers and editors of research papers, and policymakers who aim to regulate efficient spending of research funds.