Extraverted introverts, cautious risk-takers, and selfless narcissists: A demonstration of why you can’t trust data collected on MTurk

Over the last several years, a number of studies have used advanced statistical and methodological techniques to demonstrate that there is an issue with the quality of data on Amazon’s Mechanical Turk (MTurk). The current preregistered study aims to provide an accessible demonstration of this issue using a face-valid indicator of data quality: Do items that assess clearly contradictory content show positive correlations on the platform? We administered 27 semantic antonyms—pairs of items that assess incompatible beliefs or behaviours (e.g., “I am an extrovert” and “I am an introvert”)—to a sample of MTurk participants (N = 400). Over 96% of the semantic antonyms were positively correlated in the sample. For example, “I talk a lot” was positively correlated with “I rarely talk”; “I am narcissistic” was positively correlated with “I am a selfless person”; and “I like order” was positively correlated with “I crave chaos.” Moreover, 67% of the correlations remained positive even after we excluded nearly half of the sample for failing common attention check measures. These findings provide clear evidence that data collected on MTurk cannot be trusted, at least without a considerable amount of screening.