How to use Mechanical Turk for Cognitive Science Research.

How to use Mechanical Turk for Cognitive Science Research Winter A. Mason (winteram@yahoo-inc.com) Yahoo! Research 111 W. 40th St. New York, NY 10018 USA Siddharth Suri (suri@yahoo-inc.com) Yahoo! Research 111 W. 40th St. New York, NY 10018 USA 2. Workers on Mechanical Turk tend to be from a very di- verse background, spanning a wide range of age, ethnic- ity, socio-economic status (SES), language, and country of origin. Unfortunately, the population of workers on AMT is not representative of any one country or region, but it does open the doors to cross-cultural and international re- search (Eriksson & Simpson, 2010) at a very low cost and can broaden the validity of studies beyond the undergradu- ate population. Keywords: methods; mechanical turk; crowdsourcing Tutorial Overview and Objectives In this half-day tutorial we will describe a new tool that has emerged in the last 5 years for conducting online behavioral research: crowdsourcing platforms. The term crowdsourcing has its origin in an article by Howe (2006), who defined it as a job outsourced to an undefined group of people in the form of an open call. One of the main benefits of these plat- forms to behavioral researchers is that they provide access to a large set of people who are willing to do tasks—including participating in research studies—for relatively low pay. The crowdsourcing site with one of the largest subject pools is Amazon’s Mechanical Turk (AMT), so it is the focus of this tutorial. Originally, Amazon built Mechanical Turk specifically for “human computation” tasks. The idea behind its design was to build a platform for humans to do tasks that are very diffi- cult or impossible for computers, such as extracting data from images, audio transcription and filtering adult content. In its essence, however, what Amazon created was a labor mar- ket for micro-tasks (Huang, Zhang, Parkes, Gajos, & Chen, 2010). Today Amazon claims hundreds of thousands of work- ers, and roughly ten thousand employers, with AMT serv- ing as the meeting place and market (Pontin, 2007; Ipeirotis, 2010). For this reason, it also serves as an ideal platform for recruiting and compensating participants in online exper- iments. In this tutorial, we will begin by discussing some of the ad- vantages of doing experiments on Mechanical Turk. Specif- ically, there are four main advantages to using Mechanical Turk as a platform for running online experiments: 3. Studies on Mechanical Turk can be conducted at a very low cost, which clearly compare favorably to paid labo- ratory participants. For example, (Paolacci, Chandler, & Ipeirotis, 2010) replicated classic studies from the judg- ment and decision-making literature at a cost of approxi- mately $1.71/hour, and obtained results that neatly paral- leled the same studies conducted with undergraduates in a laboratory setting. 4. All too often, research is delayed because of the time it takes to recruit participants and recover from errors in the methodology. For instance, many academic researchers ex- perience the drought / flood cycle of undergraduate subject pools, with supply of participants exceeding demand at the beginning and end of a semester, and then dropping to al- most nothing at all other times. The participant availabil- ity on Mechanical Turk is relatively stable, with fluctua- tions in supply largely due to variability in the number of jobs available in the market. Moreover, experiments can be built and put on Mechanical Turk easily and rapidly, which further reduces the time to iterate the cycle of theory devel- opment and experimental execution. We will then discuss how the behavior of workers com- pares to laboratory subjects, citing work by researchers from computer science and psychology. Then, we will walk through the mechanics of putting a task on Mechanical Turk including recruiting subjects, executing the task, and review- ing the work that was submitted. We will also provide solu- tions to common problems that a researcher might face when executing their research on this platform such as techniques for conducting synchronous experiments, methods to ensure high quality work, how to keep data private, and how to main- tain code security. 1. While researchers at large universities typically have ac- cess to large numbers of undergraduates participating in experiments in exchange for academic credit, these sub- ject pools may be much smaller or even non-existent in smaller colleges and universities, or may be unavailable to all researchers. The options for non-academic researchers are even fewer, with recruitment generally limited to ads posted online and flyers posted in public areas. Mechani- cal Turk offers a very large pool of online participants for these researchers.