Prototypes, Platforms and Protocols: Identifying Common Issues with Remote, Unmoderated Studies and their Impact on Research Participants

Remote, unmoderated research platforms have increased the efficiency of traditional design research approaches such as usability testing, while also allowing practitioners to collect more diverse user perspectives than afforded by lab-based methods. The self-service nature of these platforms has also increased the number of studies created by requestors without formal research training. Past research has explored the quality and validity of research findings on these platforms, but little is known about the everyday issues participants face while completing these studies. We conducted an interview-based study with 22 experienced research participants to understand what issues are most commonly encountered and how participants mitigate issues as they arise. We found that a majority of the issues surface across research platforms, requestor protocols and prototypes, and participant responses range from filing support tickets to simply quitting studies. We discuss the consequences of these issues and provide recommendations for researchers and platforms.