Running Online Behavioral Experiments Using R: Implementation of a Response-Time Decision Making Task as an R-Shiny App

Online experiments make it possible to reach a large number of participants in a short time at a relatively low cost. However, uncontrolled conditions in online experiments can be problematic, particularly when it is paramount to measure participant’s response-time (RT) with millisecond-level precision. To evaluate the reliability of RT data collected online, we implemented a rapid two-digit number comparison task in a mobile-friendly open-source application using R-Shiny, a popular R package for web app development. In the task, subjects were briefly presented with a two-digit target number and had to decide if it was larger or smaller than a standard fixed number (65). A total of N=169 participants (109 with a mobile device, 60 on a desktop computer) completed 116 trials over a ~7-minute session. Using generalized linear mixed models estimated with Bayesian inference methods, we observed a classical numerical distance effect: RT decreases with the logarithm of the absolute difference between the target and the standard. Our results support the use of R-Shiny for RT-data collection. Furthermore, we report systematic delays in RTs induced by different OS, web-browsers, and devices, highlighting the relevance of this information to dissect the technological component from the psychological component of RTs. Our work provides the reader an example of a seamless and robust implementation of an RT decision making task running online over desktop and mobile devices. It further paves the ground for the design of simple cognitive tasks using only R, a widely popular programming framework among cognitive scientists.