Not-quite-naïve listeners: Students as an audience for gamified crowdsourcing

Collecting independent listeners’ judgments of speech accuracy/intelligibility is an essential component of research on speech disorders. Raters can be trained clinicians, students in speech-language pathology, or naive listeners (now commonly recruited online via crowdsourcing platforms such as Amazon Mechanical Turk/AMT). However, limited comparison data exist to guide researchers in determining which rater population to use. We describe a study (Hitchcock et al., in prep) in which 2,256 tokens of English /r/ at the word level, produced by five children receiving intervention for /r/ misarticulation, were rated in a binary fashion using the online platform Experigen. Raters were certified clinicians (n = 3), students in speech-language pathology (n = 9 unique listeners per token), or naive listeners recruited on AMT (n = 9 unique listeners per token). Interrater reliability was higher when comparing modal ratings between clinicians and students (Cohen’s kappa=.73, CI=.7-.77) than between clinicians and ...