Auditory Icons, Earcons, Spearcons, and Speech: A Systematic Review and Meta-Analysis of Brief Audio Alerts in Human-Machine Interfaces

ABSTRACTAuditory displays commonly are used in safety-critical domains and are a vital component of universal and inclusive design practices. Despite several decades of research on brief auditory alerts for representing status and processes in user interfaces, there is no clear heuristic guidance for which type(s) of auditory alerts should be preferred for designing interfaces. We used evidence synthesis (systematic review and meta-analysis) to examine the effectiveness of different types of brief audio alerts. We identified articles comparing auditory icons (real-world sounds with an ecological relationship to their referent), earcons (abstract sounds with no ecological relationship to their referent), spearcons (accelerated/compressed speech), and speech alerts. We used meta-analysis to compare alerts across five different outcomes: accuracy, reaction time, subjective ratings, workload, and dual-task interference. For accuracy and reaction time, results indicated speech, spearcons, and other types of alerts (usually hybrid, e.g., spearcons plus speech) were superior to auditory icons, which in turn were superior to earcons. Earcons also were inferior to all other options with respect to subjective ratings. Analyses generally suggested parity among alert types for workload and dual-task interference. Based on currently available evidence, it appears that speech, spearcons, and hybrid (e.g., spearcons plus speech) auditory alerts result in better performance than auditory icons and especially earcons. Still, high heterogeneity in our analyses cannot rule out a wide range of possible effects, and our analyses could not directly address some of the concerns that have been raised regarding speech-based alerts. These findings can help to guide the selection of brief audio alerts in interface design.KEYWORDS: Auditory displayssonificationalarmsuniversal designassistive technologynetwork meta-analysis Disclosure statementNo potential conflict of interest was reported by the authors.Supplementary materialSupplemental data for this article can be accessed online at https://doi.org/10.1080/25742442.2023.2219201.Data availability statementData, preprint, and additional materials are available at https://osf.io/cvk5h/References Used in Meta-analysesLegenda Used in accuracy analysisb Used in reaction time analysisc Used in subjective ratings analysisd Used in workload analysise Used in dual-task analysisa,bAcker-Mills, B. E. (2006). The effect of auditory alerts on the performance of concurrent tasks. https://smartech.gatech.edu/handle/1853/50650aAlseid, M., & Rigas, D. (2011). The Role of Earcons and Auditory Icons in the Usability of Avatar-Based E-Learning Interfaces. Proceedings of the 2011 Developments in E-Systems Engineering, 276–281. https://doi.org/10.1109/DeSE.2011.34aAmer, T. S., & Johnson, T. L. (2018). Earcons Versus Auditory Icons in Communicating Computing Events: Learning and User Preference. Int. J. Technol. Hum. Interact., 14(4), 95–109. https://doi.org/10.4018/IJTHI.2018100106cAnkolekar, A., Sandholm, T., & Yu, L. (2013). Play It by Ear: A Case for Serendipitous Discovery of Places with Musicons. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2959–2968. https://doi.org/10.1145/2470654.2481411c,dBeattie, D., Baillie, L., & Halvey, M. (2015). A Comparison of Artificial Driving Sounds for Automated Vehicles. PROCEEDINGS OF THE 2015 ACM INTERNATIONAL JOINT CONFERENCE ON PERVASIVE AND UBIQUITOUS COMPUTING (UBICOMP 2015), 451–462. https://doi.org/10.1145/2750858.2807519a,b,cBonebright, T. L., & Nees, M. A. (2007). Memory for Auditory Icons and Earcons with Localization Cues. https://smartech.gatech.edu/handle/1853/50014a,b,cBonebright, T. L., & Nees, M. A. (2009). Most earcons do not interfere with spoken passage comprehension. Applied Cognitive Psychology, 23(3), 431–445. https://doi.org/https://doi.org/10.1002/acp.1457aBrewster, S. A., Wright, P. C., & Edwards, A. D. N. (1993). An Evaluation of Earcons for Use in Auditory Human-Computer Interfaces. Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, 222–227. https://doi.org/10.1145/169059.169179a,cDuarte, C., & Carrico, L. (2007). Conveying browsing context through audio on digital talking books. In Stephanidis, C (Ed.), UNIVERSAL ACCESS IN HUMAN-COMPUTER INTERACTION: APPLICATIONS AND SERVICES, PT 3, PROCEEDINGS (p. 259+). ICS FORTH, Human Comp Interact Lab.aEdworthy, J., Page, R., Hibbard, A., Kyle, S., Ratnage, P., & Claydon, S. (2014). Learning three sets of alarms for the same medical functions: A perspective on the difficulty of learning alarms specified in an international standard. Applied Ergonomics, 45(5), 1291–1296. https://doi.org/https://doi.org/10.1016/j.apergo.2013.10.003cFagerlon, J. (2007). Expressive Musical Warning Signs. https://smartech.gatech.edu/handle/1853/49996a,dFinlayson, J. L., & Mellish, C. (2005). The “audioview” - providing a glance at Java source code. https://smartech.gatech.edu/handle/1853/50187a,b,c,dGable, T. M., Tomlinson, B., Cantrell, S., & Walker, B. N. (2017). Spindex and Spearcons in Mandarin: Auditory Menu Enhancements Successful in a Tonal Language. https://doi.org/10.21785/icad2017.025b,eGable, T. M., Walker, B. N., Moses, H. R., & Chitloor, R. D. (2013). Advanced Auditory Cues on Mobile Phones Help Keep Drivers’ Eyes on the Road. Proceedings of the 5th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, 66–73. https://doi.org/10.1145/2516540.2516541aGamper, H., Dicke, C., Billinghurst, M., & Puolamaki, K. (2013). Sound Sample Detection and Numerosity Estimation Using Auditory Display. ACM TRANSACTIONS ON APPLIED PERCEPTION, 10(1). https://doi.org/10.1145/2422105.2422109a,b,cGarzonis, S., Bevan, C., & O’Neill, E. (2008). Mobile Service Audio Notifications: Intuitive Semantics and Noises. Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat, 156–163. https://doi.org/10.1145/1517744.1517793aGarzonis, S., Jones, S., Jay, T., & O’Neill, E. (2009). Auditory Icon and Earcon Mobile Service Notifications: Intuitiveness, Learnability, Memorability and Preference. In Greenberg, S and Hudson, SE and Hinkley, K and RingelMorris, M and Olsen, DR (Ed.), CHI2009: PROCEEDINGS OF THE 27TH ANNUAL CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, VOLS 1-4 (pp. 1513–1522). ACM SIGCHI; Autodesk; Google; Microsoft; NSF; eLearn Magazine; interations; Yahoo.bGlatz, C., Krupenia, S. S., Bülthoff, H. H., & Chuang, L. L. (2018). Use the Right Sound for the Right Job: Verbal Commands and Auditory Icons for a Task-Management System Favor Different Information Processes in the Brain. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, 1–13. https://doi.org/10.1145/3173574.3174046a,b,cGraham, R. (1999). Use of auditory icons as emergency warnings: evaluation within a vehicle collision avoidance application. Ergonomics, 42(9), 1233–1248. https://doi.org/10.1080/001401399185108bHaas, E. C. (1998). Can 3-D Auditory Warnings Enhance Helicopter Cockpit Safety? Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 42, 1117–1121. https://doi.org/10.1177/154193129804201513a,c,eHofmann, H., Hermanutz, M., Tobisch, V., Ehrlich, U., Berton, A., & Minker, W. (2016). Evaluation of In-Car SDS Notification Concepts for Incoming Proactive Events. In A. Rudnicky, A. Raux, I. Lane, & T. Misu (Eds.), Situated Dialog in Speech-Based Human-Computer Interaction (pp. 111–124). Springer International Publishing. https://doi.org/10.1007/978-3-319-21834-2_11cHutchinson, J., & Metatla, O. (2018). An Initial Investigation into Non-Visual Code Structure Overview Through Speech, Non-Speech and Spearcons. Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems, 1–6. https://doi.org/10.1145/3170427.3188696a,b,cIsherwood, S. J., & McKeown, D. (2017). Semantic congruency of auditory warnings. Ergonomics, 60(7), 1014–1023. https://doi.org/10.1080/00140139.2016.1237677b,cJeon, M. (2019). Multimodal Displays for Take-over in Level 3 Automated Vehicles While Playing a Game. Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, 1–6. https://doi.org/10.1145/3290607.3313056b,d,eJeon, M., Gable, T. M., Davison, B. K., Nees, M. A., Wilson, J., & Walker, B. N. (2015). Menu Navigation With In-Vehicle Technologies: Auditory Menu Cues Improve Dual Task Performance, Preference, and Workload. International Journal of Human–Computer Interaction, 31(1), 1–16. https://doi.org/10.1080/10447318.2014.925774a,bJeon, M., & Sun, Y. (2014). Design and Evaluation of Lyricons (Lyrics + Earcons) for Semantic and Aesthetic Improvements of Auditory Cues. https://smartech.gatech.edu/handle/1853/52086b,cJeon, M., & Walker, B. N. (2011). Spindex (Speech Index) Improves Auditory Menu Acceptance and Navigation Performance. ACM Trans. Access. Comput., 3(3). https://doi.org/10.1145/1952383.1952385a,b,c,dJeon, M., Walker, B. N., & Srivastava, A. (2012). “Spindex” (Speech Index) Enhances Menus on Touch Screen Devices with Tapping, Wheeling, and Flicking. ACM Trans. Comput.-Hum. Interact., 19(2). https://doi.org/10.1145/2240156.2240162a,cKenny, C. W. L., & Wei, C. L. (2009). Association Testing-A Methodology for Selecting and Evaluating Audio Alerts. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 53(18), 1309–1313. https://doi.org/10.1177/154193120905301831cLandry, S., Jeon, M., Lautala, P., & Nelson, D. (2019). Design and assessment of in-vehicle auditory alerts for highway-rail grade crossings. Transportation Research Part F: Traffic Psychology and Behaviour, 62, 228–245. https://doi.org/https://doi.org/10.1016/j.trf.2018.12.024bLemmens, P. M. C., Bussemakers, M. P., & de Haan, A. (2001). Effects of auditory icons and earcons on visual categorization: The bigger picture. https://smartech.gatech.edu/handle/1853/50617bLemmens, P. M. C., de Haan, A., & van Galen, G. P. (2003). Do location and context operate independently? https://smartech.gatech.edu/handle/1853/50458a,bLucas, P. A. (1994). An evaluation of the communicative ability of auditory icons and earcons. https://smartech.gatech.edu/handle/1853/50825a,b,cLudi, S., Simpson, J., & Merchant, W. (2016). Exploration of the Use of Auditory Cues in Code Comprehension and Navigation for Individuals with Visual Impairments in a Visual Programming Environment. Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, 279–280. https://doi.org/10.1145/2982142.2982206a,bMcCarthy, J. W., DiGiovanni, J. J., Ries, D. T., Boster, J. B., & Riffle, T. L. (2020). Exploration of Head Related Transfer Function and Environmental Sounds as a Means to Improve Auditory Scanning for Children Requiring Augmentative and Alternative Communication. Assistive Technology, 32(6), 325–334. https://doi.org/10.1080/10400435.2018.1559897a,bMcGee-Lennon, M., Wolters, M. K., McLachlan, R., Brewster, S., & Hall, C. (2011). Name That Tune: Musicons as Reminders in the Home. 29TH ANNUAL CHI CONFERENCE ON HUMAN FACTORS IN COMPUTING SYSTEMS, 2803–2806.b,cMcKeown, D. (2005). Candidates for within-vehicle auditory displays. https://smartech.gatech.edu/handle/1853/50104a,bMcKeown, D., Isherwood, S., & Conway, G. (2010). Auditory Displays as Occasion Setters. Human Factors, 52(1), 54–62. https://doi.org/10.1177/0018720810366861b,c,d,eMoskovitch, Y., Jeon, M., & Walker, B. N. (2010). Enhanced Auditory Menu Cues on a Mobile Phone Improve Time-Shared Performance of a Driving-Like Dual Task. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 54(18), 1321–1325. https://doi.org/10.1177/154193121005401804aMurphy, E., Bates, E., & Fitzpatrick, D. (2010). Designing Auditory Cues to Enhance Spoken Mathematics for Visually Impaired Users. Proceedings of the 12th International ACM SIGACCESS Conference on Computers and Accessibility, 75–82. https://doi.org/10.1145/1878803.1878819cNadri, C., Lee, S. C., Kekal, S., Li, Y., Li, X., Lautala, P., Nelson, D., & Jeon, M. (2021). Effects of Auditory Display Types and Acoustic Variables on Subjective Driver Assessment in a Rail Crossing Context. Transportation Research Record, 0(0), 03611981211007838. https://doi.org/10.1177/03611981211007838bNees, M. A., & Best, K. (2013). Modality And Encoding Strategy Effects On A Verification Task With Accelerated Speech, Visual Text, And Tones. https://smartech.gatech.edu/handle/1853/51678a,bNees, M. A., & Best, K. (2014). A Verification Task with Lateralized Tones and Accelerated Speech. https://smartech.gatech.edu/handle/1853/52102a,c,dNees, M. A., Helbein, B., & Porter, A. (2016). Speech Auditory Alerts Promote Memory for Alerted Events in a Video-Simulated Self-Driving Car Ride. Human Factors, 58(3), 416–426. https://doi.org/10.1177/0018720816629279bNeurauter, M. L. (2005). Multimodal Warnings: Curve-Warning Design. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 49(22), 1945–1949. https://doi.org/10.1177/154193120504902213bPalladino, D. K. (2008). Efficiency of Spearcon-Enhanced Navigation of One Dimensional Electronic Menus. https://smartech.gatech.edu/handle/1853/26302a,bPerry, N. C., Stevens, C. J., Wiggins, M. W., & Howell, C. E. (2007). Cough Once for Danger: Icons Versus Abstract Warnings as Informative Alerts in Civil Aviation. Human Factors, 49(6), 1061–1071. https://doi.org/10.1518/001872007X249929a,bRamos, D., & Folmer, E. (2011). Supplemental Sonification of a Bingo Game. Proceedings of the 6th International Conference on Foundations of Digital Games, 168–173. https://doi.org/10.1145/2159365.2159388bRichie, E., Offer-Westort, T., Shankar, R., & Jeon, M. (2018). Auditory Displays for Take-Over in Semi-automated Vehicles. In V. G. Duffy (Ed.), Digital Human Modeling. Applications in Health, Safety, Ergonomics, and Risk Management (pp. 623–634). Springer International Publishing. https://doi.org/10.1007/978-3-319-91397-1_51cRigas, D., & Ciuffreda, A. (2007). An Empirical Investigation of Multimodal Interfaces for Browsing Internet Search Results. Proceedings of the 7th Conference on 7th WSEAS International Conference on Applied Informatics and Communications - Volume 7, 194–199.a,b,cRobb, J., Garner, T., Collins, K., & Nacke, L. E. (2017). The Impact of Health-Related User Interface Sounds on Player Experience. Simulation & Gaming, 48(3), 402–427. https://doi.org/10.1177/1046878116688236a,b,dŠabić, E., Chen, J., & MacDonald, J. A. (2021). Toward a Better Understanding of In-Vehicle Auditory Warnings and Background Noise. Human Factors, 63(2), 312–335. https://doi.org/10.1177/0018720819879311a,bSabic, E., Mishler, S., Chen, J., & Hu, B. (2017). Recognition of Car Warnings: An Analysis of Various Alert Types. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, 2010–2016. https://doi.org/10.1145/3027063.3053149cSikora, C. A., & Roberts, L. A. (1997). Defining a Family of Feedback Signals for Multimedia Communication Devices. In S. Howard, J. Hammond, & G. Lindgaard (Eds.), Human-Computer Interaction INTERACT ’97: IFIP TC13 International Conference on Human-Computer Interaction, 14th–18th July 1997, Sydney, Australia (pp. 373–380). Springer US. https://doi.org/0cSkantze, D., & Dahlback, N. (2003). Auditory icon support for navigation in speech-only interfaces for room-based design metaphors. https://smartech.gatech.edu/handle/1853/50443b,cSuh, H., Jeon, M., & Walker, B. N. (2012). Spearcons Improve Navigation Performance and Perceived Speediness in Korean Auditory Menus. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 56(1), 1361–1365. https://doi.org/10.1177/1071181312561390a,bSun, Y., & Jeon, M. (2015). Lyricon (Lyrics + Earcons) Improves Identification of Auditory Cues. In A. Marcus (Ed.), Design, User Experience, and Usability: Users and Interactions (pp. 382–389). Springer International Publishing. https://doi.org/10.1007/978-3-319-20898-5_37a,b,c,dThapa, R. B., Ferati, M., & Giannoumis, G. A. (2017). Using Non-Speech Sounds to Increase Web Image Accessibility for Screen-Reader Users. Proceedings of the 35th ACM International Conference on the Design of Communication. https://doi.org/10.1145/3121113.3121231aTislar, K., Duford, Z., Nelson, B., Peabody, M., & Jeon, M. (2018). Examining the learnability of auditory displays: Music, earcons, spearcons, and lyricons. https://doi.org/10.21785/icad2018.029cTran, T. V., Letowski, T., & Abouchacra, K. S. (2000). Evaluation of acoustic beacon characteristics for navigation tasks. Ergonomics, 43(6), 807–827. https://doi.org/10.1080/001401300404760b,dVargas, M. L. M., & Anderson, S. (2003). Combining speech and earcons to assist menu navigation. https://smartech.gatech.edu/handle/1853/50451a,b,eVilimek, R., & Hempel, T. (2005). Effects of speech and non-speech sounds on short-term memory and possible implications for in-vehicle use. https://smartech.gatech.edu/handle/1853/50156a,bWalker, B. N., & Kogan, A. (2009). Spearcon Performance and Preference for Auditory Menus on a Mobile Phone. In C. Stephanidis (Ed.), Universal Access in Human-Computer Interaction. Intelligent and Ubiquitous Interaction Environments (pp. 445–454). Springer. https://doi.org/10.1007/978-3-642-02710-9_49a,b,cWalker, B. N., Lindsay, J., Nance, A., Nakano, Y., Palladino, D. K., Dingler, T., & Jeon, M. (2013). Spearcons (Speech-Based Earcons) Improve Navigation Performance in Advanced Auditory Menus. Human Factors, 55(1), 157–182. https://doi.org/10.1177/0018720812450587bWarnock, D., McGee-Lennon, M., & Brewster, S. (2013). Multiple Notification Modalities and Older Users. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1091–1094. https://doi.org/10.1145/2470654.2466139eWarnock, D., McGee-Lennon, M. R., & Brewster, S. (2011). The Impact of Unwanted Multimodal Notifications. Proceedings of the 13th International Conference on Multimodal Interfaces, 177–184. https://doi.org/10.1145/2070481.2070510bWarren-Noell, H. L., Kaber, D. B., & Sheik-Nainar, M. A. (2006). Human Performance with Vocal Cueing of Automation State Changes in an Adaptive System. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 50(3), 415–419. https://doi.org/10.1177/154193120605000343a,eWolters, M., Isaac, K., & Doherty, J. (2012). Hold That Thought: Are Spearcons Less Disruptive than Spoken Reminders? CHI ’12 Extended Abstracts on Human Factors in Computing Systems, 1745–1750. https://doi.org/10.1145/2212776.2223703cXu, S. (2015). Improving Accessibility Design on Touchscreens. In M. Antona & C. Stephanidis (Eds.), Universal Access in Human-Computer Interaction. Access to Interaction (pp. 161–173). Springer International Publishing. https://doi.org/10.1007/978-3-319-20681-3_15Notes1. The NMA rankings are imperfect and should be interpreted cautiously – especially when several rankings have similarly grouped p values (Mbuagbaw et al., Citation2017; Riley et al., Citation2017). Rankings with wide differences in p values can provide an important piece of information about which treatments are likely to be better than others.