Prior audio-visual learning facilitates auditory-only speech and voice-identity recognition in noisy listening conditions

Perception of human communication signals is often more robust when there is concurrent input from the auditory and visual sensory modality. For instance, seeing the dynamic articulatory movements of a speaker, in addition to hearing their voice, can help with understanding what is said. This is particularly evident in noisy listening conditions. Even in the absence of concurrent visual input, visual mechanisms continue to be recruited to optimise auditory processing: auditory-only speech and voice-identity recognition is superior for speakers who have been previously learned with their corresponding face, in comparison to an audio-visual control condition; an effect termed the “face-benefit”. Whether the face-benefit can assist in maintaining robust perception in noisy listening conditions, in a similar manner to concurrent visual input, is currently unknown. Here, in two behavioural experiments, we explicitly examined this hypothesis. In each experiment, participants learned a series of speakers’ voices together with their corresponding dynamic face, or a visual control image depicting the speaker’s occupation. Following learning, participants listened to auditory-only sentences spoken by the same speakers and were asked to recognise the content of the sentences (i.e., speech recognition, Experiment 1) or the identity of the speaker (i.e., voice-identity recognition, Experiment 2) in different levels of increasing auditory noise (SNR +4 dB to -8 dB). For both speech and voice-identity recognition, we observed that for participants who showed a face-benefit, the benefit increased with the degree of noise in the auditory signal (Experiment 1, 2). Taken together, these results support an audio-visual model of human auditory communication and suggest that the brain has developed a flexible system to deal with auditory uncertainty – learned visual mechanisms are recruited to enhance the recognition of the auditory signal.