Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection in Low-Resource Realistic Scenarios

This study presents an audio-visual information fusion approach to soundevent localization and detection (SELD) in low-resource scenarios. We aim atutilizing audio and video modality information through cross-modal learning andmulti-modal fusion. First, we propose a cross-modal teacher-student learning(TSL) framework to transfer information from an audio-only teacher model,trained on a rich collection of audio data with multiple data augmentationtechniques, to an audio-visual student model trained with only a limited set ofmulti-modal data. Next, we propose a two-stage audio-visual fusion strategy,consisting of an early feature fusion and a late video-guided decision fusionto exploit synergies between audio and video modalities. Finally, we introducean innovative video pixel swapping (VPS) technique to extend an audio channelswapping (ACS) method to an audio-visual joint augmentation. Evaluation resultson the Detection and Classification of Acoustic Scenes and Events (DCASE) 2023Challenge data set demonstrate significant improvements in SELD performances.Furthermore, our submission to the SELD task of the DCASE 2023 Challenge ranksfirst place by effectively integrating the proposed techniques into a modelensemble.