Wav2vec-MoE: An unsupervised pre-training and adaptation method for multi-accent ASR

In real life, either the subjective factors of speakers or the objective environment degrades the performance of automatic speech recognition (ASR). This study focuses on one of the subjective factors, accented speech, and attempts to build a multi-accent ASR system to solve the degradation caused by different accents, one of whose characteristic is the low resource. To deal with the challenge of the low-resource data and the different speech styles, a wav2vec-MoE (mixture of experts) is proposed to adapt the wav2vec 2.0 for multi-accent ASR. In the wav2vec-MoE, a domain MoE is developed by introducing pseudo-domain information in the pre-training stage, where the domain denotes a collection of speech varied by the same influence factors. The MoE is trained with two strategies according to the proposed domain mismatch assessment between unlabeled speech and target speech, without requiring any explicit domain information. Experiments show that the wav2vec-MoE achieves a 14.69% relative word error rate reduction (WERR) on the AESRC2020 accent dataset and an 8.79% relative WERR on the Common Voice English dataset.