Automatic Assessment of Dysarthria Using Audio-visual Vowel Graph Attention Network

Automatic assessment of dysarthria remains a highly challenging task due tohigh variability in acoustic signals and the limited data. Currently, researchon the automatic assessment of dysarthria primarily focuses on two approaches:one that utilizes expert features combined with machine learning, and the otherthat employs data-driven deep learning methods to extract representations.Research has demonstrated that expert features are effective in representingpathological characteristics, while deep learning methods excel at uncoveringlatent features. Therefore, integrating the advantages of expert features anddeep learning to construct a neural network architecture based on expertknowledge may be beneficial for interpretability and assessment performance. Inthis context, the present paper proposes a vowel graph attention network basedon audio-visual information, which effectively integrates the strengths ofexpert knowledges and deep learning. Firstly, various features were combined asinputs, including knowledge based acoustical features and deep learning basedpre-trained representations. Secondly, the graph network structure based onvowel space theory was designed, allowing for a deep exploration of spatialcorrelations among vowels. Finally, visual information was incorporated intothe model to further enhance its robustness and generalizability. The methodexhibited superior performance in regression experiments targeting Frenchayscores compared to existing approaches.