Neural2Speech: A Transfer Learning Framework for Neural-Driven Speech Reconstruction

Reconstructing natural speech from neural activity is vital for enablingdirect communication via brain-computer interfaces. Previous efforts haveexplored the conversion of neural recordings into speech using complex deepneural network (DNN) models trained on extensive neural recording data, whichis resource-intensive under regular clinical constraints. However, achievingsatisfactory performance in reconstructing speech from limited-scale neuralrecordings has been challenging, mainly due to the complexity of speechrepresentations and the neural data constraints. To overcome these challenges,we propose a novel transfer learning framework for neural-driven speechreconstruction, called Neural2Speech, which consists of two distinct trainingphases. First, a speech autoencoder is pre-trained on readily available speechcorpora to decode speech waveforms from the encoded speech representations.Second, a lightweight adaptor is trained on the small-scale neural recordingsto align the neural activity and the speech representation for decoding.Remarkably, our proposed Neural2Speech demonstrates the feasibility ofneural-driven speech reconstruction even with only 20 minutes of intracranialdata, which significantly outperforms existing baseline methods in terms ofspeech fidelity and intelligibility.