Decoding Continuous Character-based Language from Non-invasive Brain Recordings
arxiv(2024)
摘要
Deciphering natural language from brain activity through non-invasive devices
remains a formidable challenge. Previous non-invasive decoders either require
multiple experiments with identical stimuli to pinpoint cortical regions and
enhance signal-to-noise ratios in brain activity, or they are limited to
discerning basic linguistic elements such as letters and words. We propose a
novel approach to decoding continuous language from single-trial non-invasive
fMRI recordings, in which a three-dimensional convolutional network augmented
with information bottleneck is developed to automatically identify responsive
voxels to stimuli, and a character-based decoder is designed for the semantic
reconstruction of continuous language characterized by inherent character
structures. The resulting decoder can produce intelligible textual sequences
that faithfully capture the meaning of perceived speech both within and across
subjects, while existing decoders exhibit significantly inferior performance in
cross-subject contexts. The ability to decode continuous language from single
trials across subjects demonstrates the promising applications of non-invasive
language brain-computer interfaces in both healthcare and neuroscience.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要