JEP-KD: Joint-Embedding Predictive Architecture Based Knowledge Distillation for Visual Speech Recognition
arxiv(2024)
摘要
Visual Speech Recognition (VSR) tasks are generally recognized to have a
lower theoretical performance ceiling than Automatic Speech Recognition (ASR),
owing to the inherent limitations of conveying semantic information visually.
To mitigate this challenge, this paper introduces an advanced knowledge
distillation approach using a Joint-Embedding Predictive Architecture (JEPA),
named JEP-KD, designed to more effectively utilize audio features during model
training. Central to JEP-KD is the inclusion of a generative network within the
embedding layer, which enhances the video encoder's capacity for semantic
feature extraction and brings it into closer alignment with the audio features
from a pre-trained ASR model's encoder. This approach aims to progressively
reduce the performance gap between VSR and ASR. Moreover, a comprehensive
multimodal, multistage training regimen for the JEP-KD framework is
established, bolstering the robustness and efficacy of the training process.
Experiment results demonstrate that JEP-KD significantly improves the
performance of VSR models and demonstrates versatility across different VSR
platforms, indicating its potential for broader application within other
multimodal tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要