Fourier Prompt Tuning for Modality-Incomplete Scene Segmentation
CoRR(2024)
摘要
Integrating information from multiple modalities enhances the robustness of
scene perception systems in autonomous vehicles, providing a more comprehensive
and reliable sensory framework. However, the modality incompleteness in
multi-modal segmentation remains under-explored. In this work, we establish a
task called Modality-Incomplete Scene Segmentation (MISS), which encompasses
both system-level modality absence and sensor-level modality errors. To avoid
the predominant modality reliance in multi-modal fusion, we introduce a
Missing-aware Modal Switch (MMS) strategy to proactively manage missing
modalities during training. Utilizing bit-level batch-wise sampling enhances
the model's performance in both complete and incomplete testing scenarios.
Furthermore, we introduce the Fourier Prompt Tuning (FPT) method to incorporate
representative spectral information into a limited number of learnable prompts
that maintain robustness against all MISS scenarios. Akin to fine-tuning
effects but with fewer tunable parameters (1.1
the efficacy of our proposed approach, showcasing an improvement of 5.84
over the prior state-of-the-art parameter-efficient methods in modality
missing. The source code will be publicly available at
https://github.com/RuipingL/MISS.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要