Few-Shot Adversarial Prompt Learning on Vision-Language Models
arxiv(2024)
摘要
The vulnerability of deep neural networks to imperceptible adversarial
perturbations has attracted widespread attention. Inspired by the success of
vision-language foundation models, previous efforts achieved zero-shot
adversarial robustness by aligning adversarial visual features with text
supervision. However, in practice, they are still unsatisfactory due to several
issues, including heavy adaptation cost, suboptimal text supervision, and
uncontrolled natural generalization capacity. In this paper, to address these
issues, we propose a few-shot adversarial prompt framework where adapting input
sequences with limited data makes significant adversarial robustness
improvement. Specifically, we achieve this by providing adversarially
correlated text supervision that is end-to-end learned from adversarial
examples. We also propose a novel training objective that enhances the
consistency of multi-modal features while encourages differentiated uni-modal
features between natural and adversarial examples. The proposed framework gives
access to learn adversarial text supervision, which provides superior
cross-modal adversarial alignment and matches state-of-the-art zero-shot
adversarial robustness with only 1
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要