MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training
arxiv(2023)
摘要
Contrastive pretraining of image-text foundation models, such as CLIP,
demonstrated excellent zero-shot performance and improved robustness on a wide
range of downstream tasks. However, these models utilize large
transformer-based encoders with significant memory and latency overhead which
pose challenges for deployment on mobile devices. In this work, we introduce
MobileCLIP – a new family of efficient image-text models optimized for runtime
performance along with a novel and efficient training approach, namely
multi-modal reinforced training. The proposed training approach leverages
knowledge transfer from an image captioning model and an ensemble of strong
CLIP encoders to improve the accuracy of efficient models. Our approach avoids
train-time compute overhead by storing the additional knowledge in a reinforced
dataset. MobileCLIP sets a new state-of-the-art latency-accuracy tradeoff for
zero-shot classification and retrieval tasks on several datasets. Our
MobileCLIP-S2 variant is 2.3× faster while more accurate compared to
previous best CLIP model based on ViT-B/16. We further demonstrate the
effectiveness of our multi-modal reinforced training by training a CLIP model
based on ViT-B/16 image backbone and achieving +2.9
improvement on 38 evaluation benchmarks compared to the previous best.
Moreover, we show that the proposed approach achieves 10×-1000×
improved learning efficiency when compared with non-reinforced CLIP training.
Code and models are available at https://github.com/apple/ml-mobileclip .
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要