CLCE: An Approach to Refining Cross-Entropy and Contrastive Learning for Optimized Learning Fusion
CoRR(2024)
摘要
State-of-the-art pre-trained image models predominantly adopt a two-stage
approach: initial unsupervised pre-training on large-scale datasets followed by
task-specific fine-tuning using Cross-Entropy loss (CE). However, it has been
demonstrated that CE can compromise model generalization and stability. While
recent works employing contrastive learning address some of these limitations
by enhancing the quality of embeddings and producing better decision
boundaries, they often overlook the importance of hard negative mining and rely
on resource intensive and slow training using large sample batches. To counter
these issues, we introduce a novel approach named CLCE, which integrates
Label-Aware Contrastive Learning with CE. Our approach not only maintains the
strengths of both loss functions but also leverages hard negative mining in a
synergistic way to enhance performance. Experimental results demonstrate that
CLCE significantly outperforms CE in Top-1 accuracy across twelve benchmarks,
achieving gains of up to 3.52
transfer learning settings with the BEiT-3 model. Importantly, our proposed
CLCE approach effectively mitigates the dependency of contrastive learning on
large batch sizes such as 4096 samples per batch, a limitation that has
previously constrained the application of contrastive learning in
budget-limited hardware environments.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要