Variance-Covariance Regularization Improves Representation Learning
CoRR(2023)
摘要
Transfer learning plays a key role in advancing machine learning models, yet
conventional supervised pretraining often undermines feature transferability by
prioritizing features that minimize the pretraining loss. In this work, we
adapt a self-supervised learning regularization technique from the VICReg
method to supervised learning contexts, introducing Variance-Covariance
Regularization (VCReg). This adaptation encourages the network to learn
high-variance, low-covariance representations, promoting learning more diverse
features. We outline best practices for an efficient implementation of our
framework, including applying it to the intermediate representations. Through
extensive empirical evaluation, we demonstrate that our method significantly
enhances transfer learning for images and videos, achieving state-of-the-art
performance across numerous tasks and datasets. VCReg also improves performance
in scenarios like long-tail learning and hierarchical classification.
Additionally, we show its effectiveness may stem from its success in addressing
challenges like gradient starvation and neural collapse. In summary, VCReg
offers a universally applicable regularization framework that significantly
advances transfer learning and highlights the connection between gradient
starvation, neural collapse, and feature transferability.
更多查看译文
关键词
representation,learning,variance-covariance
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要