LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL Architectures
ICLR 2024(2023)
摘要
Joint embedding (JE) architectures have emerged as a promising avenue for
acquiring transferable data representations. A key obstacle to using JE
methods, however, is the inherent challenge of evaluating learned
representations without access to a downstream task, and an annotated dataset.
Without efficient and reliable evaluation, it is difficult to iterate on
architectural and training choices for JE methods. In this paper, we introduce
LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the
quality of representations within JE architectures. Our metric addresses
several shortcomings of recent approaches based on feature covariance rank by
discriminating between informative and uninformative features. In essence,
LiDAR quantifies the rank of the Linear Discriminant Analysis (LDA) matrix
associated with the surrogate SSL task -- a measure that intuitively captures
the information content as it pertains to solving the SSL task. We empirically
demonstrate that LiDAR significantly surpasses naive rank based approaches in
its predictive power of optimal hyperparameters. Our proposed criterion
presents a more robust and intuitive means of assessing the quality of
representations within JE architectures, which we hope facilitates broader
adoption of these powerful techniques in various domains.
更多查看译文
关键词
Self Supervised Learning,Joint Embedding Architectures
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要