RanDumb: A Simple Approach that Questions the Efficacy of Continual Representation Learning
CoRR(2024)
摘要
We propose RanDumb to examine the efficacy of continual representation
learning. RanDumb embeds raw pixels using a fixed random transform which
approximates an RBF-Kernel, initialized before seeing any data, and learns a
simple linear classifier on top. We present a surprising and consistent
finding: RanDumb significantly outperforms the continually learned
representations using deep networks across numerous continual learning
benchmarks, demonstrating the poor performance of representation learning in
these scenarios. RanDumb stores no exemplars and performs a single pass over
the data, processing one sample at a time. It complements GDumb, operating in a
low-exemplar regime where GDumb has especially poor performance. We reach the
same consistent conclusions when RanDumb is extended to scenarios with
pretrained models replacing the random transform with pretrained feature
extractor. Our investigation is both surprising and alarming as it questions
our understanding of how to effectively design and train models that require
efficient continual representation learning, and necessitates a principled
reinvestigation of the widely explored problem formulation itself. Our code is
available at https://github.com/drimpossible/RanDumb.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要