How Good is a Single Basin?
CoRR(2024)
摘要
The multi-modal nature of neural loss landscapes is often considered to be
the main driver behind the empirical success of deep ensembles. In this work,
we probe this belief by constructing various "connected" ensembles which are
restricted to lie in the same basin. Through our experiments, we demonstrate
that increased connectivity indeed negatively impacts performance. However,
when incorporating the knowledge from other basins implicitly through
distillation, we show that the gap in performance can be mitigated by
re-discovering (multi-basin) deep ensembles within a single basin. Thus, we
conjecture that while the extra-basin knowledge is at least partially present
in any given basin, it cannot be easily harnessed without learning it from
other basins.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要