Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets

ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), pp. 14574-14583, 2019.

Cited by: 26|Views156
EI
Weibo:
Mode connectivity and spurious valleys Fixing a neural network architecture, a data set D and a loss function, we say two sets of parameters/solutions ✓A and ✓B are ✏-connected if there is a path that is continuous with respect to t and satisfies: 1

Abstract:

Mode connectivity (Garipov et al., 2018; Draxler et al., 2018) is a surprising phenomenon in the loss landscape of deep nets. Optima-at least those discovered by gradient-based optimization-turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with...More

Code:

Data:

0
Your rating :
0

 

Tags
Comments