Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
ADVANCES IN NEURAL INFORMATION PROCESSING SYSTEMS 32 (NIPS 2019), pp. 14574-14583, 2019.
Mode connectivity and spurious valleys Fixing a neural network architecture, a data set D and a loss function, we say two sets of parameters/solutions ✓A and ✓B are ✏-connected if there is a path that is continuous with respect to t and satisfies: 1
Mode connectivity (Garipov et al., 2018; Draxler et al., 2018) is a surprising phenomenon in the loss landscape of deep nets. Optima-at least those discovered by gradient-based optimization-turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with...More
PPT (Upload PPT)