Implicit Bias Of Gradient Descent On Linear Convolutional Networks

We show that gradient descent on full width linear convolutional networks of depth L converges to a linear predictor related to the l(2/L) bridge penalty in the frequency domain. This is in contrast to fully connected linear networks, where regardless of depth, gradient descent converges to the l(2) maximum margin solution.