Non-Gaussianity of Stochastic Gradient Noise

Panigrahi Abhishek
Panigrahi Abhishek
Goyal Navin
Goyal Navin
Cited by: 7|Bibtex|Views7|Links

Abstract:

What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes 256 and above, the...More

Code:

Data:

Your rating :
0

 

Tags
Comments