Projection Based Weight Normalization: Efficient Method for Optimization on Oblique Manifold in DNNs

•We propose to constrain the incoming weights of each neuron to be unit norm to address ill conditioned problem in DNNs.•Constrain ing weight s can be formulated as an optimization problem over the Oblique manifold.•We propose a simple yet efficient method referred to as projection based weight normalization (PBWN) to solve the optimization problem.•PBWN has the property of regularization and collaborates well with the commonly used batch normalization technique.•Extensive experiments on several widely used image datasets show the consistent performance improvement over the baseline DNNs.