Nonlinear Optimization in Machine Learning

3 Gradient Descent and Line Search Methods 11 3.1 Gradient Descent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.2 Back Propagation and GD on Neural Networks . . . . . . . . . . . . . . 13 3.3 Online Principle Component Analysis (PCA) . . . . . . . . . . . . . . . 14 3.4 Line Search Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3.5 Convergence to Approximate Second Order Necessary Points . . . . . . 16