Non-convex Optimization for Machine Learning

AbstractA vast majority of machine learning algorithms train their models andperform inference by solving optimization problems. In order to capturethe learning and prediction problems accurately, structural constraintssuch as sparsity or low rank are frequently imposed or else the objectiveitself is designed to be a non-convex function. This is especially trueof algorithms that operate in high-dimensional spaces or that trainnon-linear models such as tensor models and deep networks.The freedom to express the learning problem as a non-convex optimizationproblem gives immense modeling power to the algorithmdesigner, but often such problems are NP-hard to solve. A popularworkaround to this has been to relax non-convex problems to convexones and use traditional methods to solve the convex relaxed optimizationproblems. However this approach may be lossy and neverthelesspresents significant challenges for large scale optimization.On the other hand, direct approaches to non-convex optimizationhave met with resounding success in several domains and remain themethods of choice for the practitioner, as they frequently outperformrelaxation-based techniques - popular heuristics include projected gradientdescent and alternating minimization. However, these are oftenpoorly understood in terms of their convergence and other properties.This monograph presents a selection of recent advances that bridgea long-standing gap in our understanding of these heuristics. We hopethat an insight into the inner workings of these methods will allow thereader to appreciate the unique marriage of task structure and generativemodels that allow these heuristic techniques to provably succeed.The monograph will lead the reader through several widely used nonconvexoptimization techniques, as well as applications thereof. Thegoal of this monograph is to both, introduce the rich literature in thisarea, as well as equip the reader with the tools and techniques neededto analyze these simple procedures for non-convex problems.