Learning Efficiently with Approximate Inference via Dual Losses

Many structured prediction tasks involve complex models where inference is computa- tionally intractable, but where it can be well approximated using a linear programming relaxation. Previous approaches for learn- ing for structured prediction (e.g., cutting- plane, subgradient methods, perceptron) re- peatedly make predictions for some of the data points. These approaches are computa- tionally demanding because each prediction involves solving a linear program to optimal- ity. We present a scalable algorithm for learn- ing for structured prediction. The main idea is to instead solve the dual of the structured prediction loss. We formulate the learning task as a convex minimization over both the weights and the dual variables corresponding to each data point. As a result, we can be- gin to optimize the weights even before com- pletely solving any of the individual predic- tion problems. We show how the dual vari- ables can be eciently optimized using co- ordinate descent. Our algorithm is compet- itive with state-of-the-art methods such as stochastic subgradient and cutting-plane.