Large Scale Max-Margin Multi-Label Classification with Priors

We propose a max-margin formulation for the multi-label classification problem where the goal is to tag a data point with a set of pre-specified labels. Given a set of L labels, a data point can be tagged with any of the 2 L possible subsets. The main challenge there- fore lies in optimising over this exponentially large label space subject to label correlations. Existing solutions take either of two ap- proaches. The first assumes, a priori, that there are no label correlations and indepen- dently trains a classifier for each label (as is done in the 1-vs-All heuristic). This reduces the problem complexity from exponential to linear and such methods can scale to large problems. The second approach explicitly models correlations by pairwise label interac- tions. However, the complexity remains ex- ponential unless one assumes that label cor- relations are sparse. Furthermore, the learnt correlations reflect the training set biases. We take a middle approach that assumes la- bels are correlated but does not incorporate pairwise label terms in the prediction func- tion. We show that the complexity can still be reduced from exponential to linear while modelling dense pairwise label correlations. By incorporating correlation priors we can overcome training set biases and improve pre- diction accuracy. We provide a principled in- terpretation of the 1-vs-All method and show Appearing in Proceedings of the 27 th International Confer- ence on Machine Learning, Haifa, Israel, 2010. Copyright 2010 by the author(s)/owner(s). that it arises as a special case of our formu- lation. We also develop efficient optimisation algorithms that can be orders of magnitude faster than the state-of-the-art.