On the Consistency of Ranking Algorithms

We present a theoretical analysis of super- vised ranking, providing necessary and suf- ficient conditions for the asymptotic consis- tency of algorithms based on minimizing a surrogate loss function. We show that many commonly used surrogate losses are incon- sistent; surprisingly, we show inconsistency even in low-noise settings. We present a new value-regularized linear loss, establish its consistency under reasonable assumptions on noise, and show that it outperforms conven- tional ranking losses in a collaborative filter- ing experiment.