Modeling the Parameter Interactions in Ranking SVM with Low-rank Approximation

Ranking SVM, which formalizes the problem of learning a ranking model as that of learning a binary SVM on preference pairs of documents, is a state-of-the-art ranking model in information retrieval. The dual form solution of a linear Ranking SVM model can be written as a linear combination of the preference pairs, i.e., $\boldsymbol {w} = \sum _{(i,j)} \alpha _{ij} (\boldsymbol {x}_i - \boldsymbol {x}_j)$w=∑(i,j)αij(xi-xj), where $\alpha _{ij}$αij denotes the Lagrange parameters associated with each preference pair $(i,j)$(i,j). It is observed that there exist obvious interactions among the document pairs because two preference pairs could share a same document as their items, e.g., preference pairs $(d_1, d_2)$(d1,d2) and $(d_1, d_3)$(d1,d3) share the document $d_1$d1. Thus it is natural to ask if there also exist interactions over the model parameters $\alpha _{ij}$αij, which may be leveraged to construct better ranking models. This paper aims to answer the question. We empirically found that there exists a low-rank structure over the rearranged Ranking SVM model parameters $\alpha _{ij}$αij, which indicates that the interactions do exist. Based on the discovery, we made modifications on the original Ranking SVM model by explicitly applying low-rank constraints to the Lagrange parameters, achieving two novel algorithms called Factorized Ranking SVM and Regularized Ranking SVM, respectively. Specifically, in Factorized Ranking SVM each parameter $\alpha _{ij}$αij is decomposed as a product of two low-dimensional vectors, i.e., $\alpha _{ij} = \langle \boldsymbol {v}_i, \boldsymbol {v}_j\rangle$αij=〈vi,vj〉, where vectors $\boldsymbol {v}_i$vi and $\boldsymbol {v}_j$vj correspond to document $i$i and $j$j, respectively; In Regularized Ranking SVM, a nuclear norm is applied to the rearranged parameters matrix for controlling its rank. Experimental results on three LETOR datasets show that both of the proposed methods can outperform state-of-the-art learning to rank models including the conventional Ranking SVM.