Constructing Reliable Gradient Exploration for Online Learning to Rank

With the rapid development of information retrieval (IR) systems, online learning to rank (OLR) approaches, which allow retrieval systems to automatically learn best parameters from user interactions, have attracted great research interests in recent years. In OLR, the algorithms usually need to explore some uncertain retrieval results for updating current parameters meanwhile guaranteeing to produce quality retrieval results by exploiting what have already been learned, and the final retrieval results is an interleaved list from both exploratory and exploitative results. However, existing OLR algorithms perform exploration based on either only one stochastic direction or multiple randomly selected stochastic directions, which always involve large variance and uncertainty into the exploration, and may further harm the retrieval quality. Moreover, little historical exploration knowledge is considered when conducting current exploration. In this paper, we propose two online learning to rank algorithms that improve the reliability of the exploration by constructing robust exploratory directions. First, we describe a \textit{Dual-Point Dueling Bandit Gradient Descent} (DP-DBGD) approach with a \textit{Contextual Interleaving} (CI) method. In particular, the exploration of \textit{DP-DBGD} is carefully conducted via two opposite stochastic directions and the proposed \textit{CI} method constructs a qualified interleaved retrieval result list by taking historical explorations into account. Second, we introduce a \textit{Multi-Point Deterministic Gradient Descent} (MP-DGD) method that constructs a set of deterministic standard unit basis vectors for exploration. In \textit{MP-DGD}, each basis direction will be explored and the parameter updating is performed by walking along the combination of exploratory winners from the basis vectors. We conduct experiments on several datasets and show that both \textit{DP-DBGD} and \textit{MP-DGD} improve the online learning to rank performance over 10\% compared with baseline methods.