A Decision Theoretic Framework for Ranking using Implicit Feedback

This paper presents a decision theoretic ranking system that incorporates both explicit and implicit feedback. The sys- tem has a model that predicts, given all available data at query time, dierent interactions a person might have with search results. Possible interactions include relevance la- belling and clicking. We define a utility function that takes as input the outputs of the interaction model to provide a real valued score to the user's session. The optimal rank- ing is the list of documents that, in expectation under the model, maximizes the utility for a user session. The system presented is based on a simple example util- ity function that combines both click behavior and labelling. The click prediction model is a Bayesian generalized linear model. Its notable characteristic is that it incorporates both weights for explanatory features and weights for each query- document pair. This allows the model to generalize to un- seen queries but makes it at the same time flexible enough to keep in a 'memory' where the model should deviate from its feature based prediction. Such a click-predicting model could be particularly useful in an application such as en- terprise search, allowing on-site adaptation to local docu- ments and user behaviour. The example utility function has a parameter that controls the tradeo between optimizing for clicks and optimizing for labels. Experimental results in the context of enterprise search show that a balance in the tradeo leads to the best NDCG and good (predicted) clickthrough.