Incorporating user behavior information in IR evaluation

Many evaluation measures in Information Retrieval (IR) can be viewed as simple user models. Meanwhile, search logs provide us with information about how real users search. This paper describes our attempts to reconcile click log in- formation with user-centric IR measures, bringing the mea- sures into agreement with the logs. Studying the discount curve of NDCG and RBP leads us to extend them, incorpo- rating the probability of click in their discount curves. We measure accuracy of user models by calculating 'session like- lihood'. This leads us to propose a new IR evaluation mea- sure, Expected Browsing Utility (EBU), based on a more sophisticated user model. EBU has better session likelihood than existing measures, therefore we argue it is a better user-centric IR measure.