Are Evaluation Metrics Identical With Binary Judgements?

Many information retrieval (IR) metrics are top-heavy, and some even have parameters for adjusting their discount curve. By choosing the right metric and parameters, the ex- perimenter can arrive at a discount curve that is appropriate for their setting. However, in many cases changing the dis- count curve may not change the outcome of an experiment. This poster considers query-level directional agreement be- tween DCG, AP, P@10, RBP(p = 0:5) and RBP(p = 0:8), in the case of binary relevance judgments. Results show that directional disagreements are rare, for both top-10 and top-1000 rankings. In many cases we considered, a change of discount is likely to have no eect on experimental out- comes.