The ir game - a tool for rapid query analysis in cross-language ir experiments

Abstract. Standard ,performance ,tests of information ,retrieval systems ,are based on measuring precision at fixed recall levels and averaging results over a large set of test topics. It has recently been demonstrated,how,seemingly,equal average,performance ,may ,obscure ,important ,differences between ,search methods. The in-depth analysis of individual ,queries is needed ,for revealing the hidden differences. This paper introduces the motivation, structure and functionality of a ,novel WWW-based ,information ,retrieval tool suitable for individual query analysis. The ,tool allows easy implementation ,of different interactive retrieval methods ,and contains new ,features for supporting rapid evaluation of the ,search results. Implementation ,of an ,effective dictionary- based,method ,for cross-language information ,retrieval is presented ,as an example,of the approach. Possible uses of the tool include exploring the effects of different query formulations at the level of individual queries and collecting user information.