User-Centered Multimedia Retrieval Evaluation based on Empirical Research

The evaluation of retrieval mechanisms for inter-method comparison is necessary in academic as well as in applied research. Major issue in evaluations is in which way and to what extent the actual perception of users from the target user group is integrated. Within multimedia retrieval systems impressions and perceptions of users vary much more than in text retrieval. Empirical studies are a common tool in social science and offer a way to research the correlation between the user perception and the computed similarity between pairs of multimedia documents or a query and the set of results. This approach can be used to complement and extend current evaluation approaches. Within this contribution we revisit and summarize general methods from social science and psychology for the interested reader in the area of computer science with some knowledge about statistics. Furthermore we give two examples of undertaken empirical experiments and their outcomes. Within the first one the perception of users is investigated and compared to factors like educational context and gender, while in the second study metrics are tested upon their ability to reflect the notion of similarity of users. Both experiments aim to give examples and insight on how empirical studies can be conducted inmultimedia research in general and multimedia retrieval evaluation in special.