A Generalized Evaluation Framework For Multimedia Recommender Systems

With the widespread availability of media technologies, such as real-time streaming, new Internet-of-Thing devices and smart phones, multimedia data are extensively increased and the big multimedia data rapidly spread over various social networks. This has created complexity and information overload for users to choose the suitable multimedia objects. Thus, different multimedia recommender systems have been emerging to help users find the useful multimedia objects that are possibly preferred by the user. However, the evaluation of these multimedia recommender systems is still in an ad-hoc stage. Given the distinct features of multimedia objects, the evaluation criteria adopted from the general recommender systems might not be effectively used to evaluate multimedia recommendations. In this paper, we therefore review and analyze the evaluation criteria that have been used in the previous multimedia recommender system papers. Based on the review, we propose a generalized evaluation framework to guide the researchers and practitioners to perform evaluations, especially user-centric evaluations, for multimedia recommender systems.