An empirical study on metamorphic testing for recommender systems

Context: Recommender systems are widely used in various fields because they can provide decision -making guidance to users facing an overwhelming set of choices. In previous studies, the accuracy of recommendations has been the focus and has significantly improved. However, the quality issues of these systems have been overlooked. In practical applications, the reliability of recommender systems plays an important role in their acceptance by users. Objective: This paper aims to develop a solution for performing metamorphic testing on recommender systems, and then to evaluate their reliability based on the test results. Methods: A metamorphic testing framework for recommender systems is first proposed to effectively alleviate the difficulty of the test oracle (i.e., the construction of the expected output of a program). Meanwhile, a set of specific metamorphic relations for recommender systems is also designed, and an empirical analysis is conducted using three open -source recommender libraries: LibRec, PREA, and Surprise. Results: The effectiveness of the proposed metamorphic testing solution is confirmed through the experiments, and the comparison analysis of the designed metamorphic relations and the three recommender libraries is also conducted, yielding the rankings of both the metamorphic relations and the program libraries, respectively. Conclusion: The study suggests that metamorphic testing is effective in automatically revealing the reliability problems in recommender systems, without requiring test oracles.