Definition of a Knowledge Base Towards a Benchmark for Experiments with Mutation Testing.

Context: Mutation testing has been investigated since the late 70’s. Ever since, researchers have devised dozens of mutation approaches, including ways of generating, executing, and analyzing mutants, as well as ways of reducing the costs for its application. However, research on this field falls short when it comes to producing a representative and manageable set of artifacts to enable experiments with the plethora of existing mutation approaches. Objective: In this paper, we describe the process and current results of creating a knowledge base of mutation-related artifacts to support experiments with mutation testing. Method: We setup the Evosuite tool for generating test cases, and the PIT tool for generating and running mutants. We also created scripts to import the results to a relational database. The database includes some procedures to generate killing matrices for the tested Java classes. Results: Beyond establishing the tooling infrastructure, we populated our database with classes extracted from five Java projects, from which four are open source projects hosted in GitHub and the fifth one is composed by simple Java programs. Currently, the database includes around 2,000 classes, 50,000 test cases, and 195,000 mutants. Conclusion: The database structure eases the addition of other Java programs and the related mutation artifacts. Furthermore, it allows for tasks such as minimizing test sets and mutant sets (e.g. by removing redundant tests and trivial mutants), thus providing researchers with a well-established and extensible basis to support varied experiments.