On the suitability of checked coverage and genetic parameter tuning in test suite reduction

As software projects evolve and grow in size and complexity, so do their test suites. Test suite reduction (TSR) aims at reducing the size of a test suite by removing redundant and obsolete test cases based on a coverage metric while preserving its fault detection capabilities. The contributions of this paper are twofold: (1) We examine a lesser-known coverage criterion, that is, checked coverage. Checked coverage not only investigates if a part of the code was executed but also if it was checked by a test oracle. In an empirical evaluation, we performed TSR based on different reduction algorithms, coverage metrics, and open-source Java projects with our own TSR tool to determine the most effective and efficient combination of metric and method. (2) Given the results of the first evaluation, we further investigate the potential of parameter optimization in regard to a genetic reduction algorithm. In particular, we focus on finding a general setting for the parameters crossover rate and mutation rate such that test suites can be reduced in a reasonable time while maintaining a high fault detection power. In this paper, we explore (1) the potential of checked coverage as a coverage metric as well as (2) parameter tuning for a genetic algorithm in the context of test suite reduction. In two empirical evaluations, we assess how different coverage metrics, algorithm combinations, and parameter values perform on open-source Java projects and our own Java reduction framework; based on the results we determine the most effective and efficient combination of metric and method. image