All-Uses versus Mutation Testing : An ExperimentalComparison of E ectiveness

The eeectiveness of a test data adequacy criterion for a given program and spec-iication is the probability that a test set satisfying the criterion will expose a fault. Experiments were performed to compare the eeectiveness of the mutation testing and all-uses test data adequacy criteria at various coverage levels, for randomly generated test sets. Large numbers of test sets were generated and executed, and for each, the proportion of mutants killed or defuse associations covered was measured. This data was used to estimate and compare the eeec-tiveness of the criteria. The results were mixed: at the highest coverage levels considered, mutation was more eeective than all-uses for ve of the nine subjects, all-uses was more eeective than mutation for two subjects, and there was no clear winner for two subjects. However, mutation testing was much more expensive than all-uses. The relationship between coverage and eeectiveness for xed-sized test sets was also explored and was found to be non-linear and, in many cases, non-monotonic.