A Comparative Study on Combinatorial and Random Testing for Highly Configurable Systems.

Highly configurable systems (HCSs), such as software product lines, have complex configuration spaces. Combinatorial Testing and Random Testing are the main approaches to testing of HCSs. In this paper, we empirically compare their strengths with respect to scalability and diversity of sampled configurations (i.e., tests). We choose IcPL and QUICKSAMPLER to respectively represent Combinatorial Testing and Random Testing. Experiments are conducted to evaluate the t way coverage criterion of generated test suites for HCS benchmarks.