Random or heuristic? An empirical study on path search strategies for test generation in KLEE

Randomness may be the most straightforward strategy and has been widely used in software engineering. One criticism for random strategy is its aimless for tasks. Therefore, many heuristic strategies have been proposed to improve the random strategy. However, it is impossible to prove that heuristics are better than randomness in theory, especial for software test generation. This open question is always left to empirical study and researchers expect to conclude some guidelines in practice. This paper studies a key technique, path search strategies, of test generation. We conducted an empirical evaluation and comparison among ten concrete path search approaches provided by KLEE, where two approaches belong to random search strategy and eight belong to heuristic search strategy, with 53 GNU Coreutils applications. We also investigated both cases with and without constraint optimization techniques for KLEE-based test generation. The experimental results show that without optimization, one approach from random strategy – random-path – performs better than other techniques in terms of the number of completed paths, statement coverage and branch coverage. These results indicate that random strategy can be a better choice for test generation in most cases without optimization, and heuristic strategies need further investigation. However, when combined with optimization, the selection of search strategy depends on the specific optimization applied. We further analyze the reasons behind the statistical results and provide guidelines to select the appropriate path search strategy for test generation in practice.