Call Me Maybe: Using NLP to Automatically Generate Unit Test Cases Respecting Temporal Constraints

A class may need to obey temporal properties in order to function correctly. For example, the correct usage protocol for an iterator is to always check whether there is a next element before asking for it; iterating over a collection when there are no items left leads to a NoSuchElementException. Automatic test case generation tools such as Randoop and Evo- Suite do not have any notion of these temporal properties. Gener- ating test cases by randomly invoking methods on a new instance of the class under test may raise run-time exceptions that do not necessarily expose software faults, but are rather a consequence of violations of temporal properties. This paper presents Call Me Maybe, a novel technique that uses natural language processing to analyze Javadoc comments to identify temporal properties. This information can guide a test case generator towards executing sequences of method calls that respect the temporal properties. Our evaluation on 73 subjects from seven popular Java systems show that Randoop flags over 10K fewer false negatives and enriches over 12K correctly failing test cases due to violations of temporal properties with clear explanation that can help software developers.