An advanced approach for modeling and detecting software vulnerabilities

Context: Passive testing is a technique in which traces collected from the execution of a system under test are examined for evidence of flaws in the system. Objective: In this paper we present a method for detecting the presence of security vulnerabilities by detecting evidence of their causes in execution traces. This is a new approach to security vulnerability detection. Method: Our method uses formal models of vulnerability causes, known as security goal models and vulnerability detection conditions (VDCs). The former are used to identify the causes of vulnerabilities and model their dependencies, and the latter to give a formal interpretation that is suitable for vulnerability detection using passive testing techniques. We have implemented modeling tools for security goal models and vulnerability detection conditions, as well as TestInv-Code, a tool that checks execution traces of compiled programs for evidence of VDCs. Results: We present the full definitions of security goal models and vulnerability detection conditions, as well as structured methods for creating both. We describe the design and implementation of TestInv-Code. Finally we show results obtained from running TestInv-Code to detect typical vulnerabilities in several open source projects. By testing versions with known vulnerabilities, we can quantify the effectiveness of the approach. Conclusion: Although the current implementation has some limitations, passive testing for vulnerability detection works well, and using models as the basis for testing ensures that users of the testing tool can easily extend it to handle new vulnerabilities.