Mechanisms for protecting software integrity in networked systems

Protecting software integrity is a key to successfully maintain its own credibility and reduce the financial and technical risks caused from a lack of integrity. Although researchers have been putting effort on improving software development techniques and preventing human errors during the software development process, it is still a daunting task to make non-vulnerable software in practice. For example, the national vulnerability database shows that a set of new software vulnerabilities are discovered every day. Since developing non-vulnerable software is hardly achievable, in this research, we look for a way to achieve software integrity while they are used. In particular, this dissertation investigates three mechanisms to protect software integrity at runtime. Firstly, this dissertation presents a protection mechanism that can thwart attacks that try to exploit memory corruption vulnerabilities of software. The protection mechanism is provided by randomizing the program's runtime memory address layout and the memory objects. As a result, it hinders memory corruption attacks by preventing an attacker being able to easily predict their target addresses. The protection mechanism is implemented by a novel binary rewriting tool that can randomly place the code and data segments of programs and perform fine-grained permutation of function bodies in the code segment as well as global variables in the data segment. Our evaluation results show minimal performance overhead with orders of magnitude improvement in randomness. Secondly, this dissertation investigates a vulnerability identification mechanism named as CBones that can discover how unknown vulnerabilities in C programs are exploited by verifying program structural constraints. CBones automatically extracts a set of program structural constraints via binary analysis of the compiled program executable. CBone then verifies these constraints while it monitors the program execution to detect and isolate the security bugs. Our evaluation with real-world applications that known to have vulnerabilities shows that CBones can discover all integrity vulnerabilities with no false alarms, pinpoint the corrupting instructions, and provide information to facilitate the understanding of how an attack exploits a security bug. Lastly, this dissertation identifies the need of dynamic attestation to overcome the limitations of existing remote attestation approaches. To the best of our knowledge, we are the first to introduce the notion of dynamic attestation and propose use of dynamic system properties to provide the integrity proof of a running system. To validate our idea, we develop an application-level dynamic attestation system named as ReDAS (Remote Dynamic Attestation System) that can verify runtime integrity of software. ReDAS provides the integrity evidence of runtime applications by checking their dynamic properties: structural integrity and global data integrity. These properties are collected from each application, representing the application's unique runtime behavior that must be satisfied at runtime. ReDAS also uses hardware support provided by TPM to protect the integrity evidence from potential attacks. Our evaluation with real-world applications shows that ReDAS is effective in capturing runtime integrity violations with zero false alarms, and demonstrates that ReDAS incurs 8% overhead on average while performing integrity measurements.