RecIPE: Revisiting the Evaluation of Memory Error Defenses.

Many detection and defense mechanisms have been proposed to prevent and mitigate memory errors. A good understanding of the pros and cons of memory defense mechanisms is essential before practical deployment. However, the environment, compilers, and defenses change over time. Thus, a benchmark providing comprehensive evaluations of a range of compilation option choices and defenses is helpful to give developers guidance. The most well-known test suite for evaluating spatial memory errors and exploits is RIPE. However, we show that it is no longer applicable, and a new benchmark is needed. Furthermore, a benchmark should be extensible to evolve over time easily. We propose RecIPE, a new extensible and comprehensive benchmark for evaluating memory error defenses. For extensibility and customisability, RecIPE consists of two components. The TestGen component generates vulnerable code from configurable templates across various vulnerable attributes allowing easy changes through the templates. Measuring the attack and exploitation is with the DefEval component, serving as an attacker at runtime and can be updated with new exploitations. We present a comprehensive evaluation of compiler defenses and sanitizers on RecIPE with gcc and clang, showing the strengths and weaknesses of the various defenses. The results show pros and cons which may not be well known, including gaps between the in-principle guarantee and practical implementation. Our results also point out directions for further improvement in defenses.