SoK: Automatic Deobfuscation of Virtualization-protected Applications

Malware authors often rely on code obfuscation to hide the malicious functionality of their software, making detection and analysis more difficult. One of the most advanced techniques for binary obfuscation is virtualization-based obfuscation, which converts the functionality of a program into the bytecode of a randomly generated virtual machine which is embedded into the protected program. To enable the automatic detection and analysis of protected malware, new deobfuscation techniques against virtualization-based obfuscation are constantly being developed and proposed in the literature. In this work, we systematize existing knowledge of automatic deobfuscation of virtualization-protected programs in a novel classification scheme and evaluate where we stand in the arms race between malware authors and code analysts in regards to virtualization-based obfuscation. In addition to a theoretical discussion of different types of deobfuscation methodologies, we present an in-depth practical evaluation that compares state-of-the-art virtualization-based obfuscators with currently available deobfuscation tools. The results clearly indicate the possibility of automatic deobfuscation of virtualization-based obfuscation in specific scenarios. Furthermore, however, the results highlight limitations of existing deobfuscation methods. Multiple challenges still lie ahead on the way towards reliable and resilient automatic deobfuscation of virtualization-based obfuscation.