Obfuscated malware detection using API call dependency

Malwares pose a grave threat to security of a network and host systems. Many events such as Distributed Denial-of-Service attacks, spam emails etc., often have malwares as their root cause. So a great deal of research is being invested in detection and removal of malwares. Thus many malware detection systems or antivirus softwares have come up. But the drawback of these antivirus softwares is they rely upon signature matching approach for malware detection which can be easily defeated using simple code obfuscation techniques. This has given rise to a new generation of metamorphic and polymorphic malwares. In this paper we proposed the approach of monitoring interdependent system calls to detect obfuscated malicious programs. We took some sample malwares and some common obfuscation techniques. We tested these obfuscated malwares against open source antivirus ClamAV and our detection model. The results obtained have been elaborated further in the paper. Again how our algorithm is sound against many drawbacks of the API call monitoring approach such as API call reordering, garbage API call insertion etc., are also described.