RAMD: registry-based anomaly malware detection using one-class ensemble classifiers

Malware is continuously evolving and becoming more sophisticated to avoid detection. Traditionally, the Windows operating system has been the most popular target for malware writers because of its dominance in the market of desktop operating systems. However, despite a large volume of new Windows malware samples that are collected daily, there is relatively little research focusing on Windows malware. The Windows Registry, or simply the registry, is very heavily used by programs in Windows, making it a good source for detecting malicious behavior. In this paper, we present RAMD, a novel approach that uses an ensemble classifier consisting of multiple one-class classifiers to detect known and especially unknown malware abusing registry keys and values for malicious intent. RAMD builds a model of registry behavior of benign programs and then uses this model to detect malware by looking for anomalous registry accesses. In detail, it constructs an initial ensemble classifier by training multiple one-class classifiers and then applies a novel swarm intelligence pruning algorithm, called memetic firefly-based ensemble classifier pruning (MFECP), on the ensemble classifier to reduce its size by selecting only a subset of one-class classifiers that are highly accurate and have diversity in their outputs. To combine the outputs of one-class classifiers in the pruned ensemble classifier, RAMD uses a specific aggregation operator, called Fibonacci-based superincreasing ordered weighted averaging (FSOWA). The results of our experiments performed on a dataset of benign and malware samples show that RAMD can achieve about 98.52% detection rate, 2.19% false alarm rate, and 98.43% accuracy.