A study on malicious software behaviour analysis and detection techniques: Taxonomy, current trends and challenges

There has been an increasing trend of malware release, which raises the alarm for security professionals worldwide. It is often challenging to stay on top of different types of malware and their detection techniques, which are essential, particularly for researchers and the security community. Analysing malware to get insights into what it intends to perform on the victim’s system is one of the crucial steps towards malware detection. Malware analysis can be performed through static analysis, code analysis, dynamic analysis, memory analysis and hybrid analysis techniques. The next step to malware analysis is the detection model’s design using malware’s extracted patterns from the analysis. Machine learning and deep learning methods have drawn attention to researchers, owing to their ability to implement sophisticated malware detection models that can deal with known and unknown malicious activities. Therefore, this survey presents a comprehensive study and analysis of current malware and detection techniques using the snowball approach. It presents a comprehensive study on malware analysis testbeds, dynamic malware analysis and memory analysis, the taxonomy of malware behaviour analysis tools, datasets repositories, feature selection, machine learning and deep learning techniques. Moreover, comparisons of behaviour-based malware detection techniques have been grouped by categories of machine learning and deep learning techniques. This study also looks at various performance evaluation metrics, current research challenges in this area and possible future direction of research.