Towards a Big Data Architecture for Facilitating Cyber Threat Intelligence

Internet and organizational network security is still threatened by devastating malicious activities. Given the continuous escalation of such attacks in terms of their frequency, sophistication and stealthiness, it is of paramount importance to generate effective cyber threat intelligence that aim at inferring, attributing, characterizing and mitigating such misdemeanors. Nevertheless, such imperative tasks are partially impeded by the lack of approaches that can produce prompt and accurate actionable intelligence by investigating various network traffic sources. In this paper, we propose and evaluate a big data architecture that is rooted in real-time network traffic processing, distributed messaging and scalable data storage. The key innovation behind the proposed architecture is that it automates the analysis of heterogeneous network data, allowing the focus to remain on devising effective cyber threat intelligence analytics, rather than being hindered by data management, aggregation, reconciliation and formatting. Empirical evaluations investigating the application of machine learning analytics by exploiting the artifacts of the proposed architecture and by using 100 GB of real network traffic, indeed demonstrate the practicality, effectiveness, and addedvalue of the proposed architecture.