Efficient Analysis of Live and Historical Streaming Data and its Application to Cybersecurity

This paper describes our experiences building a coherent framework for ef- ficient simultaneous querying of live and archived stream data. This work was motivated by the need to analyze the network traffic patterns of research labora- tories funded by the U.S. Department of Energy. We review the requirements of such a system and implement a prototype based on the TelegraphCQ streaming query processor and the FastBit bitmap index. The combined system uses Tele- graphCQ to analyze streams of traffic information and FastBit to correlate current behaviors with historical trends. We present a detailed performance analysis of our system based on a complex query workload and real network traffic collected at Lawrence Berkeley National Laboratory (Berkeley Lab). Our experiments identify key performance bottlenecks for stream query processing systems that incorporate historical data. We also identify strategies for mitigating these bottlenecks. With these strategies in place, we demonstrate that it is possible for our system to ana- lyze the entire traffic of the DOE lab network on a small cluster of machines.