Mercury: A High-Performance Streaming Graph Method for Broad and Deep Flow Inspection.

Deep flow inspection (DFI) has been increasingly important for network management and security protection. Network flows have rich correlations due to multiparty communications such as video conferences and DDoS attacks, and furthermore, rich features are vital for artificial intelligence-driven DFI models and algorithms. However, existing approaches do not correlate network flows by just providing simple counters for key-valued network flow records due to the limitations of the measurement models.We present Mercury, a broad and deep flow inspection system, which continuously collects a wide range of features for large scales of correlated network flows and supports expressive query interfaces for diverse network measurement tasks. Mercury organizes rich and dynamic features of the network traffic based on a streaming graph model, which defines rich states for each network flow to the edge of the graph and incrementally updates these states in high-performance pipelines. Mercury raises significant challenges for storing and updating states in the streaming graph model due to the fast arrival rate and the in-memory storage limits. We present a nested two-dimensional data store, called cuckoo matrix, to provide a two-dimensional index for the streaming graph model. The cuckoo matrix optimizes the state-update throughput and reduces the memory cost through the multiple dimensions of the cuckoo tables. We speed up the ingestion process based on a threaded lock-free butter and update the feature statistics based on a unified streaming framework.Extensive evaluation with real-world data sets demonstrates that Mercury achieves a significantly better trade-off between throughput and memory consumption than state-of-the-art methods. Mercury supports both traditional machine learning methods and accurate graph neural networks for traffic classification through extensive and efficient graph query APIs.