Lightweight Hierarchical Network Traffic Clustering

We summarize our work with ADHIC (Approximate Divisive HIerarchical Clusterer), a lightweight, online, divisive hierarchical clustering algorithm tailored to the domain of network traffic clustering. We then briefly describe our implementation of ADHIC, NetADHICT, which serves as a tool to system administrators. The key innovation is that it can identify and present a hierarchical decomposition of traffic based upon the learned structure of whole packets without prior knowledge of protocol structures. ADHIC needs only a small fraction of packets to generate the cluster decision tree, and the generated tree can be used to cluster packets at wire speeds. Our experiments show NetADHICT can appropriately segregate well-known protocols, cluster traffic of the same protocol together even if it is running on multiple ports, and segregate p2p traffic that uses non-standard ports. We believe that ADHIC and NetADHICT are a useful complement to critical applications used for performance analysis, identification of worms and flash crowds, and Denial-of-Service resistant bandwidth management. ]