FECC: DNS tunnel detection model based on CNN and clustering

As the basic service of the network, the Domain Name System (DNS) is almost never blocked by the fire -wall. DNS tunnel takes advantage of this feature of DNS service to achieve barrier-free communication be-tween the internal network and the external network, thereby implementing malicious network activities such as data theft and Command & Control (C2) control. As an intrusion means, DNS tunnel constitutes a serious threat to network security. At present, some research work on DNS tunnel detection is imple-mented based on manual features, which are "explicit" features. DNS tunnels can specifically circumvent these "explicit" features as they are designed, rendering detection methods ineffective. Additionally, the quality of these manual features also relies heavily on expert knowledge. The features extracted by the detection method based on convolutional neural network (CNN) have the characteristics of "implicit" that are not easily circumvented by DNS tunnels. However, the homogeneity and exclusivity of features are not considered in the existing research works that only perform the model training for classification tasks on datasets with limited sample types. This can cause the model to fail when detecting variants or new types of DNS tunnels. This paper proposes a novel DNS tunnel detection model, called FECC (Feature Extraction CNN and Clustering), to effectively detect various DNS Tunnel traffic. FECC takes the payload of the transport layer as input without preprocessing. The model uses a CNN-based module to extract features, then the homogeneity and exclusivity of features are evaluated based on the clustering method. FECC uses a cluster-based loss function to optimize the model parameters, which greatly improves the detection rate of the model for both classification tasks and unknown types of DNS tunnels, and further reduces the false positive rate. We have deployed a variety of DNS tunnel tools in the laboratory environ-ment and constructed multiple datasets for sufficient comparison experiments. The experimental results show that FECC is a very effective DNS tunnel detection model.(c) 2023 Elsevier Ltd. All rights reserved.