Evaluation and Optimization of learning-based DNS over HTTPS Traffic Classification.

A fundamental piece of the TCP/IP model is the Domain Name System (DNS) that looks like Internet’s address book. DNS does a simple but crucial job by converting readable names (e.g., url) into destination IPs. However, since the entire process is handled without a robust security design, the DNS is vulnerable to diverse attacks, which reduce availability or stability of any service. Recently, DNS over HTTPS (DoH), is proposed to encrypt the different queries/answers of DNS, making them indistinguishable from HTTPS and thus increasing DNS protection. Although, DoH has its security issues as well. Face to newly sophisticated attacks, it is crucial today for any operator or enterprise to detect the malicious DoH traffic automatically. In this paper, we implement a methodology to investigate the application of Machine and Deep Learning approaches for the classification and detection in real-time of the malicious DoH traffic. We evaluate and optimize the performance of several models by tuning the hyperparameters and compare them with respect to four performance metrics: Precision, Accuracy, Recall and F1-Score. Extensive simulation results show that Random Forest and Decision Tree models outperform other considered Machine and Deep Learning models (i.e., KNN, 1D CNN, 2D CNN and LSTM) with Precision and Recall more that 99% and 98% respectively with hyperparameter optimization.