Evaluation of passive OS fingerprinting methods using TCP/IP fields

An important part of network management is to keep knowledge about the connected devices. One of the tools that can provide such information in real-time is passive OS fingerprinting, in particular the method based on analyzing values of specific TCP/IP headers. The state-of-the-art approach is to use machine learning to create such OS classifier. In this paper, we focus on the evaluation of this approach from several perspectives. We took two existing public datasets and created a new one from our network and trained machine learning models to classify the 4 most common operation system families based on selected TCP/IP fields. We compare different models, discuss the need to round TTL values to avoid over-fitting, and test the transferability of models trained on data from different networks. Although TCP/IP-related characteristics of individual operating systems should be independent on where the device is located, our experiments show that a model trained in one network performs much worse in another one, making model creation and deployment more difficult in practice. A good solution may be to combine data from multiple networks. A model trained on a combination of all three datasets exhibited the best results on average across the datasets.