Automatic recovering the number k of clusters in the data by active query selection

ABSTRACTOne common parameter of many clustering algorithms is the number k of clusters required to partition the data. This is the case of k-means, one of the most popular clustering algorithms from the Machine Learning literature, and its variants. Indeed, when clustering a dataset, the right number k of clusters to use is often not obvious, and choosing k automatically is a hard algorithmic problem. In this context, one popular procedure used to estimate the number of clusters present in a dataset is to run the clustering algorithm multiple times varying the number of clusters and one of the solutions obtained is chosen based on a given internal clustering validation measure (e.g., silhouette coefficient). This process can be very time consuming as the clustering algorithm must be run several times. In this paper we present some strategies that can be integrated to constrained clustering methods so as to recover automatically the number k of clusters. The idea is that constrained clustering algorithms allow one to incorporate prior information such as if some pairs of instances from the dataset must be placed in the same cluster or not. Still in the context of constrained clustering algorithms, in order to improve the quality of the pairwise constraints given as input to the algorithm, there are approaches that use active methods for pairwise constraint selection. In our proposed strategies we make use of the prior information provided by the pairwise constraints and the concept of neighborhood from active methods not only to build a partition, but also to identify automatically the number k of clusters in the data. Based on nine datasets, we show experimentally that our strategies, besides automatically recovering the number of clusters in the data, lead to the generation of partitions having high quality when evaluated by indicators of clustering performance such as the adjusted Rand index.