Cluster validity index for irregular clustering results

Different clustering algorithms with different parameter settings can produce various partitions on the input data. Without the priori knowledge, it is difficult for users to select the proper clustering algorithm and the parameters for the specific data in advance. Therefore, the cluster validity index (CVI) is crucial to help select the best partition that fits the underlying structure of the data. However, most existing CVIs (including some recent ones that designed for complex partitions) have strong assumptions. They only work well for partitions where clusters are spherically distributed, with similar sizes and densities, and with large separation distances. In complicated situations where irregular clustering results (i.e., clustering results having clusters in arbitrary shapes, different sizes and densities, and with small separation distances) exist, they usually fail to find the best fitting partition. Focusing on the insufficiencies of the existing CVIs (designed for hard clustering), this paper presents a new index that helps to find the best partition produced by hard clustering algorithms when irregular clustering results exist. The proposed index uses the density changes inside a cluster, and the density changes from the inner to the inter-cluster regions of the cluster to evaluate its quality. Experiments are implemented on both real and synthetic datasets. 11 other CVIs including some well known as well as recently proposed ones are also tested for comparison. Experimental results are given to demonstrate the effectiveness of the new index.