Penalized multiple distribution selection method for imbalanced data classification

In reality, the amount of data from different categories varies significantly, which results in learning bias towards prominent classes, hindering the overall classification performance. In this paper, by proving that traditional classification methods that use single softmax distribution are limited for modeling complex and imbalanced data, we propose a general Multiple Distribution Selection (MDS) method for imbalanced data classification. MDS employs a mixture distribution that is composed of a single softmax distribution and a set of degenerate distributions to model imbalanced data. Furthermore, a dynamic distribution selection method, based on L1 regularization, is also proposed to automatically determine the weights of distributions. In addition, the corresponding two-stage optimization algorithm is designed to estimate the parameters of models. Extensive experiments conducted on three widely used benchmark datasets (IMDB, ACE2005, 20NewsGroups) show that our proposed mixture method outperforms previous methods. Moreover, under highly imbalanced setting, our method achieves up to a 4.1 absolute F1 gain over high-performing baselines.