Learning Nearest-Neighbor Quantizers from Labeled Data by Information Loss Minimization

This paper proposes a technique for jointly quan- tizing continuous features and the posterior dis- tributions of their class labels based on minimiz- ing empirical information loss, such that the in- dex K of the quantizer region to which a given feature X is assigned approximates a sufficient statistic for its class label Y. We derive an al- ternating minimization procedure for simultane- ously learning codebooks in the Euclidean fea- ture space and in the simplex of posterior class distributions. The resulting quantizer can be used to encode unlabeled points outside the training set and to predict their posterior class distribu- tions, and has an elegant interpretation in terms of universal lossless coding. The promise of our method is demonstrated for the application of learning discriminative visual vocabularies for bag-of-features image classification.