A Unified Energy-Based Framework for Unsupervised Learning

We introduce a view of unsupervised learn- ing that integrates probabilistic and non- probabilistic methods for clustering, dimen- sionality reduction, and feature extraction in a unified framework. In this framework, an energy function associates low energies to in- put points that are similar to training sam- ples, and high energies to unobserved points. Learning consists in minimizing the energies of training samples while ensuring that the energies of unobserved ones are higher. Some traditional methods construct the architec- ture so that only a small number of points can have low energy, while other methods explicitly "pull up" on the energies of unob- served points. In probabilistic methods the energy of unobserved points is pulled by min- imizing the log partition function, an expen- sive, and sometimes intractable process. We explore different and more efficient methods using an energy-based approach. In particu- lar, we show that a simple solution is to re- strict the amount of information contained in codes that represent the data. We demon- strate such a method by training it on natu- ral image patches and by applying to image denoising.