A Framework for Tuning Posterior Entropy in Unsupervised Learning

We present a general framework for unsupervised and semi-supervised learning containing a graded spectrum of Expectation Maximization (EM) algorithms. We call our framework Unified Expectation Maximization (UEM.) UEM allows us to tune the entropy of the inferred posterior distribution during the E-step to impact the quality of learning. Furthermore, UEM covers existing algorithms like standard EM and hard EM as well as constrained versions of EM such as Constraint-Driven Learning (Chang et al., 2007) and Posterior Regularization (Ganchev et al., 2010). Within the UEM framework, we can adapt the learning procedure to the data, initialization point, and supervision signals like constraints. Experiments on POS tagging, information extraction, and wordalignment show that often the best performing algorithm in the UEM family is a new algorithm that wasn’t available earlier, exhibiting the benefits of the UEM framework.