Reasoning in Latent Space with the Bayes Factor

We propose a novel way to answer probabilistic queries that span multiple datapoints using deep generative models that enforce a separation of representation learning and reasoning. We give a construction for a query-conditional emph{latent reasoning network}, a parametric distribution over the latent space of a deep generative model. The reasoning network emph{learns to compile} the posterior distributions of the set of query points into a single distribution via the use of novel neural architecture. With labeled data, we show how to learn the latent reasoning network using a max-margin learning algorithm. The score function we use is given by the Bayes factor of points given the query versus the points unconditionally. We show how to differentiate emph{through} the latent reasoning model to emph{fine-tune} the latent space of the deep generative model. We show how the algorithm may be used to focus the data variations captured by the generative model and to build new models for few-shot learning that perform comparably to several recently proposed state-of-the-art methods.