Natural-Gradient Stochastic Variational Inference for Non-Conjugate Structured Variational Autoencoder

We propose a new method for amortized inference in graphical models that contain deep generative models. Our method generalizes existing approaches to a larger class of models where the graphical model can contain non-conjugate components. Our main contribution is the proposal of structured recognition models that preserve the correlations between all local variables. For this general class of models, we derive a scalable inference method that employs natural-gradient updates and can be implemented by reusing existing software for graphical models and deep models.