Matrix-variate Variational Auto-Encoder with Applications to Image Process

Variational Auto-Encoder (VAE) is an important probabilistic technology to model 1D vectorial data. However, when applying VAE model to 2D image, vectorization is necessary. Vectorization process may lead to dimension curse and lose valuable spatial information. To avoid these problems, we propose a novel VAE model based on matrix variables named as Matrix-variate Variational Auto-Encoder (MVVAE). In this model, input, hidden and latent variables are all in matrix form, therefore inherent spatial structure of 2D images can be maintained and utilized better. Especially, the latent variable is assumed to follow matrix Gaussian distribution which is more suitable for describing 2D images. To solve the weights and the posterior of latent variable, the variational inference process is given. The experiments are designed for three real-world application: reconstruction, denoising and completion. The experimental results demonstrate that MVVAE shows better performance than VAE and other probabilistic methods for modeling and processing 2D data.