Learning multi-scale local conditional probability models of images

Deep neural networks can learn powerful prior probability models for images, as evidenced by high-quality synthesis results achieved with VAEs, GANs, and recent score-based diffusion methods. But these models are implicit, and the means by which these networks capture complex global statistical structure, apparently without suffering from the curse of dimensionality, remain a mystery. To study this, we generalize a multi-scale model class motivated by the renormalization group of theoretical physics. It circumvents the curse of dimensionality by assuming Markov structure of multi-scale wavelet coefficients conditioned on coarser scale coefficients. We parameterize the model using conditional convolutional neural networks with local receptive fields, which enforce stationary Markov properties. We test the capabilities of the model on a dataset of face images, which are highly non-stationary and contain long-range geometric structures. Remarkably, denoising, super-resolution, and image synthesis results demonstrate that these structures are well-captured, using significantly smaller neighborhoods than required by a CNN operating in the pixel domain.