Reinforcement Learning for Adaptive MCMC

An informal observation, made by several authors, is that the adaptive designof a Markov transition kernel has the flavour of a reinforcement learning task.Yet, to-date it has remained unclear how to actually exploit modernreinforcement learning technologies for adaptive MCMC. The aim of this paper isto set out a general framework, called Reinforcement LearningMetropolis–Hastings, that is theoretically supported and empiricallyvalidated. Our principal focus is on learning fast-mixing Metropolis–Hastingstransition kernels, which we cast as deterministic policies and optimise via apolicy gradient. Control of the learning rate provably ensures conditions forergodicity are satisfied. The methodology is used to construct a gradient-freesampler that out-performs a popular gradient-free adaptive Metropolis–Hastingsalgorithm on ≈ 90 % of tasks in the PosteriorDB benchmark.