Sampling from Gaussian Process Posteriors using Stochastic Gradient Descent
NeurIPS(2023)
摘要
Gaussian processes are a powerful framework for quantifying uncertainty and
for sequential decision-making but are limited by the requirement of solving
linear systems. In general, this has a cubic cost in dataset size and is
sensitive to conditioning. We explore stochastic gradient algorithms as a
computationally efficient method of approximately solving these linear systems:
we develop low-variance optimization objectives for sampling from the posterior
and extend these to inducing points. Counterintuitively, stochastic gradient
descent often produces accurate predictions, even in cases where it does not
converge quickly to the optimum. We explain this through a spectral
characterization of the implicit bias from non-convergence. We show that
stochastic gradient descent produces predictive distributions close to the true
posterior both in regions with sufficient data coverage, and in regions
sufficiently far away from the data. Experimentally, stochastic gradient
descent achieves state-of-the-art performance on sufficiently large-scale or
ill-conditioned regression tasks. Its uncertainty estimates match the
performance of significantly more expensive baselines on a large-scale Bayesian
optimization task.
更多查看译文
关键词
gaussian process posteriors,stochastic gradient
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要