Entropy-regularized Diffusion Policy with Q-Ensembles for Offline Reinforcement Learning
CoRR(2024)
摘要
This paper presents advanced techniques of training diffusion policies for
offline reinforcement learning (RL). At the core is a mean-reverting stochastic
differential equation (SDE) that transfers a complex action distribution into a
standard Gaussian and then samples actions conditioned on the environment state
with a corresponding reverse-time SDE, like a typical diffusion policy. We show
that such an SDE has a solution that we can use to calculate the log
probability of the policy, yielding an entropy regularizer that improves the
exploration of offline datasets. To mitigate the impact of inaccurate value
functions from out-of-distribution data points, we further propose to learn the
lower confidence bound of Q-ensembles for more robust policy improvement. By
combining the entropy-regularized diffusion policy with Q-ensembles in offline
RL, our method achieves state-of-the-art performance on most tasks in D4RL
benchmarks. Code is available at
\href{https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble}{https://github.com/ruoqizzz/Entropy-Regularized-Diffusion-Policy-with-QEnsemble}.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要