Maximum Entropy Model Correction in Reinforcement Learning
ICLR 2024(2023)
摘要
We propose and theoretically analyze an approach for planning with an
approximate model in reinforcement learning that can reduce the adverse impact
of model error. If the model is accurate enough, it accelerates the convergence
to the true value function too. One of its key components is the MaxEnt Model
Correction (MoCo) procedure that corrects the model's next-state distributions
based on a Maximum Entropy density estimation formulation. Based on MoCo, we
introduce the Model Correcting Value Iteration (MoCoVI) algorithm, and its
sampled-based variant MoCoDyna. We show that MoCoVI and MoCoDyna's convergence
can be much faster than the conventional model-free algorithms. Unlike
traditional model-based algorithms, MoCoVI and MoCoDyna effectively utilize an
approximate model and still converge to the correct value function.
更多查看译文
关键词
reinforcement learning,model-based reinforcement learning,maximum entropy,planning
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要