Deep autoregressive density nets vs neural ensembles for model-based offline reinforcement learning
ICLR 2023(2024)
摘要
We consider the problem of offline reinforcement learning where only a set of
system transitions is made available for policy optimization. Following recent
advances in the field, we consider a model-based reinforcement learning
algorithm that infers the system dynamics from the available data and performs
policy optimization on imaginary model rollouts. This approach is vulnerable to
exploiting model errors which can lead to catastrophic failures on the real
system. The standard solution is to rely on ensembles for uncertainty
heuristics and to avoid exploiting the model where it is too uncertain. We
challenge the popular belief that we must resort to ensembles by showing that
better performance can be obtained with a single well-calibrated autoregressive
model on the D4RL benchmark. We also analyze static metrics of model-learning
and conclude on the important model properties for the final performance of the
agent.
更多查看译文
关键词
Offline reinforcement learning,batch reinforcement learning,ensemble,autoregressive,D4RL,model-based
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要