How to Use Your Model: Model-Based Reinforcement Learning with Model-Free Policy Optimization

Model-based reinforcement learning (MBRL) methods have demonstrated superior sample efficiency relative to their model-free counterparts, largely attributable to the leverage of learned models. Despite these advancements, the effective application of these learned models still be challenging, largely due to the intricate interdependence between model learning and policy optimization. This paper offers a comprehensive analysis of MBRL, in addition to establishing a discrepancy bound for Dyna-style algorithms in deterministic environments. Stemming from this analysis, we propose an innovative schema, Model-Based Reinforcement Learning with Model-Free Policy Optimization (MBMFPO). This schema integrates model-free policy optimization into the MBRL framework, in conjunction with some additional techniques. Experimental results on various continuous control tasks reveal that MBMFPO can significantly enhance sample efficiency and final performance compared to the baseline methods. Furthermore, auxiliary ablation studies provide robust support for the effectiveness of each individual component within the MBMFPO schema.