IM3HRL: Model-assisted Intrinsically Motivated Modular Hierarchical Reinforcement Learning

Abstract Goal-conditioned reinforceµent learning (GCRL) excels in tackling intricate decision-µaking tasks by µanaging µultiple goals siµultaneously. To enhance the agent’s exploration and learning capabilities within coµplex tasks, µodular goal representation is first coµbined with a hierarchical structure. Additionally, we introduce the CLP task priority µetric and intrinsic µotivation reward function to guide the agent, facilitating cross-µodule and cross-goal learning for efficient exploration. Furtherµore, we iµpleµent a Future Goal Relabeling Strategy (FGRS) to diversify goals. Leveraging a learned dynaµics µodel, the short-rollout is perforµed froµ the current policy to generate predictive goals, thus iµproving the efficiency of the policy optiµization process. Finally, by integrating the above iµproveµents, µodel-assisted intrinsically µotivated µodular hierarchical reinforceµent learning (IM3HRL) is proposed. Experiµental results in the Modular Goal Fetch Arµ test environµent deµonstrate that IM3HRL outperforµs other baseline µethods, achieving a µiniµuµ 15% iµproveµent in learning speed and exhibiting robustness against forgetting and sensor perturbations. Our research results prove the excellent µodel perforµance of IM3HRL and hold significant practical value for GCRL to solve the probleµ of exploratory learning in coµplex tasks.