Meta Actor-Critic Framework for Multi-Agent Reinforcement Learning

In recent years, multi-agent reinforcement learning has received sustained attention in the last few years. The typical Actor-Critic methods learn mappings directly from observation to action without understanding the tasks themselves. In this paper, we present a meta actor-critic framework for meta-actor critique based on observational learning processes and the additional loss of meta-learning actors to accelerate and improve multi-agent learning across agents' experiences. Within our framework, all agents are deliberately designed to share the same meta-critic loss to achieve the optimum actor learning progress. Meanwhile, by minimizing the loss of meta-actors, the meta actor learns the features of the meta-observation, leading to better actions. We implemented the MADDPG and MATD3 algorithms in our proposed framework and empirically demonstrated the superiority of our framework on two kinds of multi-agent tasks. In addition, the framework can be flexibly incorporated into various contemporary multi-agent Actor-Critic methods.