Hierarchical Multi-Agent Deep Reinforcement Learning To Develop Long-Term Coordination

Multi-agent systems need to communicate to coordinate a shared task. We show that a recurrent neural network (RNN) can learn a communication protocol for coordination, even if the actions to coordinate lie outside of the communication range. We also show that a single RNN is unable to do this if there is an independent action sequence necessary before the coordinated action can be executed. We propose a hierarchical deep reinforcement learning model for multi-agent systems that separates the communication and coordination task from the action picking through a hierarchical policy. As a testbed, we propose the Dungeon Lever Game and we extend the Differentiable Inter-Agent Learning (DIAL) framework [ 3]. First we prove that the agents need a hierarchical policy to learn communication and actions and, second, we present results from our successful model of the Dungeon Lever Game.