Multi-Intersection Management for Connected Autonomous Vehicles by Reinforcement Learning

The rapid development of connected autonomous vehicles (CAVs) makes it foreseeable that CAVs will dominate future road traffic. To manage CAV traffic, researchers developed a revolutionary paradigm, which uses intelligent intersection managers (IMs) for a finer-grained control of CAVs' cruising at intersections than traditional traffic lights. However, existing IM-based methods mostly focus on optimizing the single-intersection CAV traffic efficiency, without solving the fundamental problem of maximizing the global efficiency of a multi-intersection road network. Therefore, we address such problem by proposing a system architecture that decomposes each IM into an oracle and a valve, where the oracle ensures safe and efficient crossing at individual intersections, and the valve selects some of the approaching CAVs for the oracle to control and postpones the crossing of the unselected ones. We further focus on distributed decision making for the valves, and propose a multi-agent reinforcement learning framework, spatial-aware multi-agent actor-credit (SMAC). Specifically, SMAC integrates a novel credit assignment method that captures agents' spatially decaying influences to stimulate agent cooperation, and a novel graph convolutional mixing network to capture the graph-structured inter-agent relationships in a road network. We conduct extensive experiments on three traffic flow datasets, and show that SMAC outperforms state-of-the-art baselines.