A cold-start-free reinforcement learning approach for traffic signal control

Typical reinforcement learning (RL) requires a huge amount of data before achieving an acceptable result, and its performance can be rather poor during initial interacting process. Sample inefficiency and cold-start phenomenon of RL limits its feasibility in a range of real-world applications such as traffic signal control (TSC). On the other hand, a large amount of data on TSC can be accumulated by various model-based controllers (MBCs) rooted in traffic engineering. In this context, we propose a new RL approach which can avoid the appearance of cold starts by taking advantage of MBC experiences. First, three frameworks of joint utilization of RL and MBC are summarized for TSC, and staged framework is considered to have the edge over the other two. Then, a staged noisy-net prioritized dueling double deep Q-network (NPDD-DQN) is described in detail for TSC, where MBC experiences are used in both pre-training and online training processes. Experimental evaluation demonstrates that staged NPDD-DQN can achieve a boost in initial performance as compared to pure NPDD-DQN that does not utilize any control experiences, and learn to improve final performance beyond the underlying MBC. The effectiveness of the proposed method opens up the possibility of real implementation of RL in TSC.