Deep Reinforcement Learning for Simulated Autonomous Driving

This research explores deep Q-learning for autonomous driving in The Open Racing Car Simulator (TORCS). Using the TensorFlow and Keras software frameworks, we train fully-connected deep neural networks that are able to autonomously drive across a diverse range of track geometries. An initial proof-of-concept of classical Q-learning was implemented in Flappy Bird. A reward function promoting longitudinal velocity while penalizing transverse velocity and divergence from the track center is used to train the agent. To validate learning, the research analyzes the reward function parameters of the models over two validation tracks and qualitatively assesses driving stability. A video of the learned agent driving in TORCS can be found online: https://www.dropbox.com/sh/ b4623soznsjmp12/AAA1UD8_oaa94FgFC6eyxReya?dl=0. 1 Background and Introduction Our research objective is to apply reinforcement learning to train an agent that can autonomously race in TORCS (The Open Racing Car Simulator) [1, 2]. TORCS is a modern simulation platform used for research in control systems and autonomous driving. Training an autonomous driving system in simulation offers a number of advantages, as applying supervised learning on real-world training data can be expensive and requires substantial amounts of labor for driving and labeling. Furthermore, a simulation is a safe, efficient, and cost-effective way to identify and test failure cases (e.g. collision events) of safety-critical control systems, without having to sacrifice physical hardware. Accurate simulation platforms provide robust environments for training reinforcement learning models which can then be applied to real-world settings through transfer learning. Figure 1: TORCS simulation environment Deep reinforcement learning has been applied with great success to a wide variety of game play scenarios. These scenarios are notable for their high-dimensional state spaces that border on real-world complexity. In particular, the deep Q-network (DQN) algorithm introduced by Google’s DeepMind team in 2015 has been shown to successfully learn policies for agents relying on complex input spaces [3]. Prior to the introduction of DQN, applicability of reinforcement learning agents was limited to hand-crafted feature selection or low-dimensional state spaces. To successfully apply reinforcement learning to situations with real-world complexity, agents must derive efficient representations of high-dimensional sensory inputs and use these features to to generalize past observations to future samples. The DQN algorithm was shown to surpass the performance of all previous algorithms and achieve a performance level comparable to that of a professional games tester across 49 Atari games. 1.1 Relevant Work There are two notable, distinct past approaches to training autonomous driving agents in TORCS. In 2015, the DeepDriving model applied deep supervised learning and convolutional neural networks (CNN) to learn a driving policy [4]. Other research,