Evolutionary Computation and the Reinforcement Learning Problem

Evolution by natural selection has built a vast array of highly efficient lifelong learningLifelong learning organisms, as evidenced by the spectacular diversityDiversity of species that rapidly adapt to environmental change and acquire new problem-solving skills through experience. Reinforcement Learning (RL) is a machine learning problem in which an agent must learn how to map situations to actions in an unknown world in order to maximise the sum of future rewards. There are no labelled examples of situation $$\rightarrow $$ action mappingsMapping to learn from and we assume that no model of environment dynamics is available. As such, learning requires active trial-and-error interaction with the world. Evolutionary Reinforcement Learning (EvoRL), the application of evolutionary computation in RL, models this search process at multiple time scales: individual learning during the lifetime of an agent (i.e., operant conditioning) and population-wide learning through natural selection. Both modes of adaptation are wildly creative and fundamental to natural systems. This chapter discusses how EvoRL addresses some critical challenges in RL including the computational costComputational cost of extended interactions, the temporal credit assignmentCredit assignment problem, partial-observability of state, nonstationary and multi-task environments, transfer learningTransfer learning, and hierarchical problem decomposition. In each case, the unique potential of EvoRL is highlighted in parallel with open challenges and research opportunities.