Evolving Reservoirs for Meta Reinforcement Learning
CoRR(2023)
摘要
Animals often demonstrate a remarkable ability to adapt to their environments
during their lifetime. They do so partly due to the evolution of morphological
and neural structures. These structures capture features of environments shared
between generations to bias and speed up lifetime learning. In this work, we
propose a computational model for studying a mechanism that can enable such a
process. We adopt a computational framework based on meta reinforcement
learning as a model of the interplay between evolution and development. At the
evolutionary scale, we evolve reservoirs, a family of recurrent neural networks
that differ from conventional networks in that one optimizes not the weight
values but hyperparameters of the architecture: the later control macro-level
properties, such as memory and dynamics. At the developmental scale, we employ
these evolved reservoirs to facilitate the learning of a behavioral policy
through Reinforcement Learning (RL). Within an RL agent, a reservoir encodes
the environment state before providing it to an action policy. We evaluate our
approach on several 2D and 3D simulated environments. Our results show that the
evolution of reservoirs can improve the learning of diverse challenging tasks.
We study in particular three hypotheses: the use of an architecture combining
reservoirs and reinforcement learning could enable (1) solving tasks with
partial observability, (2) generating oscillatory dynamics that facilitate the
learning of locomotion tasks, and (3) facilitating the generalization of
learned behaviors to new tasks unknown during the evolution phase.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要