Why Do Animals Need Shaping? A Theory of Task Composition and Curriculum Learning
arxiv(2024)
摘要
Diverse studies in systems neuroscience begin with extended periods of
training known as 'shaping' procedures. These involve progressively studying
component parts of more complex tasks, and can make the difference between
learning a task quickly, slowly or not at all. Despite the importance of
shaping to the acquisition of complex tasks, there is as yet no theory that can
help guide the design of shaping procedures, or more fundamentally, provide
insight into its key role in learning. Modern deep reinforcement learning
systems might implicitly learn compositional primitives within their multilayer
policy networks. Inspired by these models, we propose and analyse a model of
deep policy gradient learning of simple compositional reinforcement learning
tasks. Using the tools of statistical physics, we solve for exact learning
dynamics and characterise different learning strategies including primitives
pre-training, in which task primitives are studied individually before learning
compositional tasks. We find a complex interplay between task complexity and
the efficacy of shaping strategies. Overall, our theory provides an analytical
understanding of the benefits of shaping in a class of compositional tasks and
a quantitative account of how training protocols can disclose useful task
primitives, ultimately yielding faster and more robust learning.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要