BP(λ): Online Learning via Synthetic Gradients
CoRR(2024)
摘要
Training recurrent neural networks typically relies on backpropagation
through time (BPTT). BPTT depends on forward and backward passes to be
completed, rendering the network locked to these computations before loss
gradients are available. Recently, Jaderberg et al. proposed synthetic
gradients to alleviate the need for full BPTT. In their implementation
synthetic gradients are learned through a mixture of backpropagated gradients
and bootstrapped synthetic gradients, analogous to the temporal difference (TD)
algorithm in Reinforcement Learning (RL). However, as in TD learning, heavy use
of bootstrapping can result in bias which leads to poor synthetic gradient
estimates. Inspired by the accumulate TD(λ) in RL, we propose
a fully online method for learning synthetic gradients which avoids the use of
BPTT altogether: accumulate BP(λ). As in accumulate
TD(λ), we show analytically that accumulate
BP(λ) can control the level of bias by using a mixture of
temporal difference errors and recursively defined eligibility traces. We next
demonstrate empirically that our model outperforms the original implementation
for learning synthetic gradients in a variety of tasks, and is particularly
suited for capturing longer timescales. Finally, building on recent work we
reflect on accumulate BP(λ) as a principle for learning in
biological circuits. In summary, inspired by RL principles we introduce an
algorithm capable of bias-free online learning via synthetic gradients.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要