Efficient Deep Learning with Decorrelated Backpropagation
arxiv(2024)
摘要
The backpropagation algorithm remains the dominant and most successful method
for training deep neural networks (DNNs). At the same time, training DNNs at
scale comes at a significant computational cost and therefore a high carbon
footprint. Converging evidence suggests that input decorrelation may speed up
deep learning. However, to date, this has not yet translated into substantial
improvements in training efficiency in large-scale DNNs. This is mainly caused
by the challenge of enforcing fast and stable network-wide decorrelation. Here,
we show for the first time that much more efficient training of very deep
neural networks using decorrelated backpropagation is feasible. To achieve this
goal we made use of a novel algorithm which induces network-wide input
decorrelation using minimal computational overhead. By combining this algorithm
with careful optimizations, we obtain a more than two-fold speed-up and higher
test accuracy compared to backpropagation when training a 18-layer deep
residual network. This demonstrates that decorrelation provides exciting
prospects for efficient deep learning at scale.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要