Linear attention is (maybe) all you need (to understand transformer optimization)
arxiv(2023)
摘要
Transformer training is notoriously difficult, requiring a careful design of
optimizers and use of various heuristics. We make progress towards
understanding the subtleties of training Transformers by carefully studying a
simple yet canonical linearized shallow Transformer model. Specifically, we
train linear Transformers to solve regression tasks, inspired by J. von Oswald
et al. (ICML 2023), and K. Ahn et al. (NeurIPS 2023). Most importantly, we
observe that our proposed linearized models can reproduce several prominent
aspects of Transformer training dynamics. Consequently, the results obtained in
this paper suggest that a simple linearized Transformer model could actually be
a valuable, realistic abstraction for understanding Transformer optimization.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要