xLSTM: Extended Long Short-Term Memory
arxiv(2024)
摘要
In the 1990s, the constant error carousel and gating were introduced as the
central ideas of the Long Short-Term Memory (LSTM). Since then, LSTMs have
stood the test of time and contributed to numerous deep learning success
stories, in particular they constituted the first Large Language Models (LLMs).
However, the advent of the Transformer technology with parallelizable
self-attention at its core marked the dawn of a new era, outpacing LSTMs at
scale. We now raise a simple question: How far do we get in language modeling
when scaling LSTMs to billions of parameters, leveraging the latest techniques
from modern LLMs, but mitigating known limitations of LSTMs? Firstly, we
introduce exponential gating with appropriate normalization and stabilization
techniques. Secondly, we modify the LSTM memory structure, obtaining: (i) sLSTM
with a scalar memory, a scalar update, and new memory mixing, (ii) mLSTM that
is fully parallelizable with a matrix memory and a covariance update rule.
Integrating these LSTM extensions into residual block backbones yields xLSTM
blocks that are then residually stacked into xLSTM architectures. Exponential
gating and modified memory structures boost xLSTM capabilities to perform
favorably when compared to state-of-the-art Transformers and State Space
Models, both in performance and scaling.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要