Enhancing Length Extrapolation in Sequential Models with Pointer-Augmented Neural Memory
arxiv(2024)
摘要
We propose Pointer-Augmented Neural Memory (PANM) to help neural networks
understand and apply symbol processing to new, longer sequences of data. PANM
integrates an external neural memory that uses novel physical addresses and
pointer manipulation techniques to mimic human and computer symbol processing
abilities. PANM facilitates pointer assignment, dereference, and arithmetic by
explicitly using physical pointers to access memory content. Remarkably, it can
learn to perform these operations through end-to-end training on sequence data,
powering various sequential models. Our experiments demonstrate PANM's
exceptional length extrapolating capabilities and improved performance in tasks
that require symbol processing, such as algorithmic reasoning and Dyck language
recognition. PANM helps Transformer achieve up to 100
in compositional learning tasks and significantly better results in
mathematical reasoning, question answering and machine translation tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要