On Limitations of the Transformer Architecture
CoRR(2024)
摘要
What are the root causes of hallucinations in large language models (LLMs)?
We use Communication Complexity to prove that the Transformer layer is
incapable of composing functions (e.g., identify a grandparent of a person in a
genealogy) if the domains of the functions are large enough; we show through
examples that this inability is already empirically present when the domains
are quite small. We also point out that several mathematical tasks that are at
the core of the so-called compositional tasks thought to be hard for LLMs are
unlikely to be solvable by Transformers, for large enough instances and
assuming that certain well accepted conjectures in the field of Computational
Complexity are true.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要