Premise Order Matters in Reasoning with Large Language Models
CoRR(2024)
摘要
Large language models (LLMs) have accomplished remarkable reasoning
performance in various domains. However, in the domain of reasoning tasks, we
discover a frailty: LLMs are surprisingly brittle to the ordering of the
premises, despite the fact that such ordering does not alter the underlying
task. In particular, we observe that LLMs achieve the best performance when the
premise order aligns with the context required in intermediate reasoning steps.
For example, in deductive reasoning tasks, presenting the premises in the same
order as the ground truth proof in the prompt (as opposed to random ordering)
drastically increases the model's accuracy. We first examine the effect of
premise ordering on deductive reasoning on a variety of LLMs, and our
evaluation shows that permuting the premise order can cause a performance drop
of over 30
examine the ordering effect for mathematical problem-solving, and we again
observe a significant drop in accuracy, relative to the original GSM8K
benchmark.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要