How often are errors in natural language reasoning due to paraphrastic variability?
arxiv(2024)
摘要
Large language models have been shown to behave inconsistently in response to
meaning-preserving paraphrastic inputs. At the same time, researchers evaluate
the knowledge and reasoning abilities of these models with test evaluations
that do not disaggregate the effect of paraphrastic variability on performance.
We propose a metric for evaluating the paraphrastic consistency of natural
language reasoning models based on the probability of a model achieving the
same correctness on two paraphrases of the same problem. We mathematically
connect this metric to the proportion of a model's variance in correctness
attributable to paraphrasing. To estimate paraphrastic consistency, we collect
ParaNLU, a dataset of 7,782 human-written and validated paraphrased reasoning
problems constructed on top of existing benchmark datasets for defeasible and
abductive natural language inference. Using ParaNLU, we measure the
paraphrastic consistency of several model classes and show that consistency
dramatically increases with pretraining but not finetuning. All models tested
exhibited room for improvement in paraphrastic consistency.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要