AbsPyramid: Benchmarking the Abstraction Ability of Language Models with a Unified Entailment Graph
arxiv(2023)
摘要
Cognitive research indicates that abstraction ability is essential in human
intelligence, which remains under-explored in language models. In this paper,
we present AbsPyramid, a unified entailment graph of 221K textual descriptions
of abstraction knowledge. While existing resources only touch nouns or verbs
within simplified events or specific domains, AbsPyramid collects abstract
knowledge for three components of diverse events to comprehensively evaluate
the abstraction ability of language models in the open domain. Experimental
results demonstrate that current LLMs face challenges comprehending abstraction
knowledge in zero-shot and few-shot settings. By training on our rich
abstraction knowledge, we find LLMs can acquire basic abstraction abilities and
generalize to unseen events. In the meantime, we empirically show that our
benchmark is comprehensive to enhance LLMs across two previous abstraction
tasks.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要