On the Tip of the Tongue: Analyzing Conceptual Representation in Large Language Models with Reverse-Dictionary Probe
CoRR(2024)
摘要
Probing and enhancing large language models' reasoning capacity remains a
crucial open question. Here we re-purpose the reverse dictionary task as a case
study to probe LLMs' capacity for conceptual inference. We use in-context
learning to guide the models to generate the term for an object concept implied
in a linguistic description. Models robustly achieve high accuracy in this
task, and their representation space encodes information about object
categories and fine-grained features. Further experiments suggest that the
conceptual inference ability as probed by the reverse-dictionary task predicts
model's general reasoning performance across multiple benchmarks, despite
similar syntactic generalization behaviors across models. Explorative analyses
suggest that prompting LLMs with description⇒word examples may
induce generalization beyond surface-level differences in task construals and
facilitate models on broader commonsense reasoning problems.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要