I Think, Therefore I am: Benchmarking Awareness of Large Language Models Using AwareBench
arxiv(2024)
摘要
Do large language models (LLMs) exhibit any forms of awareness similar to
humans? In this paper, we introduce AwareBench, a benchmark designed to
evaluate awareness in LLMs. Drawing from theories in psychology and philosophy,
we define awareness in LLMs as the ability to understand themselves as AI
models and to exhibit social intelligence. Subsequently, we categorize
awareness in LLMs into five dimensions, including capability, mission, emotion,
culture, and perspective. Based on this taxonomy, we create a dataset called
AwareEval, which contains binary, multiple-choice, and open-ended questions to
assess LLMs' understandings of specific awareness dimensions. Our experiments,
conducted on 13 LLMs, reveal that the majority of them struggle to fully
recognize their capabilities and missions while demonstrating decent social
intelligence. We conclude by connecting awareness of LLMs with AI alignment and
safety, emphasizing its significance to the trustworthy and ethical development
of LLMs. Our dataset and code are available at
https://github.com/HowieHwong/Awareness-in-LLM.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要