ChatGPT Rates Natural Language Explanation Quality Like Humans: But on Which Scales?
arxiv(2024)
摘要
As AI becomes more integral in our lives, the need for transparency and
responsibility grows. While natural language explanations (NLEs) are vital for
clarifying the reasoning behind AI decisions, evaluating them through human
judgments is complex and resource-intensive due to subjectivity and the need
for fine-grained ratings. This study explores the alignment between ChatGPT and
human assessments across multiple scales (i.e., binary, ternary, and 7-Likert
scale). We sample 300 data instances from three NLE datasets and collect 900
human annotations for both informativeness and clarity scores as the text
quality measurement. We further conduct paired comparison experiments under
different ranges of subjectivity scores, where the baseline comes from 8,346
human annotations. Our results show that ChatGPT aligns better with humans in
more coarse-grained scales. Also, paired comparisons and dynamic prompting
(i.e., providing semantically similar examples in the prompt) improve the
alignment. This research advances our understanding of large language models'
capabilities to assess the text explanation quality in different configurations
for responsible AI development.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要