RORA: Robust Free-Text Rationale Evaluation
CoRR(2024)
摘要
Free-text rationales play a pivotal role in explainable NLP, bridging the
knowledge and reasoning gaps behind a model's decision-making. However, due to
the diversity of potential reasoning paths and a corresponding lack of
definitive ground truth, their evaluation remains a challenge. Existing
evaluation metrics rely on the degree to which a rationale supports a target
label, but we find these fall short in evaluating rationales that inadvertently
leak the labels. To address this problem, we propose RORA, a Robust free-text
Rationale evaluation against label leakage. RORA quantifies the new information
supplied by a rationale to justify the label. This is achieved by assessing the
conditional V-information with a
predictive family robust against leaky features that can be exploited by a
small model. RORA consistently outperforms existing approaches in evaluating
human-written, synthetic, or model-generated rationales, particularly
demonstrating robustness against label leakage. We also show that RORA aligns
well with human judgment, providing a more reliable and accurate measurement
across diverse free-text rationales.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要