What if you said that differently?: How Explanation Formats Affect Human Feedback Efficacy and User Perception
arxiv(2023)
摘要
Eliciting feedback from end users of NLP models can be beneficial for
improving models. However, how should we present model responses to users so
they are most amenable to be corrected from user feedback? Further, what
properties do users value to understand and trust responses? We answer these
questions by analyzing the effect of rationales (or explanations) generated by
QA models to support their answers. We specifically consider decomposed QA
models that first extract an intermediate rationale based on a context and a
question and then use solely this rationale to answer the question. A rationale
outlines the approach followed by the model to answer the question. Our work
considers various formats of these rationales that vary according to
well-defined properties of interest. We sample rationales from language models
using few-shot prompting for two datasets, and then perform two user studies.
First, we present users with incorrect answers and corresponding rationales in
various formats and ask them to provide natural language feedback to revise the
rationale. We then measure the effectiveness of this feedback in patching these
rationales through in-context learning. The second study evaluates how well
different rationale formats enable users to understand and trust model answers,
when they are correct. We find that rationale formats significantly affect how
easy it is (1) for users to give feedback for rationales, and (2) for models to
subsequently execute this feedback. In addition, formats with attributions to
the context and in-depth reasoning significantly enhance user-reported
understanding and trust of model outputs.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要