Granular Change Accuracy: A More Accurate Performance Metric for Dialogue State Tracking
arxiv(2024)
摘要
Current metrics for evaluating Dialogue State Tracking (DST) systems exhibit
three primary limitations. They: i) erroneously presume a uniform distribution
of slots throughout the dialog, ii) neglect to assign partial scores for
individual turns, iii) frequently overestimate or underestimate performance by
repeatedly counting the models' successful or failed predictions. To address
these shortcomings, we introduce a novel metric: Granular Change Accuracy
(GCA). GCA focuses on evaluating the predicted changes in dialogue state over
the entire dialogue history. Benchmarking reveals that GCA effectively reduces
biases arising from distribution uniformity and the positioning of errors
across turns, resulting in a more precise evaluation. Notably, we find that
these biases are particularly pronounced when evaluating few-shot or zero-shot
trained models, becoming even more evident as the model's error rate increases.
Hence, GCA offers significant promise, particularly for assessing models
trained with limited resources. Our GCA implementation is a useful addition to
the pool of DST metrics.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要