In Defense of Softmax Parametrization for Calibrated and Consistent Learning to Defer
NeurIPS(2023)
摘要
Enabling machine learning classifiers to defer their decision to a downstream
expert when the expert is more accurate will ensure improved safety and
performance. This objective can be achieved with the learning-to-defer
framework which aims to jointly learn how to classify and how to defer to the
expert. In recent studies, it has been theoretically shown that popular
estimators for learning to defer parameterized with softmax provide unbounded
estimates for the likelihood of deferring which makes them uncalibrated.
However, it remains unknown whether this is due to the widely used softmax
parameterization and if we can find a softmax-based estimator that is both
statistically consistent and possesses a valid probability estimator. In this
work, we first show that the cause of the miscalibrated and unbounded estimator
in prior literature is due to the symmetric nature of the surrogate losses used
and not due to softmax. We then propose a novel statistically consistent
asymmetric softmax-based surrogate loss that can produce valid estimates
without the issue of unboundedness. We further analyze the non-asymptotic
properties of our method and empirically validate its performance and
calibration on benchmark datasets.
更多查看译文
关键词
softmax parametrization,learning,calibrated
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要