Predictive Churn with the Set of Good Models
CoRR(2024)
摘要
Machine learning models in modern mass-market applications are often updated
over time. One of the foremost challenges faced is that, despite increasing
overall performance, these updates may flip specific model predictions in
unpredictable ways. In practice, researchers quantify the number of unstable
predictions between models pre and post update – i.e., predictive churn. In
this paper, we study this effect through the lens of predictive multiplicity –
i.e., the prevalence of conflicting predictions over the set of near-optimal
models (the Rashomon set). We show how traditional measures of predictive
multiplicity can be used to examine expected churn over this set of prospective
models – i.e., the set of models that may be used to replace a baseline model
in deployment. We present theoretical results on the expected churn between
models within the Rashomon set from different perspectives. And we characterize
expected churn over model updates via the Rashomon set, pairing our analysis
with empirical results on real-world datasets – showing how our approach can
be used to better anticipate, reduce, and avoid churn in consumer-facing
applications. Further, we show that our approach is useful even for models
enhanced with uncertainty awareness.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要