Does AI help humans make better decisions? A methodological framework for experimental evaluation
arxiv(2024)
摘要
The use of Artificial Intelligence (AI) based on data-driven algorithms has
become ubiquitous in today's society. Yet, in many cases and especially when
stakes are high, humans still make final decisions. The critical question,
therefore, is whether AI helps humans make better decisions as compared to a
human alone or AI an alone. We introduce a new methodological framework that
can be used to answer experimentally this question with no additional
assumptions. We measure a decision maker's ability to make correct decisions
using standard classification metrics based on the baseline potential outcome.
We consider a single-blinded experimental design, in which the provision of
AI-generated recommendations is randomized across cases with a human making
final decisions. Under this experimental design, we show how to compare the
performance of three alternative decision-making systems–human-alone,
human-with-AI, and AI-alone. We apply the proposed methodology to the data from
our own randomized controlled trial of a pretrial risk assessment instrument.
We find that AI recommendations do not improve the classification accuracy of a
judge's decision to impose cash bail. Our analysis also shows that AI-alone
decisions generally perform worse than human decisions with or without AI
assistance. Finally, AI recommendations tend to impose cash bail on non-white
arrestees more often than necessary when compared to white arrestees.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要