Who Is a Good Decision Maker? Data-Driven Decision Ranking under Unobservable Quality

The capacity to rank expert workers by their decision quality is a key managerial task of substantial significance to business operations. Yet, when no ground truth information exists, such evaluation typically requires enlisting peer-experts and is prohibitively costly in many important settings. In this work, we develop a data-driven approach for producing effective rankings based on the decision quality of expert workers; our approach leverages historical data on past decisions, which are commonly available in organizational information systems. Specifically, we first formulate a new business data science problem of Ranking Expert decision makers’ unobserved Quality (REQ) exclusively on the basis of historical decision data and without resorting to evaluation by peer experts. The REQ problem is challenging because the correct decisions in our settings are unknown (unobserved), and because some of the information used by decision makers may not be available for retrospective evaluation. To address the REQ problem, we develop a machine-learning-based approach and analytically and empirically explore conditions under which our approach is advantageous. Our empirical results over diverse settings and datasets show that our method yields robust performance: its rankings of expert workers are consistently either superior or at least comparable to those obtained by the best alternative approach.