Is It Better To Elicit Quantile Or Probability Judgments to Estimate a Continuous Distribution? Elicit Both and Create an Inner Crowd.

Managers frequently rely on the judgment of an expert to estimate a probability distribution for a continuous random variable. Two elicitation methods are commonly used to gather this information: (i) quantile judgments for a set of fixed probability values, or (ii) probability judgments based on a set of fixed variable values, but a consensus on which format yields more accurate distribution estimates has not been reached. We report results from a series of eight experiments conducted with 1,456 participants with a range of quantitative backgrounds to compare these elicitation formats for a variety of variables, including synthetically generated numbers displayed in a video, daily high and low temperatures, the ages of people with a given name, commute times, home prices, and household incomes. In total, our data set includes 30,870 complete distribution judgments. The data show that probability distributions constructed from quantile judgments tend to be more accurate than those obtained from cumulative probability judgments. However, visual probability elicitation tools offered similar performance to quantile judgments, with each format holding a slight advantage according to different metrics. On average, quantile judgments were slightly further from the empirical distribution while visual probability judgments were slightly overconfident. We find that averaging the distributions obtained from quantile judgments with those obtained from visual probability judgments provides superior performance to either approach, delivering significantly better accuracy and calibration of confidence intervals. Based on these results, we recommend, when possible, eliciting judgments using both the quantile and visual probability formats and combining the resulting distributions.