WoCMan : Harnessing the Wisdom of the Crowds for High-Quality Estimates

Estimation is common to many computational problems. “Where are the person’s eyes in this photo?”, “At what time in this audio recording does the interviewee accidentally swear?”, and “How many calories are in the food shown in this image?” are questions where the answer is an estimate of an unknown real value. Estimates are fundamentally approximate. Machine learning-based techniques are capable of producing some estimates, but developing and using such software typically requires expert domain knowledge. Surprisingly, non-expert groups of people are also capable of producing accurate estimates. This phenomenon, known as the wisdom of the crowds, holds promise in making estimation tasks accessible to ordinary programmers. We introduce WOCMAN, a domain-specific language (DSL) designed to make it easy for programmers to obtain high-quality estimates from the crowd. WOCMAN obtains interval estimates over arbitrary user-defined functions of crowd responses. Programmers declare their desired precision and budget, and WOCMAN iteratively increases the sample size until either the estimate is sufficiently refined or the budget is exhausted. We demonstrate with a “calorie counting camera” app.