Fair Payments in Adaptive Voting

Adaptive voting is a commonly used scheme for aggregating consensus in crowdsourced binary labeling tasks. Workers assign labels to an item until the votes for one class outnumber the votes for the other class by a given integer threshold. Modeling the process as a Markov random walk, we offer results on how workers with different accuracy levels should be paid comparatively to each other under an adaptive voting scheme. We calculate the expected accuracy of the final consensus vote and estimate the number of votes necessary for the process to finish. We show how to derive a fair payment policy for two groups of workers with different accuracy rates in a way that guarantees that the task is associated with the same cost and generates results of the same quality when assigned to either group. This paper also compares the adaptive voting scheme with majority voting, demonstrating evidence for a strict dominance of the former. Our model is simple yet flexible and provides the foundation for analysis of more complex settings.