A/B Testing Of Auctions

ABSTRACTA common method in the practice of large scale auction design, e.g., in auctions placing advertisements on online media and Internet search engines, is A/B testing. In A/B testing, the auction house is running an incumbent mechanism A, and would like to determine if a novel mechanism B obtains higher revenue. This is done by splitting the traffic so that most of it goes to A and some of it, e.g., five to ten percent, goes to B. An issue with this approach is that if the bidders are unaware of which mechanism their bid will be considered in, the bid equilibrium is neither for A nor B but for a mechanism C that is a convex combination of A and B. A miscalculation sometimes performed in practice is to consider and compare average revenue from A (resp. B) from the times when A (resp. B) is run. This miscalculation is equivalent to simulating A on the bids in C and can often give the opposite conclusion; e.g., if A and B are one- and two-unit highest-bids-win winner-pays-bid auctions, respectively, then B will always appear in this miscalculation to have higher revenue. For a fixed set of bids, a winner-pays-bid mechanism's revenue is monotone in its allocation probabilities. Of course, in equilibrium, increased allocation probabilities can cause reduced revenue as bidders may lower their bids. We present an A/B testing method that applies generally to the position auction model popularized by the Varian [2007] and Edelman et al. [2007] analyses of auctions for sponsored search and now a fundamental model for the study of auction theory; e.g., see Hartline [2013]. A position auction is defined by a decreasing sequence of weights, bidders are assigned to positions in decreasing order of bids, and payments are charged. Typical payment rules are "generalized first price" and "generalized second price"; the former requires bidders to pay their weighted bid, whereas the latter requires bidders to pay the weighted bid of the next highest bidder.