A 500,000-Person Experiment Evaluating the Impact of Geographically-Targeted, High-Payoff Vaccine Lotteries

Lotteries have been shown to motivate behavior change in many settings. However, the value of large-scale, geographically-targeted lotteries as a policy tool for changing the behaviors of entire populations is a matter of heated debate. In mid-2021, we implemented a pre-registered, city-wide experiment in Philadelphia to test the effects of three, high-payoff (up to $50,000) geographically-targeted lotteries designed to motivate adult residents of Philadelphia to get vaccinated against COVID-19. All Philadelphia residents ages 18 and older were eligible for inclusion in each drawing but, if selected, could not accept a prize unless they had received at least one dose of a COVID-19 vaccine. In each drawing, residents of a randomly selected “treatment” zip code received half of the 12 lottery prizes (boosting their chances of a win to 50-100x those of other Philadelphians). This experimental design makes possible a causal estimate of the impact of vastly increasing people’s odds of winning a vaccine lottery. We estimate that the first treated zip code, which drew considerable media attention, may have experienced a small bump in vaccinations compared to control zip codes: vaccinations rose by an estimated 61 per 100,000 people (an 11% increase). Pooling results from all three zip codes treated over the course of our six-week experiment, however, we do not detect any overall benefits. This unsustained effect may be because media attention waned, salience of the lottery declined, or attitudes about vaccination became increasingly entrenched over time. Further, our 95% confidence interval provides an upper bound on the overall benefits of treatment in our study of 9%. Given that lotteries of this scale cost hundreds of thousands of dollars to implement, the lack of a substantial benefit from this experiment strengthens the policy case for other, more impactful ways to encourage health behavior change.