Web Privacy Measurement : Scientific principles , engineering platform , and new results Draft – Jun 1 , 2014

The results of web privacy measurement have been very influential in online privacy debates. As a research field, however, web privacy measurement is immature and fragmented, and has not yet acquired an identity as a unified discipline. There are significant scientific and engineering challenges but the solutions tend to be ad-hoc. We identify 32 web privacy measurement studies, cast them as instances of a generic experimental framework, and perform a thorough methodological analysis. We analyze design and implementation alternatives and make recommendations based on considerations of experimental rigor and engineering feasibility. Next, we present a flexible, modular web privacy measurement platform that supports any experiment that fits the framework. It is also highly scalable and avoids many common pitfalls. Finally, as a case study of our methods and infrastructure, we measure the “filter bubble”, i.e., the extent of personalization based on a user’s history, by crawling approximately 300,000 pages across nine news sites and present evidence that this personalization effect has been greatly overstated in the popular press.