Update Delivery Mechanisms for Prospective Information Needs: A Reproducibility Study

Real-time summarization systems monitor continuous streams of documents with the goal of delivering relevant, novel, and timely updates to users. These updates can either be sent to users' mobile devices as push notifications or be silently deposited in an inbox to be consumed - the important difference is whether the user is interrupted by the delivery. Previously, a two-year study examining user attention under these different mechanisms revealed interesting findings about users' information consumption behavior, but the conclusions were marred by a few methodological shortcomings. We present a reproducibility study that follows the same design as the original evaluation, but corrects its flaws. We find that most conclusions from the original study are confirmed, although there are some surprising differences as well. Overall, the magnitude of the observed effects are not as strong as in the original study.