A suite of testbeds for the realistic evaluation of peer-to-peer information retrieval systems

Peer-to-peer (P2P) networking continuously gains popularity among computing science researchers. The problem of information retrieval (IR) over P2P networks is being addressed by researchers attempting to provide valuable insight as well as solutions for its successful deployment. All published studies have, so far, been evaluated by simulation means, using well-known document collections (usually acquired from TREC). Researchers test their systems using divided collections whose documents have been previously distributed to a number of simulated peers. This practice leads to two problems: First, there is little justification in favour of the document distributions used by relevant studies and second, since different studies use different experimental testbeds, there is no common ground for comparing the solutions proposed. In this work, we contribute a number of different document testbeds for evaluating P2P IR systems. Each of these has been deduced from TREC's WT10g collection and corresponds to different potential P2P IR application scenarios. We analyse each methodology and testbed with respect to the document distributions achieved as well as to the location of relevant items within each setting. This work marks the beginning of an effort to provide more realistic evaluation environments for P2P IR systems as well as to create a common ground for comparisons of existing and future architectures.