Effective Change Detection Using Sampling

For a large-scale data-intensive environment, such as the World-Wide Web or data warehous- ing, we often make local copies of remote data sources. Due to limited network and compu- tational resources, however, it is often difficult to monitor the sources constantly to check for changes and to download changed data items to the copies. In this scenario, our goal is to detect as many changes as we can using the fixed download resources that we have. In this paper we propose three sampling-based down- load policies that can identify more changed data items effectively. In our sampling-based approach, we first sample a small number of data items from each data source and down- load more data items from the sources with more changed samples. We analyze the ef- fectiveness of the sampling-based policies and compare our proposed policies to existing ones, including the state-of-the-art frequency-based policy in (8, 11). Our experiments on synthetic and real-world data will show the relative mer- its of various policies and the great potential of our sampling-based policy. In certain cases, our sampling-based policy could download twice as many changed items as the best existing policy.