An Adaptive Scheme for Overload Handling in Active Data Warehouses

This paper presents a novel adaptive approach for scheduling updates in a data warehouse that processes "near- real-time" data streams. Data is pushed to the warehouse from a variety of external sources with a wide range of inter-arrival times (e.g., once a minute to once a day). Due to network conditions, the volume of incoming data can widely vary and data streams can experience intermittent outages. Maintaining data freshness in the presence of outages and load variations can be challenging. In prior work, ad hoc heuristic algorithms have been proposed for doing this. In this paper, a systematic approach based upon global multiprocessor real-time scheduling theory is considered. The proposed approach can handle overload and recovery situations and maintain guarantees on data freshness for warehouse tables that are not impacted by outages. Simulation experiments are presented that show the effectiveness of the proposed approach.