Predictable high performance data management - leveraging system resource characteristics to efficiently improve performance and predictability

Much of today's IT infrastructure, including high-performance systems, suffers from poorer and less predictable performance than necessary, due to ineffective resource management. While processor performance is increasing at a rapid rate, increases in storage and memory performance are rather marginal, turning them the into serious bottlenecks, particularly for data-intensive applications. At the same time, memory and most storage subsystems operate in best-effort mode without any performance guarantees We show that better and more predictable performance can be achieved by considering system resource characteristics. Our work on disk scheduling shows how this unpredictable resource with performance differences of up to three orders of magnitude can be effectively managed, and guaranteed. Our analysis of memory performance reveals a somewhat similar behavior to disk I/O with two orders of magnitude performance difference between best and worst case. With search—inarguably one of the most crucial applications today, as we begin to drown in the information age—we demonstrate how performance of memory bound applications can be predicted and improved. We developed P-ary search, a novel algorithm that scales with parallel memory accesses and yields performance gains of up to 130% over conventional approaches.