Practical Issues in Dependability Benchmarking

Much of the work to date on dependability benchmarks has focused on costly, comprehensive measurements of whole-system dependability. But benchmarks should also be useful for developers and researchers to quickly evalu- ate incremental improvements to their systems. To address both audiences, we propose dividing the space of depend- ability benchmarks into two categories: competitive benchmarks that take the holistic approach, and less expensive developer benchmarks aimed at day-to-day development tasks. In this paper we differentiate the goals of these two types of benchmarks, discuss how each type might be appropriately realized, and propose simplifying assumptions for making them cost-effective.