FIG: A Prototype Tool for Online Verification of Recovery Mechanisms

Network applications of the future will require advanced mech- anisms for automatic failure recovery in order to provide an acceptable quality of service. Because of this require- ment, there is a need for tools that can inject simulated faults to verify a system's fault detection and recovery meth- ods. We present a set of objectives that such a fault injec- tion tool must meet, and describe a prototype we have de- veloped that meets these criteria. This tool, FIG, is capable of selectively introducing and logging errors at the applica- tion/library boundary on a running system. Our expectation is that FIG will be used for software de- velopment and verication, as well as for on-site testing of production systems. We use a working version of this tool to test the behavior of several common UNIX applications un- der simulated failures and oer suggestions from these tests on how to develop software with a higher degree of failure recoverability.