Toward a Benchmark for Multi-Threaded Testing Tools

Looking for intermittent bugs is a problem that has been getting prominence in testing. Multi-threaded code is be- coming very common, mostly on the server side. As there is no silver bullet solution, research focuses on a vari- ety of partial solutions. We outline a road map for com- bining the research on the different disciplines of testing multi-threaded programs and on evaluating its quality. The project goals are to create a benchmark that can be used to evaluate different solutions, to create a framework with open API's that enables combining techniques in the multi- threading domain, and to create a focus for the research in this area around which a community of people who try to solve similar problems with different techniques, could congregate. The benchmark, apart from containing pro- grams with documented bugs, includes other artifacts, such as traces, that are useful for evaluating some of the tech- nologies. We have started creating such a bench mark and detail the lesson learned in the process. The framework will enable technology developers, for example, race detectors, to concentrate on their components and use other ready made components, (e.g., instrumentor) to create a testing solution.