Debugging Scientific Applications With Statistical Assertions

Traditional debuggers are of limited value for modern scientific codes that manipulate large complex data structures. Current parallel machines make this even more complicated, because the data may be distributed across multiple processors, making it difficult to view, interpret and validate the contents of a distributed structure. As a result, many applications' developers resort to placing validation and display code directly in the source program itself. This paper discusses a novel debug-time assertion, called a "Statistical Assertion", that allows a user to reason about large data structures. We present the design and implementation of statistical assertions, and illustrate the debugging technique with a molecular dynamics simulation. We evaluate the performance of the system on a 12,000 processor Cray XE6, and show that it is useful for real time debugging.