SPEC HPG benchmarks: performance evaluation with large-scale science and engineering applications

It is being increasingly understood that the evaluation of high-performance com- puter systems must be based on realistic, full applications. Kernel, algorithm, and small-application benchmarks are important for measuring and discussing the performance of individual components of a computer system. However, the complex performance behavior of these components in concert and in the con- text of a large computational problem can only be realistically understood if we observe the problem as a whole. These needs are further increased by the fact that computer systems are able to solve ever larger and more complex prob- lems. Doing performance evaluation at this large scope is a grand challenge and exceeds the resources of small companies and most research projects. To address this challenge, a joint industrial-academic eort has been ini- tiated with the mission to facilitate performance evaluation and benchmarking with large-scale applications. There are two complementary goals. The industrial participants pursue the development of systems benchmarks that provide mean- ingful and relevant applications for measuring and comparing high-performance computer systems across diverse platforms. The academic participants seek re- alistic program suites for evaluating and guiding research of novel computer systems concepts and prototypes. Both performance evaluation eorts have the need for large-scale, industrially-relevant applications.