High-Frequency Simulations of Global Seismic Wave Propagation Using SPECFEM3D_GLOBE

SPECFEM3D_GLOBE is a spectral-element application enabling the simulation of global seismic wave propagation in 3D anelastic, anisotropic, rotating and self-gravitating Earth models at unprecedented resolution. A fundamental challenge in global seismology is to model the propagation of waves with periods between 1 and 2 seconds, the highest frequency signals that can propagate clear across the Earth. These waves help reveal the 3D structure of the Earth's deep interior and can be compared to seismographic recordings. We broke the 2 second barrier using the 62K processor Ranger system at TACC. Indeed we broke the barrier using just half of Ranger, by reaching a period of 1.84 seconds with sustained 28.7 Tflops on 32K processors. We obtained similar results on the XT4 Franklin system at NERSC and the XT4 Kraken system at University of Tennessee Knox- ville, while a similar run on the 28K processor Jaguar sys- tem at ORNL, which has more memory per processor, sus- tained 35.7 Tflops (a higher flops rate) with a 1.94 shortest period. For the final run we obtained access to the ORNL Petaflop System, a new very large XT5 just coming online, and achieved 1.72 shortest period and 161 Tflops using 149,784 cores. With this landmark calculation we have enabled a powerful new tool for seismic wave simulation, one that operates in the same frequency regimes as nature; in seismology there is no need to pursue periods much smaller because higher frequency signals do not propagate across the entire globe. We employed performance modeling methods to identify performance bottlenecks and worked through issues of par- allel I/O and scalability. Improved mesh design and num- bering results in excellent load balancing and few cache misses. The primary achievements are not just the scalabil- ity and high teraflops number, but a historic step towards understanding the physics and chemistry of the Earth's inte- rior at unprecedented resolution.