High Performance Data Management And Analysis For Tomography

The Advanced Light Source (ALS) is a third-generation synchrotron X-ray source that operates as a user facility with more than 40 beamlines hosting over 2000 users per year from around the world. Users of the Hard X-ray Micro-Tomography Beamline (8.3.2) often collect more than 1 Terabyte of raw data per day that in turn generates additional Terabytes of processed data. The data rate continues to increase rapidly due to faster detectors and new sample automation capabilities. We will present the development and deployment of a computational pipeline, fed by data from the ALS, and powered by the storage, networking, and computing resources of the local National Energy Research Scientific Computing Center (NERSC) and the Energy Sciences Network (ESNET). After one year of operation, the system contained 70,000 datasets and 350 TB of data from 85 users. All datasets now collected at the Hard X-ray Tomography Beamline are automatically reconstructed using parameters set by users and/or that are automatically detected from the data acquisition control system. Results are presented to users for visualization through a secure web portal. Users can then download their data or launch a (currently limited but) growing number of operations based on the data-such as filtering, segmentation, and simulation. The massive computational resources of NERSC are thus made available on a level that is easily accessible to the full range of micro-tomography users.