NOBUGS 2016

Advances in High-Performance Data Analysis & Data Management at the APS

18 Oct 2016, 14:10

20m

Marble Hall (Copenhagen University)

Oral Contribution Contributions 5

Speaker

Mr Nicholas Schwarz (Argonne National Laboratory)

Description

High-performance data analysis software and computing infrastructure are of particular importance for synchrotron facility productivity. Demands for increased computing at user facilities is driven by new scientific opportunities enabled by new measurement techniques, technological advances in detectors, multi-modal data utilization, and advances in data analysis algorithms. The proposed high-energy, MBA-based storage-ring upgrade to the APS will increase brightness and coherence, leading to further increases in data rates and experiment complexity, creating yet further demands for the application of advanced scientific computational techniques. The APS and its parent institution, Argonne National Laboratory (ANL), are well poised to address these computational challenges. ANL is home to world-leading supercomputing infrastructure and computer science expertise. This colocation provides an unprecedented opportunity for collaboration. The APS has been bringing together expertise in high-performance data analysis and large data management, and leveraging computational and storage resources in order to address the facility’s current and upcoming challenges. At present, the APS collects approximately 2 PB of raw experimental data per year. This rate is quickly increasing due to the aforementioned reasons. To address the need for larger capacity storage in the near-term, the APS has implemented storage solutions in cooperation with the Argonne Leadership Computing Facility. The Petrel data system is a 1.7 PB data store that has been serving the needs of the APS for the past year. As of the middle of 2016, the APS has brought another storage system, Extrepid, online making an additional 1.7 PB of storage available for APS experiments. Each system is housed in a separate computing building on campus, and connected to the APS via individual dedicated 2 x 10 Gbps network links. Due to recent intra-campus network infrastructure upgrades, network bandwidth between the APS and these storage systems may be increased as needed. To best use these storage systems, software engineers and beamline staff at the APS continue to work closely with the Globus Services team to implement and deploy data management tools that integrate with beamline data workflows. These tools help automate the transfer of data between acquisition devices, computing resources, and data storage systems. Ownership and access permissions are maintained based on an experiment’s user group. A metadata catalog allows beamline staff to populate experiment conditions and information for access via a web portal. User groups can download data at their home institutions using Globus Online. These data management tools are now deployed at ten APS beamlines. More powerful computational resources and newly developed and applied high-performance computing software is being utilized in order to deliver analyzed data within shorter time frames. For example, staff and users at the APS are regularly utilizing a 128-node cluster equipped with 256 K80 GPUs for fast, parallel 3D volume rendering of large tomographic datasets during beam time. Scalable, virtualized computing clusters are tightly coupled with beamline workflows and are in regular use for most X-ray Photon Correlation Spectroscopy (XPCS) experiments at the facility’s dedicated XPCS beamline. Elemental mapping, reciprocal-space mapping, and Bragg coherent diffraction imaging, software is being ported to run on multi-threaded and distributed-memory computing resources. *Work supported by U.S. Department of Energy, Office of Science, under Contract No. DE-AC02-06CH11357.