Interactive Supercomputing for Experimental Data-Driven Workflows

Abstract

Large scale experimental facilities such as the Swiss Light Source and the free-electron X-ray laser SwissFEL at the Paul Scherrer Institute, and the particle accelerators and detectors at CERN are experiencing unprecedented data generation growth rates. Consequently, management, processing and storage requirements of data are increasing rapidly. Historically, online and on-demand processing of data generated by the instruments used to be tightly-coupled with a dedicated, domains-specific, site-local IT infrastructure. Cost and performance scaling of these facilities not only pose technical but also planning and scheduling challenges. Supercomputing ecosystems optimize cost and scaling for computing and storage resources but typically exploit a shared batch access model, which is optimized for high utilization of compute resources. In comparison, in public clouds, on-demand service delivery models address the concept of elasticity while maintaining isolation with performance trade-offs. Furthermore, these on-demand access models allow for different degrees of privileges to users for managing IT infrastructure services, in contrast with shared, bare-metal supercomputing ecosystems. This paper outlines an approach for enabling interactive, on-demand supercomputing for experimental data-driven workflows, which are characterised by a managed but bursty data and computing requirements. We present a delegated batch reservation model, controlled by the customer and provisioned by the supercomputing site, that allows scientists at the experimental facility to couple generation of data to the allocation of compute, data and network resources at the supercomputing centre. Scientists are then able to manage resources both at the experimental and supercomputing facilities interactively for managing their scientific workflows. Prototype implementation demonstrates that this rather simple co-designed extension to a supercomputing classic batch scheduling system with a controlled degree of privilege can be easily incorporated to the experimental facilities existing IT resource management and scheduling pipelines.