SELVEDAS: Innovative approach to promote efficient data workflow between institutions

June 17, 2021 – by CSCS 

As the largest research institution for natural and engineering sciences in Switzerland, the Paul Scherrer Institute (PSI) generates petabytes of data per year with experiments at research facilities such as the Swiss Light Source (SLS), the Swiss X-ray Free Electron Laser (SwissFEL) and Swiss Spallation Neutron Source (SINQ). PSI has been storing several petabytes of this data on a highly scalable storage solution at the Swiss National Supercomputing Centre (CSCS) since 2018. From 2024 onwards, however, PSI predicts exponential data growth — several hundred petabytes of data are expected to be generated, in fact. This has forced researchers, computer scientists and software engineers to rethink data workflow between the institutions and how data is accessed, processed, analysed and stored.

Data processing during ongoing experiments

One of the most challenging use cases of the project was to enable a data workflow where PSI scientists conducting experiments on one of the research facilities at PSI could transfer data immediately to CSCS, leveraging on the dedicated network connection. For example, when researchers study the internal nano structures of objects with the Swiss Light Source, the results must be immediately processed and returned to PSI, as the next step of the experiment is dependent on the processed form of the data. 

As both PSI and CSCS serve different clients, it is highly complex to integrate such a use case, as both institutions are expected to maintain their autonomy to meet their internal and external requirements. The final design takes these constraints into consideration and successfully demonstrates a solution that can be applied across institutional and physical boundaries, with respect to customer requirements and their respective political restrictions. 

To bring this design to life, CSCS developed Sarus and FirecREST as core components. Sarus enables scientists to run their portable scientific workflows in portable containers while ensuring native performance on HPC systems. FirecREST provides a ubiquitous interface to support complex scientific workflows while facilitating institutional needs to maintain their respective autonomy. 

Pathfinders for the future

The PSI-CSCS team is convinced that collaborative efforts such as SELVEDAS are pathfinders for identifying future, scalable data infrastructure and solutions. They are a blueprint for future collaborations under "Alps", the new supercomputer infrastructure being developed at CSCS. "Alps", will provide users with a software-defined research infrastructure that uses cloud methodology for multitenancy, supercomputing, and data infrastructures, and where PSI can be a tenant. "With SELVEDAS, we have achieved an important milestone in showing the potential of cloud-based HPC services to PSI and other potential CSCS collaborators", says CSCS’s SELVEDAS project leader Cerlane Leong. Alun Ashton, Leong’s counterpart at PSI, points out: "Through the successful collaboration, CSCS has demonstrated how "Alps" could offer a future software-defined research infrastructure to PSI."

Image above: Swiss Light Source, PSI (Image: Schölla Schwarz)

This article may be used on other media and online portals provided the copyright conditions are observed.