Appentra, the CEA, Cray, CSCS operated by ETH Zurich, ECMWF, Jülich Supercomputing Centre and Seagate Technologies today announced the formation of a new consortium that will address the ubiquitous problems of data movement in data-intensive applications and workflows. The “Maestro” consortium will be supported by a three-year grant from the European Commission's H2020 Future Enabling Technologies for HPC (FETHPC) programme, to build a data- and memory-aware middleware framework for data-intensive computing.
The Maestro project has been set up to tackle one of the most important and difficult problems in HPC, namely the orchestration of data across multiple levels of the memory and storage hardware as well as the software stack. Although data movement is now recognized as the primary obstacle to performance efficiency, much of the software stack is not well suited to optimizing data movement, and was instead designed in an age where optimizing arithmetic operations was the priority. The Maestro project aims to capture the data- and memory-aware aspects of applications and the software stack into a new middleware layer which will perform basic data movement and optimisation on behalf of the application, also making use of modern memory systems.
Prof. Dirk Pleiter, Coordinator of the Maestro project said: “The Maestro project will provide a unique opportunity to challenge traditional approaches for handling data objects and data movements in complex HPC applications and workflows, which will be key for efficient exploitation of future exascale level supercomputers.”
About MAESTRO
Maestro will build a data-aware and memory-aware middleware framework that addresses ubiquitous problems of data movement in complex memory hierarchies and at many levels of the HPC software stack. The Maestro consortium consists of 7 expert partners, each bringing specialist knowledge and expertise to the technical challenge. Forschungszentrum Juelich will provide overall coordination and leadership of the project, as well as being responsible for one example use-case. Cray will provide technical coordination and will lead the core middleware design efforts. The European Centre for Medium-Range Weather Forecasts (ECMWF) will provide their production and research numerical weather prediction workflows as an example use-case to drive the direction of the project, and will co-design many aspects of the middleware. Seagate Technologies will develop an Object storage backend to the middleware and interface Maestro to it. The French Alternative Energies and Atomic Energy Commission (the CEA) will develop an in-situ framework based on the Maestro middleware. Appentra Solutions will enhance their Parallelware software for data awareness to support the Maestro middleware. The Swiss National Supercomputing Centre (CSCS) operated by ETH Zurich will develop a dynamic provisioning system targeting data-intensive applications.
Stay up to date with the MAESTRO project on Twitter, LinkedIn and the project website at: maestro-data.eu.