17 April 2020

During its first year, ESCAPE paved the way for an open access Data Lake infrastructure, the ESCAPE Data Infrastructure for Open Science (DIOS).  This will allow large national research data centres to work together and build a single, robust, service to store, distribute and provide seamless access to data, and be able to scale up to multi-Exabyte needs.

By observing FAIR data management services at the basis, the Data Lake will serve global users to efficiently manage large volumes of data, making them accessible to distributed communities, while optimizing the cost of storage.

Scientific Community Needs – defining the DIOS architecture

How do we put the astronomy, particle physics & astroparticle physics data to use in data systems from science initiatives, namely the ESFRIs (European Strategy Forum on Research Infrastructures)? This was the main focus of the ESCAPE DIOS team during its first year, who shaped the architecture and its functional elements were structured on the basis of the data management requirements of the ESCAPE ESFRIs.

A Data Lake pilot is in place, composed of a range of storage services provided by the ESCAPE partners and orchestrated in such a way it is seen as a single service by the ESFRIs and the scientists. The first datasets from several ESFRIs already arrived: astrophysics, cosmology and particle physics experiments injected real data into the current pilot Data Lake. Data access methods and tools are being explored to provide scientists the flexibility to process the data from several resources, from sites accessible through Grid interfaces, to Cloud resources and High Performance Computers (HPC).

The data at ESCAPE DIOS is being organised, orchestrated and catalogued, with clear policies for data replication and deletion, along with a set of application programming interfaces (APIs). These APIs will allow the end-user to manage and access the data.

                                                                   

Figure – The building blocks of the ESCAPE DIOS infrastructure and the connection with compute services

Finally, given the heterogeneous and distributed nature of the system, the Authentication, Authorization and Identity management (AAI) play a crucial role in the architecture. The system must meet the ESFRI policies and allow open access outside the data embargo periods. At the same time, the mechanisms for AAI need to scale for exa-scale data management, reducing the overhead and ensuring an adequate level of security.

THE ESCAPE DIOS Components and Reference Implementation

Different components are being used to build the ESCAPE DIOS and made available to the ESFRI as stand-alone services:

  • Storage Services: leveraging stand-alone services that have the capability to operate at the hundred-petabyte scale, namely dCache, DPM, EOS, StoRM and xrootd.
  • Asynchronous Data Transfer Services and Networking: by using the File Transfer Service (FTS) from the Worldwide LHC Computing Grid as a reference. ESCAPE implemented an asynchronous data transfer service across the partner sites composing the ESCAPE Data Lake.
  • Storage Orchestration Service: RUCIO Data Management system has been selected as the reference. RUCIO is an open-source data management system for scientific computing which manages data at a file granularity level and organises them in collections such as archives, datasets and containers, while doing monitoring. An ESCAPE RUCIO service has been deployed and is acting as the Data Lake orchestrator.
  • Content Delivery and Caching: different models to deliver data content to processing units, which may be or not co-located with the data.
  • AAI services: enable secure and controlled data access in the ESCAPE DIOS.
  • Event-driven data management and data processing: multi-level architecture with interchangeable components designed to maximize the flexibility and to allow changes to the architecture, if required.
  • Information and configuration system: to allow interoperability of ESCAPE DIOS different distributed services fulfilling different roles
  • Monitoring: logs of the services running on the infrastructure will be collected, stored and visualized using industry standard solutions. This allows a real-time view of the performance and load of the ESCAPE DIOS, as well as spot and investigate potential issues or downtime.
  • Functional and Stress Testing: to measure the ESCAPE DIOS performance in terms of content delivery on its distributed computing environment workflows.

Cooperative Actions at Regional and International Level for a Cooperative Science

The development of ESCAPE DIOS is leveraging as much as possible on technologies that already exist, developed in the context of other initiatives, namely the European projects, such as:

  • XDC project: File Transfer Service and RUCIO for data transfer and orchestration and the dCache, StoRM and EOS storage technologies
  • Indigo-DataCloud & EOSC-hub: Identity and Access Management service for ESCAPE AAI management and other technologies such as StoRM;
  • GEANT: perfSONAR for network monitoring and further work.
  • PRACE: Integration of HPC machines to process data stored in the ESCAPE DIOS.
  • ESFRIs: XCache for content delivery & caching and Hammercloud for commissioning and testing workflows in a distributed computing environment.

To bring the results closer to market, the ESCAPE DIOS relies on open source protocols, such as HTTP/WebDav that will enable commercial storage providers to be integrated. After demonstrating the ESCAPE DIOS computing interface and scalability, the platform will also integrate commercial resources.

What lies ahead? Integrating and Deploying the ESCAPE DIOS

The implementation of ESCAPE DIOS has already started and a Data Lake pilot is being deployed with the goal to demonstrate the model with a small scale and functional system which integrates the technologies identified in the preparation phase. The pilot should be assessed early 2021 to show that data can be organised and distributed across sites, and how a storage orchestration service at the level of a few Terabytes can be used across different storage technologies will also be demonstrated.

The prototype phase will focus on deploying a full-scale system, allowing functional tests and stress tests on all the capabilities needed by the ESFRIs for FAIR data management. More partner sites will be integrated into ESCAPE DIOS, consolidating functionalities and scaling up performance.

These will include large data centres with experience in long-term bit preservation on archival media, such as INFN-CNAF, SURF-SARA, IN2P3-CC, CERN, PIC. During this phase, ESCAPE DIOS will be ready to suit the real functionality, performance and usability needs of the ESFRI projects, with sizeable storage resources of up to 100TB.

A Pillar infrastructure to address next decades’ data challenges

The ESCAPE DIOS provides a flexible and robust data lake in terms of storage, security, safety and transfer and the basic orchestration machinery to enable the technology to be combined with high quality data from different communities and, therefore, the exploration of new areas in science. It will serve the international user communities, while being connected to the European Open Science Cloud (EOSC).

ESCAPE DIOS is a pillar infrastructure to be connected to EOSC, address the next decades’ data challenges and serve the international user communities. Know more about ESCAPE DIOS