ESCAPE - The European Science Cluster of Astronomy & Particle Physics ESFRI Research Infrastructures has received funding from the European Union’s Horizon 2020 research and innovation programme under Grant Agreement no. 824064.
November
17,
2020
The ESCAPE Data Lake infrastructure is currently integrating ten storage endpoints provided by the ESCAPE partner institutes across Europe, these are: INFN-CNAF, INFN-ROMA, INFN-Napoli, DESY, SURF-SARA, IN2P3-CC, CERN, IFAE-PIC, LAPP and GSI. This is the first implementation of the conceptual design of building a common storage and data management infrastructure for open science involving different scientific communities.
A 24h Full Dress Rehearsal Exercise was organized on the 17th of November 2020. This allowed us to perform a thorough assessment about the status of the Data Management tools integration with the RIs. This exercise consisted in running multiple experiment data workflows on a single day, focused on data injection, data replication and data access.
During that day the experiments made use of 12 different storage endpoints across the Data Lake and transferred a million files while replicating real production workflows, in some cases ending up with end-user data analysis. The RIs involved were: LOFAR, CTA, MAGIC, FAIR, SKA, LSST, ATLAS, CMS and EGO/VIRGO.
Incorporation of third party centers was proven. As an example, this is the case of the MAGIC telescope where an external storage was integrated in the Data Lake. This external storage is the telescope’s data acquisition (DAQ) buffer space, in charge of recording data from the telescope.
We are currently in the process to commission commercial cloud resources; the procurement phase for a proof of concept started and in the next months the integration of the ESCAPE Data Lake should be finished and ready for a small scale test involving one or more RIs.
More detailed report on this milestone is attached at the end of this page.
File:
