DIRAC and Rucio are two software projects used for scientific computing. Both projects are led by CERN, and the EP department is one of the major contributors to both. 

DIRAC is used by CERN experiments such as LHCb, CLIC, FCC, NA62, and non-CERN experiments like Belle II and Juno, among others. It aims to offer a complete grid solution for scientific communities to exploit distributed resources. 

Rucio focuses on managing large volumes of scientific data in distributed environments. It is used by LHC-experiments ATLAS and CMS, neutrino experiments DUNE and ICARUS, CERN recognized experiments AMS, Belle II, LIGO/VIRGO and CTA as well as non-HEP experiments such as LSST, the ESCAPE open collaboration and many others.

In recent years, scientific experiments have raised interest in using both software systems together to solve their scientific computing challenges. For this reason, both projects decided to come together to organize a first joint workshop to address the computing challenges of the coming years. The DIRAC & Rucio workshop was held from October 16th-20th 2023 in Japan. As Belle II was the first experiment to actually operate both systems in an interconnected way, it was very fitting to come together in the Belle II host-lab KEK to discuss the future of both software systems in a joint workshop.

Participants in the combined DIRAC & Rucio workshop that was held from October 16th-20th 2023 in Japan.

The principal idea of the in-person workshop, which was attended by 44 international participants, was to discuss the computing challenges and evolution of the different experiments in a joint way, and to understand how these could best be addressed by both software systems. The workshop provided a crucial platform for developers, service administrators, and users to exchange valuable insights, updates, and advancements in the world of scientific computing. The event, spanning the whole week, featured a rich agenda of welcome sessions, community talks, technology discussions, and hands-on demonstrations.

The workshop started with presentations, including updates on the state of the DIRAC projects and the state of the Rucio project. A global perspective was established by a set of community talks: communities such as LHCb, ATLAS, Belle II, EGI, CMS, CTAO, GridPP, Juno/BES3, FermiLab/DUNE, JINR, Rubin/LSST presented their usage of DIRAC and Rucio.

The workshop then delved into the latest technological advancements in the DIRAC and Rucio ecosystems. Major discussion points included the evolution of the compute and storage landscape with the inclusion of High Performance Computers (HPCs) and commercial clouds. Another crucial topic was the migration of the authentication and authorization infrastructure within WLCG, and beyond, from X509 certificates to oAuth2/OIDC based tokens. This topic required special attention at the workshop, since it cannot be solved in isolation, but needs to be addressed in a coordinated architecture design by both projects.

DIRAC developers demonstrated the first prototype of the DiracX project. DiracX is a full rewrite of DIRAC, and stands to be the next DIRAC incarnation. The current, production, DIRAC framework is aging and, in order to keep the project successful, a new implementation is needed. DiracX is a cloud native application meant to embrace up-to-date standards of web services developments and deployments. The first production-level services of DiracX will be ready by the end of the first quarter of 2024. A transition plan from DIRAC to DiracX was presented.

Intensive discussions among the participants continued over the coffee-breaks.

On the Rucio side, the long anticipated upgrade of the web interface was presented, which offers a fully responsive design for all types of (mobile) devices as well as an accessible interface for users with disabilities. Novel deployment methods, research into alternative file transfers based on torrent technology, as well as monitoring updates were presented as well.

The workshop wasn't just about technology: it also offered opportunities for social interaction. A welcome drink, many coffee breaks, and a social dinner gave participants a chance to connect. We also had the unique opportunity for a site visit to the Belle II experiment hall, gaining hands-on insights into the experimental setup. The workshop concluded with summaries and conclusions presented by the DIRAC and Rucio project leaders, announcing future workshops and joint hackathons. 

The DIRAC & Rucio Workshop 2023 served as a vital platform for the exchange of ideas, collaboration, and learning. As the DIRAC and Rucio communities continue to evolve, workshops like these play a pivotal role in shaping the future of distributed computing. The insights gained and connections made during this workshop will undoubtedly contribute to the ongoing success and innovation in the field.