CERN Accelerating science

LHCb adopts GPUs in Run 3 trigger

Dorothea Vom Bruch (CNRS), Daniel Hugo Campora Perez (Nikhef), Olli Lupton (Uni. Warwick) 22nd Jun 2020
LHCb Trigger Run 3
LHCb-coverThe LHCb experiment is currently being upgraded to continue its study of beauty and charm hadrons at a five times higher instantaneous luminosity in LHC Run 3 that is now set to start in 2022. At the increased luminosity, these hadrons are produced so abundantly that selections based only on calorimeter or muon detector information are inefficient. Consequently,  one major change in the LHCb upgrade is the readout of all sub-detectors at 30 MHz  and a software-only trigger processing 40 Tbit/s, where charged particle track reconstruction is performed up front. Inclusive selections on the reconstructed objects allow the event rate to be reduced by a factor between 30 and 60 at the first trigger level, called HLT1.
 
To address this computing challenge, the collaboration has completely rewritten its reconstruction and selection software to enable it to take advantage of modern highly parallel processing architectures. In particular, all HLT1 algorithms and the framework were adapted both for modern CPUs and also for GPUs, as described in this previous EP newsletter [Trigger Strategies for the LHCb experiment]. Knowledge transfer between the two solutions drove the development and led to factors of speedup in both implementations. The presence of the more established framework for CPU code, partially inherited from Run 2, helped to guide and accelerate the development of the new GPU framework.
 
An in-depth comparison of the physics performance and a study of the price-performance difference showed that both implementations fulfilled and in places surpassed the criteria outlined in the 2014 Trigger and Online TDR. Thus, LHCb was in the privileged position of being able to choose between two viable and highly performant options.
 
In the end, LHCb decided to become the first experiment with a complete high-throughput trigger on GPUs. The choice will impact the composition of LHCb's data acquisition system, where GPUs will discard data earlier, leading to reduced network costs. It will also ensure LHCb collaborators are in a position to drive future heterogeneous computing innovation both within LHCb and more widely in HEP. 
 
As Vava Gligorov (Project Leader) explains: "With commercial data processing increasingly carried out by heterogeneous mixtures of CPU, GPU, and FPGA processors, Allen offers LHCb both a performant system for Run 3 and the chance to stay ahead of the technology curve and maximally benefit from future industry developments. And the advances made in CPU processing speed will not be wasted but pay off in making LHCb's second-level CPU trigger more performant and hence deliver more physics for the collaboration's analyses -- the best of both worlds!"

Following the technology decision, the focus has shifted to developing and commissioning the trigger system for Run 3.

Development and optimisation has continued during the technology review, resulting in increased performance and new opportunities to expand the reconstruction performed in the first stage of the software trigger. For example, improved calorimeter clustering and bremsstrahlung recovery are areas where spare processing capacity could be used to expand the physics reach of the system beyond what was originally envisaged.

LHCb data acquisition system for Run 3 with the originally planned CPU HLT1 (left) versus the now adopted GPU HLT1 (right).
 
Commissioning the system for Run 3 will involve integrating GPU-friendly developments into the Gaudi framework that underpins the LHCb and ATLAS software ecosystems. This work should also benefit other systems and experiments that move to heterogeneous architectures, and indeed is exactly the type of development envisaged in the HSF community white paper for HEP software and computing in the 2020s [1].
 
The three authors of this piece were recently awarded LHCb early career researcher prizes for their contributions to the Run 3 trigger.

Further Reading:

[1] https://arxiv.org/abs/1712.06982

[2] https://link.springer.com/article/10.1007/s41781-020-00039-7

[3] Allen TDR: http://cds.cern.ch/record/2717938?ln=en

 

LHCb honours its Thesis and Early Career Awards Winners

On 11 June, LHCb announced the winners of the 2020 PhD Thesis and Early Career Scientist Awards. The LHCb Thesis Awards recognize excellent PhD theses and additional work that have made an exceptional contribution to LHCb. In parallel, the Early Career Scientist prizes are awarded to recognize outstanding achievements of early career scientists to the benefit of LHCb.

This year’s winners of the Thesis prize are Philippe D'Argent (Heidelberg University) and Laurent Dufour (Nikhef/Groningen University. Carlos Abellan Beteta (Zurich), Claudia Bertella (CERN), Daniel Campora (Nikhef), Nadim Conti (INFN, Milan), Edgar Lemos Cid (Santiago de Compostela), Olli Lupton (Warwick), Mark Smith (Imperial College), Dorothea vom Bruch (LPNHE, Paris) were awarded the Early Career prize.

 

Left to right, top to bottom: Smith, Passaleva (LHCb Spokesperson), Hansmann-Menzenmer (Thesis Chair), Conti, Lupton, Campora, Abellan-Beteta, Vom Bruch, Lemos Cid, Bertella, Dufour, Dordei (Early Career Chair), D'Argent (Image: CERN).

“The number of brilliant winners of the Early Career Scientist prize and the extraordinary level of the PhD theses evaluated, show how crucial the contribution of younger colleagues to the experiment activities truly is”, point out Francesca Dordei and Stephanie Hansmann-Menzemer, Chairs of the Prize Committees. “It was really hard for the Committees to select only a few names among the many early career scientists and PhD students that not only contribute but often lead cutting edge developments in LHCb physics, detector and software developments”. Almost 350 PhD students study in the collaboration on diverse areas of LHCb physics, ranging from physics analysis to advanced detector and software developments.

 

Read more about this year's winners and their work on the LHCb public website: HERE