Before the start of the LHC Run 3, CMS improved many aspects of its trigger system in order to boost the physics reach of the experiment. To select events of potential physics interest, the CMS trigger system divides the processing into two levels: the Level-1 trigger (L1T), implemented in custom hardware, and the high-level trigger (HLT), implemented in software, running on a farm of commercial computers. In Run 3, the L1T outputs events at a peak rate of 100 – 110 kHz, for standard proton-proton (pp) collisions at a peak luminosity of  2.2 x 10³⁴ cm^-2 s^-1. The HLT performs a more detailed reconstruction and selects events at a rate of about 2.6 kHz at peak luminosity for “prompt” offline reconstruction, usually within 48 hours of data collection. The HLT also stores extra samples called the parking data sets, which will only be reconstructed when the resources are not needed for the core activities, at a rate of around 3 kHz. An additional data set called “HLT data scouting'' is stored at a higher rate of 30 kHz with a smaller event content and no offline processing. Furthermore, the addition of a 40 MHz L1T scouting system, commissioned in the early part of Run 3, has the potential to further broaden the physics reach of CMS. In this article, we describe the Run 3 developments for the L1T and the HLT.

The Level-1 Trigger

The L1T [1] consists of electronics responsible for making a fast (within about 4 μs) selection of events based on the presence of interesting particle signatures in the detector. The L1T receives energy and position measurements, so-called trigger primitives (TPs), from the calorimeters and the muon detectors. The TPs are evaluated by custom-built electronics and field programmable gate arrays (FPGAs) that perform the trigger decision based on predefined algorithms. The L1T calorimeter and muon systems reconstruct jets, electrons, photons, hadronically decaying τ leptons, and muons, and the calorimeter trigger computes energy sums. The L1T global trigger (GT) then makes a trigger decision based on the multiplicity and kinematic information of these reconstructed particles, their proximity to each other, timing information, and beam presence. The L1T configuration is implemented in a “menu” including several hundred trigger algorithms. Upon a positive GT decision, the full detector data are read out for further filtering in the HLT. Between 2013 and 2015, the L1T hardware was entirely upgraded and has subsequently operated successfully.

Furthermore, the L1T architecture is extremely flexible and has allowed the exploration of new triggering techniques. For example, in Run 2, the GT was expanded from one to three and then to six FPGA boards to accommodate a more demanding physics program. Also in Run 2, CMS implemented a time-multiplexed trigger, which allows for parallel processing of events and has now become the standard in triggering in high-energy physics. For Run 3, although no major trigger hardware upgrade was performed, new capabilities have become available through new algorithmic approaches. Given the flexibility of the system, other hardware such as L1T scouting and a test crate can be run in parallel, as will be explained below. In addition, the L1T system was designed to output a rate of 100 kHz, although in Run 3, it now runs at 110 kHz. This is a clear demonstration of the capability of the CMS data acquisition system and sub-detector readout.

The L1 Calorimeter Trigger Run 3 Upgrades

An important new Run 3 feature in the L1 calorimeter trigger is a long-lived particle (LLP) jet identification algorithm that uses hadronic calorimeter (HCAL) timing and depth information. This information was only made available after LS2 due to the Phase 1 HCAL Upgrade. In addition to tagging LLP jets, the boosted jet reconstruction with substructure and the use of machine-learning techniques to calibrate the jet energy and perform pileup subtraction are being investigated for Run 3.

The L1 Muon Trigger Run 3 Upgrades

In Run 3, the L1 muon trigger provides measurements of muon displacements from the primary interaction point. This information is used in L1 muon triggers targeting displaced muons originating from LLPs. The new algorithms include a transverse momentum p_T measurement computed without a constraint to the beamspot and a measurement of the transverse displacement d_xy from the beam line. The algorithm that reconstructs muon tracks in the barrel shows efficiencies of around 80% for muons with p_T > 10 GeV and displacements up to 100 cm, using a dedicated displaced algorithm with unconstrained p_T (see Fig. 1). At the same time, it retains high efficiency for prompt muon tracks with d_xy < 50 cm with a prompt algorithm (constrained p_T). Running both the constrained and unconstrained algorithms is a new feature of the CMS L1T, and it is made possible by its flexible architecture. Similar algorithms are being implemented in the overlap and endcap regions, which will improve the trigger efficiency for displaced muons within the full pseudorapidity coverage of the muon system.

Figure 1: The trigger efficiency for the barrel muon track finder algorithms with unconstrained p_T (blue), which targets displaced muons, and constrained p_T (red), which targets prompt muons, as a function of the muon track d_xy. The efficiencies were measured using a sample of cosmic ray muons from 2023 data, with L1 muon p_T > 10 GeV. Taken from Ref. [2].

In Run 3, for the first time, the L1T additionally triggers on events with a high multiplicity of hits in the muon endcaps. Such showers could come from LLPs decaying electromagnetically or hadronically in the cathode strip chambers (CSCs). The CSCs flag all chambers with high hit multiplicity, and the endcap muon trigger processes this information and selects high multiplicity showers with a high quality. High multiplicity triggers (HMTs) are implemented in the GT and further used at the HLT.

The Global Trigger

The GT processes the L1 calorimeter and muon trigger system information and provides a global decision on whether to save an event at the L1T. While the hardware of the GT has not changed since Run 2, the flexibility of the L1T has allowed for the addition of a test crate in Run 3, which contains the same hardware and is used for testing and development purposes. Importantly, this test crate provides the capacity to test new Run 3 trigger menus and new Phase 2 algorithms, including anomaly detection for the High-Luminosity LHC. This is a new feature of the CMS L1T. Furthermore, the GT can now perform triggering on vector boson fusion processes, which helps improve the analysis sensitivity.

The GT can also now make two kinematic computations: the three-body invariant mass, and the ratio of the invariant mass of two objects to their separation ΔR = √(Δη² + Δφ²). The three-body invariant mass is used in Run 3 to target the decay of a τ lepton into three muons, allowing lower muon p_T thresholds and wider acceptance in η. The mass/ΔR ratio is used to improve a low-mass dimuon resonant search for dark photon signals.

The L1 Scouting System

For Run 3, the L1T project added dedicated hardware for the triggerless recording of objects reconstructed in the L1T, at a rate of 40 MHz, without filtering. The information is received by dedicated FPGA-based processing cards, and is then provided to a computing farm via Ethernet. The data is stored in the form of objects reconstructed by the L1T system and not with the full offline CMS reconstruction, therefore having a worse resolution. This triggerless data recording is referred to as “40 MHz scouting” or “L1 scouting.” The L1 scouting system deployed for Run 3 is meant as the first large-scale demonstration of a larger system planned for the Phase 2 L1T Upgrade [3]. Besides providing a testing ground for the implementation of the readout, it provides the opportunity to study large-scale distributed processing of high-rate data.

The High-Level Trigger

The HLT is a software-based system in which the full event information is used to select events of interest based on their physics content. Since the start of Run 3, the HLT has made use of graphical processing units (GPUs) in the trigger filter farm. Several reconstruction modules were developed to take advantage of these heterogeneous computing architectures and to meet the challenging LHC conditions. Algorithms implemented to run on both central processing units (CPUs) and GPUs are automatically directed to run on a GPU if one is available; otherwise, the CPU-based version of the algorithm is executed. The HLT can offload the track reconstruction based on the pixel detector and parts of the calorimeter reconstruction to GPUs. In particular, the Patatrack project [4] developed parallelized versions of pixel track and vertex reconstruction algorithms that can run on NVIDIA GPUs, while a collaboration between CMS and OpenLab ported the electromagnetic and hadronic calorimeter local reconstruction algorithms to GPUs. Based on these efforts, the overall event processing time has been reduced by about 40%.

Similarly to the L1T, the individual triggers are configured in an HLT menu, typically containing around 600 triggers in 2022 and 2023. The high-level triggers are built using the physics objects reconstructed from all subdetectors, including the inner tracker. A foundation of the HLT object reconstruction is the particle-flow (PF) algorithm [5], which uses information from all these systems to identify charged and neutral hadrons, electrons, photons, and muons. The following will briefly highlight the object reconstruction algorithms that have been most improved for Run 3.

HLT Object Improvements

For Run 3, the tracking in the pixel and strip trackers was significantly revised. Tracking is now performed using a single global iteration, as opposed to the three iterations that were used in Run 2, and the pixel tracks are reconstructed by the Patatrack algorithm mentioned above. This algorithm offers improved performance over the pixel tracking used in 2018 [4]. Figure 2 shows the Run 3 tracking efficiency and fake rate as determined from simulated ttbar events.

Figure 2: Tracking efficiency for the Run 2 HLT tracking (blue) and the Run 3 HLT single-iteration tracking (red), as a function of the simulated track p_T (upper left) and track eta (upper right). The tracking fake rate (lower) is shown as a function of the reconstructed track eta for the Run 2 HLT tracking (blue) and the Run 3 HLT single-iteration tracking (red). Taken from Ref. [6].

Furthermore, the identification of b jets at the HLT is essential to collect events containing such jets that would otherwise not pass the standard lepton, jet, or missing transverse momentum (MET) triggers at their nominal thresholds. Two new neural network taggers, DeepJet [7] and ParticleNet [8], were deployed in 2022, with improved performance over Run 2. In addition to tracks, the DeepJet algorithm also uses information from neutral and charged PF jet constituents, while the ParticleNet algorithm provides a multinomial jet-flavor classification for categories of b, c, and light quarks, gluons, and hadronically-decaying taus. Figure 3 shows the light-flavor jet misidentification rate versus the b jet efficiency for the different tagging algorithms. Compared to the performance of the DeepCSV algorithm used in Run 2, the DeepJet algorithm has a light-flavor jet misidentification rate that is lower by about a factor of 3, up to efficiencies of about 0.75. The ParticleNet algorithm reduces the misidentification rate by another factor of 2.5.

Figure 3: The light-flavor jet misidentification rate versus the b jet efficiency for the various b tagging algorithms, evaluated on simulated top-quark pair production events. The solid curves show the performance of the DeepCSV (blue), DeepJet (red), and ParticleNet (magenta) algorithms in the HLT. The dashed curves show the corresponding offline performance for the DeepJet (red) and ParticleNet (magenta) taggers, using offline reconstruction and training. Taken from Ref. [6].

New High-Level Triggers for Long-Lived Particles

In addition to the improved reconstruction algorithms, the Run 3 HLT menu has been significantly expanded to explore new and unconventional physics signatures such as LLPs.

There are now several flavors of displaced and delayed jet triggers available at the HLT in Run 3. Some of the displaced jet triggers were already available to a certain extent in Run 2, but have gone through major improvements over time, while the delayed jet triggers are completely new for Run 3. The signal efficiency for the displaced di-jet triggers was improved, targeting low-mass LLPs and making particular use of the new L1 triggers that use HCAL timing and depth information. There are also new HLT triggers that exploit the timing of the electromagnetic calorimeter. For LLPs that produce jets with delays of about 1 ns or more, the signal efficiency is improved by an order of magnitude, with respect to the MET triggers that were available in Run 2.

Neutral LLPs with particularly long lifetimes could decay beyond the calorimeters, creating a high-multiplicity shower in the muon system. Such showers are expected to consist of hundreds of hits, but no tracks or jets reconstructed in the inner detectors. As mentioned earlier, new HMTs at L1 have been developed to collect these high-multiplicity events in the CSCs and provide input to triggers at the HLT. Several high-multiplicity triggers at the HLT were developed, reconstructing clusters in both the CSCs and the drift tubes. As compared with the MET triggers that were available in Run 2, the trigger efficiency for these unique signals is improved by a factor of 3 to 20, depending on the trigger.

Triggers for displaced muons at the HLT have also been improved in Run 3. Displaced dimuon high-level triggers take L1 muons with low p_T thresholds and the new displaced double muon triggers described above as input. These L1 muons feed into several types of displaced dimuon triggers at the HLT, designed to cover a wide range of displacements and improve the signal efficiency over that of Run 2. The efficiency of the most displaced dimuon triggers in 2022 data was measured to be 100% for displacements smaller than 100 cm and 90% for d_xy < 350 cm. These displaced dimuon triggers are used in the first CMS search with Run 3 data, namely, the search for long-lived particles decaying to a pair of muons in pp collisions at √s = 13.6 TeV with 2022 data [9]. Because these triggers were improved, this Run 3 search provides results that are comparable with the Run 2 search, despite the Run 3 search having much less integrated luminosity (36.7 fb^-1 in Run 3, compared to 97.6 fb^-1 in Run 2).

Data parking

One limiting factor for the HLT output rate is the affordable rate of the offline event reconstruction. An alternative approach to increase the amount of data, known as “data parking”, is to increase the storage rate to disk, while delaying the data reconstruction until computing resources are available.

Data parking was already implemented in Run 1 [10] and Run 2 to record data for B physics and other studies. In 2018, the collection of bbbar events was expanded by storing events containing at least one displaced muon, e.g., from a semileptonic B decay. That year, CMS accumulated about 10¹⁰ B hadron events [11].

In Run 3, data parking still targets B physics, but it also includes a rich set of other physics data. 

At the end of 2022, the parking streams recorded events with a single displaced muon, events with two low pT muons, and events with two low pT, central electrons. In 2023, the parking strategy was extended by, for example, improving the purity of the dielectron triggers, such that now CMS also triggers on events with two b-tagged jets, Higgs events produced via vector boson fusion, and LLP signatures. In Fig. 4 (left), the trigger efficiencies for prompt and parking data are shown, for signal events with two b-tagged jets. A clear improvement of more than 20% is observed with respect to Run 2. Measurements in 2023 data confirm a plateau efficiency of > 90% (Fig. 4 (right)).

Figure 4: Left: comparison of the trigger efficiency of the HH → bbbar trigger as a function of the diHiggs invariant mass, among the three different strategies used in Run 2 (black), 2022 (blue), and 2023 (orange), using signal MC events. Right: trigger efficiency of the HH → bbbar trigger using events collected by the single muon trigger in 2023. The efficiency is shown with respect to the mean of b-tagging scores of the two most energetic jets in the event (y-axis) and the scalar sum of the transverse momenta of all jets passing minimal kinematic requirements (x-axis). Taken from Ref. [6].

Data scouting at the HLT

A limiting factor for the data-acquisition rate is the bandwidth of the data to record to disk (a few GB/s). Thus, if the size of the data per event and the offline processing requirements are reduced, a higher rate of events can be recorded, for instance by significantly lowering the trigger thresholds. In the so-called “HLT data scouting”, only the most relevant physics information is stored. Scouting was first implemented in Run 1 and was developed further during Run 2, for selected physics objects, such as jets and dimuons.

For Run 3, a special version of the PF reconstruction algorithm using pixel tracks reconstructed with Patatrack was deployed. As a result, in Run 3, HLT scouting data were recorded at an increased rate of 30 (22) kHz in 2022 (2023) and with an event size of about 7 kB, compared to the full raw event size of about 1 MB. In addition to the PF objects that were already stored during Run 2, HLT scouting in Run 3 was expanded to include improved electrons, photons, and tracks with respect to those available in Run 2.

Since HLT scouting is able to access low-momentum objects with higher rates than conventional HLT triggers, it is well suited for analyses targeting low-momentum and low-mass particles. Current studies include analyses of low-mass dimuons, diphotons, and dielectrons as well as an H → bbbar analysis which benefits from the decreased threshold on HT.

Summary and Outlook

The CMS trigger system has been improved for Run 3, both at the Level-1 Trigger and at the High Level Trigger. New approaches were developed to collect more data from rare and interesting physics signatures including B hadron decays, low-momentum particles, Higgs bosons, and potentially new particles, particularly long-lived particles. A significant expansion to the unique CMS parking and scouting programs was critical in this effort.

Further enhancements to the trigger, making increased use of tools such as machine learning, are planned for 2024 and 2025, as well as for the Phase 2 upgrade. In particular, the L1T 40 MHz scouting system in Phase 2 has the potential to dramatically broaden the physics sensitivity of CMS.

References

[1] CMS Collaboration, “Performance of the CMS Level-1 trigger in proton-proton collisions at s = 13 TeV”, JINST 15 (2020) P10017. arXiv:2006.10165. doi=10.1088/1748-0221/15/10/P10017.

[2] CMS Collaboration, “Displaced BMTF Efficiency Using 2023 Data”, CMS DP-2023/056, 2023. https://cds.cern.ch/record/2868797

[3] CMS Collaboration, “The Phase 2 upgrade of the CMS Level-1 trigger”, CMS Technical Proposal CERN-LHCC-2020-004, CMS-TDR-021, 2020. https://cds.cern.ch/record/2714892

[4] A. Bocci, V. Innocente, M. Kortelainen, F. Pantaleo, and M. Rovere, “Heterogeneous reconstruction of tracks and primary vertices with the CMS pixel tracker”, Front. Big Data 3 (2020) 601728. arXiv:2008.13461. doi=10.3389/fdata.2020.601728.

[5] CMS Collaboration, “Particle-flow reconstruction and global event description with the CMS detector”, JINST 12 (2017) P10003. arXiv:1706.04965. doi=10.1088/1748-0221/12/10/P10003.

[6] CMS Collaboration, “Development of the CMS detector for the CERN LHC Run 3”, arXiv:2309.05466, 2023. Submitted to JINST.

[7] E. Bols, J. Kieseler, M. Verzetti, M. Stoye, and A. Stakia, “Jet flavour classification using DeepJet”, JINST 15 (2020) P12012. arXiv:2008.10519. doi=10.1088/1748-0221/15/12/P12012.

[8] H. Qu and L. Gouskos, “ParticleNet: Jet tagging via particle clouds”, Phys. Rev. D 101 (2020) 056019. arXiv:1902.08570. doi=10.1103/PhysRevD.101.056019.

[9] CMS Collaboration, “Search for long-lived particles decaying to a pair of muons in pp collisions at √s = 13.6 TeV with 2022 data”, EXO-23-014 (2023). https://cds.cern.ch/record/2868338.

[10] CMS Collaboration, “The CMS trigger system”, JINST 12 (2017) P01020. arXiv:1609.02366. doi=10.1088/1748-0221/12/01/P01020.
[11] CMS Collaboration, “Recording and reconstructing 10 billion unbiased b hadron decays in CMS”, EPJ Web Conf. 245 (2020) 01025. doi=10.1051/epjconf/202024501025.