At 1 a.m. on Monday, 17 July, the LHC beams were dumped due to an electrical perturbation. Approximately 300 milliseconds later, several magnets lost their superconducting state (“quenched”). During a quench, the magnet warms up, which in turn warms and pressurizes the liquid helium that surrounds it.
While not common, this sequence of events is expected to happen to protect the superconducting cable of the magnet when an electrical glitch occurs; the mechanical stress exerted on different parts of the magnet can be quite strong.
Among the magnets that quenched on 17 July were the inner triplet magnets located to the left of Point 8 of the LHC, which play a crucial role in focusing the beams for the LHCb experiment. Unfortunately, this time, the quenches led to a helium leak in these magnets and stopped regular LHC operations.
Scroll through the photo diary below to re-live the ten day race against the clock to successfully repair the leak.
Monday 17 July, 1 a.m.: ROOT CAUSE
The reason for the electrical glitch that caused the safety systems in the LHC to dump the beam and several magnets to quench was found: a tree on the Swiss side (about 55 km from CERN in the Canton of Vaud) fell on the power lines and perturbed the power system. Monday 17 July, 11 a.m.: A CHILLING DISCOVERY
Ten hours later, on entering the tunnel, the investigating team found that the cryostats* of the triplet magnets near Point 8 were partly covered in ice. Tests quickly confirmed that a small amount of helium had escaped through a leak and filled the insulation vacuum.
Action was taken immediately: the adjacent magnets were electrically isolated, circuits were locked-off and grounded, and the quench heaters for this sector switched off. Additionally, to be able to work on the triplet, the 3 km of superconducting magnets in the affected sector were stabilised at a temperature of 20 K, instead of their usual 2 K (-271°C).
*All LHC superconducting magnets are housed in cryostats. Under normal operation the external wall of the cryostat is at room temperature, whilst the magnet operates at 2 K. The cryostat is designed to maintain the magnet at such a low temperature by minimising in-flow of heat – and insulation vacuum is essential to achieve that. Tuesday 18 July - Wednesday 19 July: LOOKING FOR THE LEAK
The exact position of the helium leak in the 50 m long cryostat was still unknown. By Tuesday 18 July, vibration and acoustic tests had been performed. Attaching accelerometers and microphones, the intervening team detected a clear signal at the interconnection zone between the first quadrupole magnet (Q1) and the second quadrupole magnet (Q2). Additional x-ray scans showed that the spacing of the bellows ridges on one of the pipes in the superconducting magnets appeared stretched. Bellows are employed in the physical connections between two magnets, giving flexibility. In this case the stretched bellows was on the M2 pipe, which contains the instrumentation connections. Thursday 20 July - Sunday 23 July: PREPARING TO OPEN
The intervening teams agreed that the Q1-Q2 interconnection between the two quadrupole magnets would have to be opened for further investigation and repairs. To make a safe intervention possible, the sector around the magnets was emptied of the liquid helium. In parallel, an electrical quality assessment showed that the electrical circuits of the triplet were fine – the problem was thus elsewhere.
Teams of experts from different CERN groups (safety, vacuum, cryogenics, magnets, engineering, powering, magnet protection, survey, beam instrumentation, operations) discussed how to tackle a problem never encountered before on a 15-year-old string of magnets, creating a procedure on the spot. Monday 24 July: OPENING THE TRIPLETS
The whole triplet cryostat reached room temperature. The external bellows and inner thermal shields at the Q1-Q2 interconnection were removed to inspect the inner helium lines. Monday 24 July: GOTCHA!
The leaky bellows in the M2 pipe. Monday 24 July, afternoon: SMALL BUT MIGHTY
The teams located the suspicious M2 bellows and indeed, there was a crack on it: with a length of 1.6 mm, it was the source of the helium leak. An action plan was put into place: remove the broken bellows, replace it, go through all necessary tests, close again and start the cool down…
…all in under 10 days. Otherwise, a complete warm-up of the affected LHC sector could not be prevented, and this would stop the whole LHC physics programme for 2023. Tuesday 25 July: WORKING THE BELLOWS
While the broken bellows was cut out, the vacuum team conducted pressure and leak tests on spare bellows to test their resilience and provide a replacement unit for the tunnel repair. Thursday 27 July: TEAMWORK...
Experts led by Sandrine Le Naour and Said Atieh discussed the possible repair solutions on site. ...MAKES THE DREAM WORK
The new bellows was installed. On the far left: threading the instrumentation through the new bellow. In the middle: many hands make light work! On the right: skilled welders do their magic. ILLUMINATED
Graeme Barlow looking at the open interconnection, with the various pipes inside visible. The M lines allow the helium to be transported between magnets (M1 contains the busbar for electrical connection, M2 contains the instrumentation connections, and M4 has a cryogenic function). In the middle sits the beampipe where the particles circulate. The M2 bellows is just visible between the M1 and the beampipe. THE REPAIR ZONE NEXT STEPS
The vacuum and mechanical teams discussed the action plan during ongoing repairs. WORK IN PROGRESS
Often, two teams were at work at the same time: on the left, reinstalling beam position monitor (BPM) cables, on the right, starting the leak test on the new bellows. LEAKPROOF
The vacuum team installing the leak test tooling. Graeme Barlow of the vacuum team installing the leak test machine with Paul Cruikshank. Paul Cruikshank, leading the LHC vacuum intervention, together with his team, starting the leak test on the newly installed bellows. RECONNECT
During the opening of the Q1-Q2 interconnection, the beam position monitoring (BPM) cables had to be removed. Here, the cable reinstallation is ongoing. The reinstalled bellows required several new welds. Each required a dedicated leak test to avoid any leak surprises once the interconnection was reclosed.
Sandrine Le Naour (far right) assessing the progress. She coordinated the mechanical interventions to open the magnet interconnections and then had to prepare the careful reclosure. Paul Cruikshank with the vacuum team and expert welder Didier Lombard. On the right, a closeup view of a clam shell leak testing tool (technology developed at CERN to leak test the entire LHC during its installation) used by his team once more for leak testing of new welds. This CERN-born innovation (also used by industry today) enables checking the vacuum quality of a tube from the outside, which is a great advantage when the objects to test are very long and difficult to evacuate, very typical in the 27 km long LHC vacuum systems. FINAL LEAK TEST UNDERWAY...
Wim Maan and Marcel Knoch checking the tightness of the final weld. FULLY REPAIRED
M2 bellows fully repaired. The bellows is surrounded by external shells to support and guide it when the helium is pressurised during different operational phases of the LHC. TEAM SUCCESS!
The bellows is repaired and the leak test was successful within the ten day deadline. Although there’s still plenty to do to reclose the interconnection, the light at end of the tunnel is in sight! After the teams repump the vacuum and cool down the magnets, the LHC can restart.
The LHC operations team is confident to see the first beam back in early September.
For more information on this story, watch a video interview with Paul Cruikshank, one of the coordinators of the repair operation:
