I had a big smile on my face on the evening of Friday, 21 October 2016, when I saw how quickly the CERN IT department, the LHC experiments, teams in the accelerator sector and many more individuals were rushing to secure their Linux systems against a new and highly critical vulnerability dubbed “DirtyCow” (i.e. CVE-2016-5195). ArsTechnica labelled this bug the "most serious Linux privilege-escalation bug ever", which stresses its severity nicely, and it was too risky to go into the weekend unprotected!
It seems that computer security problems tend to occur at weekends. “DirtyCow” was a particularly nasty one that, when exploited, allows any local user to inherit administrator privileges and, subsequently, become master of the corresponding Linux system. Although CERN’s SLC5 and 6 were said to be unaffected, a few brave members of the IT department spent the Thursday evening analysing the exploitation vector in depth and finally disproved this initial statement: it turned out that SLC5 and 6, as well as CentOS7, were very much affected… Unfortunately, a prompt patch was not immediately available, so the security risk was uncomfortably high for the CERN Data Centre, its interactive computing clusters – namely LXPLUS and LXBATCH – and many other interactive Linux services in the experiments and the accelerator sector. The risk was especially high as the weekend lay ahead.
Fortunately, however, the IT department was able to propose a mitigating workaround as a temporary protective measure. Intense hours were spent on Friday preparing new Linux “system-tap kernel modules” and proving that the impact on Linux systems was minimal (in fact, only debugging functions would be affected). Finally, at around 3 p.m., the green light was given for the massive roll-out to thousands of Linux LXBATCH servers and hundreds of LXPLUS servers in the CERN Data Centre. An official warning was sent out to all relevant stakeholders at CERN, including SWAN, ATLAS, CMS and others, who promptly applied the workaround to their systems. By late night, all critical services had been secured and were ready to run through the weekend. Great job, CERN! Congratulations to you all!
Addendum: The workaround is no longer needed. CVE-2016-5195 can be fixed by deploying the most recent kernel version available from CERN Puppet or the YUM repositories. Time to bring your system up to date!
For further information, questions or help, check our website or contact us at Computer.Security@cern.ch.
Do you want to learn more about computer security incidents and issues at CERN? Follow our Monthly Report.