Computer Security: in numbers

CERN is under attack. During working hours. On weekends. During vacations. Throughout annual closures. CERN is under attack. Permanently. Its Internet and web presence are constantly being probed for weaknesses and vulnerabilities. Its computing accounts are being targeted by phishing attacks or social engineering. Individual end-devices, laptops and PCs are vulnerable to hackers. While it is hard to quantify the number of daily attacks¹ CERN is subject to, we can have a look at CERN’s attack surface and the different measures the Computer Security team takes to protect that surface.

If you check out CERN IT in numbers, you’ll be able to fully appreciate its digital vastness and, hence, attack surface! The CERN network, its several public Class-B IPv4 & IPv6 networks, plus many non-routable ones, is comprised of 4000 switches, 300 routers and 5000 wireless access points. It serves more than 270 000 registered devices with 20 Gb/s flat bandwidth. Given CERN’s “Bring-Your-Own-Device” (or BYOD) policy, many of those devices are made of arbitrary hardware (laptops, tablets, PCs, embedded systems, IoT, etc.) and run arbitrary operating systems from Windows 95 via DSPs, Xilinxes and SoCs, to any flavour of Linux as well as any imaginable software program. These devices are linked to 37 000 computing accounts with attached CERN mailboxes.

CERN runs seven computer centres with up to 6000 servers/200 000 CPU cores each, over 30 000 different virtual machines, and a data storage capacity exceeding 1 exabyte provided by more than 110 000 hard disks/SSDs and 50 000 tapes. Those data centres provide all the computer and storage capacity as well as the IT services and infrastructure needed for the operations of CERN’s accelerator complex and attached experiments, for the processing and analysis of recorded physics data, and for the general needs of CERN’s researcher community (such as the provisioning of operating systems, office/engineering/HR/finance apps, databases, versioning/build systems, virtualisation and container services, collaboration tools, video/audio conferencing, printing, etc.). The web service alone serves centrally more than 10 000 websites/1.5 million webpages. More than 500 decentralised web servers provide additional sites and pages. 

All these resources need to be monitored and protected. As coordinator, the CERN Computer Security team must fulfil its duty to prevent, protect, detect and respond in a way that does not obstruct the operations of CERN's accelerators and experiments or limit the academic freedom of our personnel and user community. Here’s how we do it:

• Starting network-wise, CERN’s outer perimeter firewall (the PaloAlto 7080 beast with 2x200 Gbps throughput) performs deep packet inspection and policy-based
  IP, domain and URL blocking. And on the Domain Name Server (DNS) level, we block an additional 800 known malicious domains and typo-squatting domains (like “CERN.CG”), with our upstream network provider, SWITCH, adding another 100 verified malicious domains per day. Given its protective power, the same DNS filtering has also been successfully applied at over 50 Swiss hospitals in collaboration with the Swiss government, as well as at a number of French hospitals, with none of them having yet fallen victim to a ransomware attack;
• The CERN Computer Security Operations Centre (SOC) automatically analyses about 3 billion network connections and 1.2 billion DNS requests daily, as well as several hundreds of thousands of logins and 4 billion commands executed on CERN’s central computing clusters. These 3 TB per day amount to about 220 TB of archived computer security data (1 PB uncompressed) stored with a 13-month retention period;
• The SOC’s capabilities are based on threat intelligence, or “Indicators-of-Compromise” (IoCs); so far, the SOC has detected as many as 22 million malicious IPs, domains, and file hashes, which we share with over 1000 peer organisations. About 100 000 IoCs are currently being actively monitored in CERN’s SOC;
• Endpoints, i.e. laptops, smartphones, etc., are also monitored separately using local anti-malware software. Our pick, “ESET”, is currently being used to protect more than 500 BYOD devices and more than 200 CERN centrally managed Windows PCs (more to come). This is complemented by the “Threatray” application, which is running on over 2700 CERN-owned Windows PCs (and number increasing);
• All those devices – computer-centre hosted, centrally-managed or BYOD – are scanned about once a month for vulnerabilities, insecure passwords and other weaknesses. Totalling about 24 000 scans per month, this generates more than 200 issues to be fixed by the corresponding device owners;
• On the training and awareness side, we give about one awareness session a month to newcomers and interested folks. So far, we have trained more than 100 CERN personnel as well as hundreds of external students in the CERN WhiteHat program to become penetration testers, and ran five very successful table-top exercises─ some with the participation of real police forces from Geneva and Gex. And, of course, so far more than 325 articles have been submitted to the CERN Bulletin. You are reading one of them right now;
• Still, account protection is paramount. As you know, two-factor authentication has been set up on over 6000 CERN accounts. In parallel, we send about 4770 email notifications to people who recently logged into CERN from an ─ to us ─ unusual location. And we check whether CERN passwords as well as external passwords used with your CERN email address have been published on so-called dark web password dumps. In fact, since we are effectively sitting on a treasure trove of billions of such email/password combinations, we also monitor about 9000 affiliated universities, institutes and organisations and notify them of approximately 11 000 leaked passwords every day;
• In parallel, the SPAM and malware filter services (Microsoft’s EOP & MDO as well as Xorlab “ActiveGuard”) analyse around 115 000 email communications to and from @cern.ch email addresses per day, of which 10% are rejected as SPAM and 2% are quarantined as potentially malicious;
• CERN’s Computer Security Incident Response Team (CSIRT) is on hand if something goes wrong; it automatically sends out about 300 alerts per month to users and device and account owners informing them about security problems with their assets. In return, they create about 200 tickets with the Team asking for assistance and help. Luckily, the majority of those incidents are local and harmless. The more major incidents at CERN remain infrequent (about 5–10 per year). However, this doesn’t mean we are twiddling our thumbs: the Team assists many external universities, institutes and organizations in resolving their computer security problems before they spill over into CERN. This year, for example, several dozen universities were spared from ransomware attacks thanks to intelligence obtained by our team; 
• Finally, on the prevention side, we conduct between 20 and 30 in-depth security reviews and audits of new and old CERN projects annually. Feel free to reach out to us at Computer.Security@cern.ch to get your project assessed.

Voilà… a few figures on CERN Computer Security with one more essential number: seven. Romain, Liviu, Luna, Christos, Roman, Pau, and Srividya ─ the names behind the numbers helping you, together with many more inside the IT department, to perform your work as securely as possible without hindering your creativity, flexibility, operations and academic freedom too much (we hope). Thank you, guys!

Do you want to learn more about computer security incidents and issues at CERN? Follow our Monthly Report. For further information, questions or help, check our website or contact us at Computer.Security@cern.ch.