In past days, the CDS website experienced an unprecedented Distributed Denial of Service (DDoS) attack. A DDoS is a malicious attempt to disrupt the normal traffic of a targeted server by overwhelming the target with a flood of Internet traffic. You can find here more information.
What happened
On Monday, April 22, 2024, at approximately 11:30 AM, our monitoring systems detected a significant increase in traffic to the website. Analysis of the incoming HTTP requests revealed a Distributed Denial of Service (DDoS) attack. As a result, the CDS website became inaccessible to most users.
As an initial response, we promptly reached out to the CERN Computer Security Team for assistance. Furthermore, we posted an announcement regarding the incident on the CERN Service Portal Status Board, referencing number OTG0149709. Additionally, we communicated the incident via our official Mattermost channel, accessible to CERN users, and we added an informational banner to the website for those who were able to access it.
First actions
We have quickly realized that mitigating the attack would take longer than anticipated. At 12:30 PM, we made the decision to restrict access to the website solely from within the CERN network. This measure ensured that CERN users could still access the website while allowing us to concentrate on implementing countermeasures.
Around 3:30 PM, it appeared that the attack rate had decreased. In collaboration with CERN Computer Security Team, we made the decision to reopen access to the website from outside the CERN network. However, less than an hour later, the attack resumed, with an even higher volume of traffic. We decided to close access again.
Resolution
Due to the overwhelming majority of incoming requests originating from a specific geographical location, we made the difficult decision to block access to the website from that entire area. Simultaneously, we reinstated access from outside the CERN network. This countermeasure was implemented on Tuesday, April 23, 2024, at approximately 3:30 PM. As an additional security measure, we completely disabled IPv6 connections. All operations have been performed by CERN Computer Security Team in collaboration with us and the Network team.
The restrictions on users accessing CDS from certain locations will remain in place until we confirm that the attack has ceased. We are continuously monitoring incoming traffic in order to lift these restrictions as soon as possible.
The reasons and specifics behind this attack targeting CDS remain unclear. We have provided all available logs and information to the CERN Computer Security Team, who will conduct the necessary investigations and take appropriate actions.
Next steps
As it is the first time that we experience such a large-scale and distributed attack, it’s evident that we were unprepared. However, this experience has provided valuable insights and lessons for both our team, the CERN Computer Security and Network teams. We’re actively leveraging these takeaways to enhance our infrastructure and ensure readiness for any future occurrences.
While the CERN Computer Security and Network teams are currently analyzing logs and enhancing detection and mitigation tools to accelerate response times, our immediate focus will be on improving our alarming systems. Additionally, we are prioritizing enhancements to our DDoS protection mechanisms. Furthermore, efforts are underway to establish a reliable internal infrastructure as a contingency in the event of external compromise, ensuring continued access to the website for CERN users.
More technical details
During the initial stages of the DDoS attack, we observed a traffic volume of roughly 5,000 requests per minute. However, the incoming traffic within the CERN network was constantly increasing (we observed the number of 20,000 requests per minute, and growing UPDATE: we observed the number of 3.5M requests per hour).
While these figures may not seem excessively high, the CDS infrastructure is not designed to handle such volumes, as we aim to avoid over-sizing the infrastructure when unnecessary. By comparison, traffic on CDS typically reaches around 500 requests per minute, with peaks of 1,000 requests per minute.
Despite implementing counter-measures such as blocking numerous IPs or scaling up our infrastructure to accommodate more traffic, the number of requests continued to escalate during the attack. It appeared that the attacker was capable of increasing the size of the attack.
A sophisticated attack
Implementing counter-measures for attacks of this scale is challenging. The attack vector exhibited a high level of sophistication.
Here is an example of a single HTTP request (with the IP address masked):
<masked ip> - - [23/Apr/2024:12:59:59 +0200] "POST /6270607l7c07z7ldmt031x/6270607l7c07z7ldmt031x-6270607l7c07z7ldmt031x/ HTTP/1.1" 404 "-" "Mozilla/5.0 (Linux; U; Android 12; V2027 Build/SP1A.210812.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/123.0.6312.118 Mobile Safari/537.36 OPR/77.0.2254.69831" 0 1246 16756As you can observe, identifying a consistent pattern to safely and effectively distinguish between legitimate and malicious traffic is challenging due to the following reasons:
- The URL path and HTTP verb appeared to be entirely random, with most requests resulting in a 404 error.
- The User-Agent was generated randomly.
- We detected over 46,000 different IPs originating from various locations.
- Each IP was responsible for a relatively low number of requests.
It was also unexpected to discover that attempting to block a large number of IPs could put pressure on many software components in the infrastructure.
Conclusion
Access to the website was restored within a few hours, successfully stopping the attack. However, this DDoS attack is still ongoing, and access to CDS from certain locations will remain blocked until it stops.
It is now even clearer to us that defending against such attacks requires a high level of expertise and investment in robust infrastructure and tools.
As service providers, we are grateful for the expertise and competence of the specialized teams at CERN. Their dedication ensures that we can effectively address challenges and maintain the reliability of our services.
