CERN Accelerating science

News, announcements and future developments of CDS services at CERN

Category: CDS

Updates on the new CDS

We’re excited to announce a set of new features and improvements to the new CERN Document Server (CDS), now running on InvenioRDM v12! Our focus has been on making CDS more user-friendly and efficient for everyone.

The long list of all changes is detailed in the full changelog. Below, the main highlights more relevant to the CERN community.

What’s new?

Enhanced Search Accuracy

We’ve improved the search functionality to make finding records easier and more precise. Whether you’re looking for specific datasets or publications, the search engine is now optimized to deliver more accurate results, helping you locate the content you need faster.

Mathematical Formula Rendering

For researchers working with complex formulas, CDS now nicely renders LaTeX formulas within search results and records.

Content Policy and Terms of Use

To ensure transparency and safeguard users, we have added a Content Policy and Terms of Use to the platform. These documents clarify the rules regarding content submission and usage on CDS, helping maintain a safe and collaborative environment.

Introduction of sub-communities

One of the biggest updates is the feature of sub-communities. Communities on CDS can now be nested, meaning a community can have a parent community. This change brings more flexibility in organizing and structuring data, catering to the diverse needs of departments and research groups.

User Interface Tweaks

We’ve implemented a series of UI improvements, including fixing issues with community logos and adding enhanced loading icons during login and logout among other fixes. These small but impactful updates make the interface more user-friendly and visually coherent.

Migration of the 1st collection

Under the hood, we are working hard to migrate the very first collection of documents from the current CDS repository to our new platform. We are now in the process of testing the migration of documents and files, and ensuring the correct redirection of web links.

This is a very important milestone: it will prove that the migration processes that we have put in place are working as expected, and it will unblock the migration of the next collection of documents.

What’s next?

Our team is already working on new features for future releases:

Collections

We are developing a way to categorize records easily within a community (or independently) based on metadata. This allows users to organize and navigate records more intuitively.

Automatic Ingestion of ORCID and ROR Values

To save time and streamline workflows, we are working on automating the ingestion of ORCID and ROR data into the system, ensuring that author and organization identifiers are up-to-date without manual input.

Integration with CERN users database

We are working on making CERN users findable when searching for authors or collaborators during an upload.

Stay tuned for more updates, and as always, feel free to share your feedback with us!

The new Web Lectures video player

Today is an exciting day at the CDS website! We’ve been working hard behind the scenes to bring you some exciting improvements to your user experience. As of this moment, you can start enjoying the benefits of these enhancements:

Brand new Web Lectures player: thanks to the fantastic work of our colleagues from the Webcast and Recording service, we’ve said goodbye to the previous Web Lectures player (based on THEOplayer and JW Player) and welcomed a new video player, powered by Paella. This sleek, modern player is designed to make your video-watching experience smoother and more enjoyable than ever before.

Seamless Access: You’ll notice a significant change in how you access the restricted Web Lectures. No more redirections or weird pop-ups asking for credentials—restricted videos now display seamlessly, thanks to the integration with the CERN Single Sign-On (SSO).

Enhanced Download Options: We’ve always offered download options, and now it’s even easier to distinguish between lecture slides and presenter recordings. Also, subtitles are now available for download as well.

Smoothly Embedded Videos: If you embed videos from the CDS platform, you’ll experience a seamless transition to the new video player.

A Post-Mortem analysis on the recent DDoS attack

In past days, the CDS website experienced an unprecedented Distributed Denial of Service (DDoS) attack. A DDoS is a malicious attempt to disrupt the normal traffic of a targeted server by overwhelming the target with a flood of Internet traffic. You can find here more information.

What happened

On Monday, April 22, 2024, at approximately 11:30 AM, our monitoring systems detected a significant increase in traffic to the website. Analysis of the incoming HTTP requests revealed a Distributed Denial of Service (DDoS) attack. As a result, the CDS website became inaccessible to most users.

As an initial response, we promptly reached out to the CERN Computer Security Team for assistance. Furthermore, we posted an announcement regarding the incident on the CERN Service Portal Status Board, referencing number OTG0149709. Additionally, we communicated the incident via our official Mattermost channel, accessible to CERN users, and we added an informational banner to the website for those who were able to access it.

First actions

We have quickly realized that mitigating the attack would take longer than anticipated. At 12:30 PM, we made the decision to restrict access to the website solely from within the CERN network. This measure ensured that CERN users could still access the website while allowing us to concentrate on implementing countermeasures.

Around 3:30 PM, it appeared that the attack rate had decreased. In collaboration with CERN Computer Security Team, we made the decision to reopen access to the website from outside the CERN network. However, less than an hour later, the attack resumed, with an even higher volume of traffic. We decided to close access again.

Resolution

Due to the overwhelming majority of incoming requests originating from a specific geographical location, we made the difficult decision to block access to the website from that entire area. Simultaneously, we reinstated access from outside the CERN network. This countermeasure was implemented on Tuesday, April 23, 2024, at approximately 3:30 PM. As an additional security measure, we completely disabled IPv6 connections. All operations have been performed by CERN Computer Security Team in collaboration with us and the Network team.

The restrictions on users accessing CDS from certain locations will remain in place until we confirm that the attack has ceased. We are continuously monitoring incoming traffic in order to lift these restrictions as soon as possible.

The reasons and specifics behind this attack targeting CDS remain unclear. We have provided all available logs and information to the CERN Computer Security Team, who will conduct the necessary investigations and take appropriate actions.

Next steps

As it is the first time that we experience such a large-scale and distributed attack, it’s evident that we were unprepared. However, this experience has provided valuable insights and lessons for both our team, the CERN Computer Security and Network teams. We’re actively leveraging these takeaways to enhance our infrastructure and ensure readiness for any future occurrences.

While the CERN Computer Security and Network teams are currently analyzing logs and enhancing detection and mitigation tools to accelerate response times, our immediate focus will be on improving our alarming systems. Additionally, we are prioritizing enhancements to our DDoS protection mechanisms. Furthermore, efforts are underway to establish a reliable internal infrastructure as a contingency in the event of external compromise, ensuring continued access to the website for CERN users.

More technical details

During the initial stages of the DDoS attack, we observed a traffic volume of roughly 5,000 requests per minute. However, the incoming traffic within the CERN network was constantly increasing (we observed the number of 20,000 requests per minute, and growing UPDATE: we observed the number of 3.5M requests per hour).

While these figures may not seem excessively high, the CDS infrastructure is not designed to handle such volumes, as we aim to avoid over-sizing the infrastructure when unnecessary. By comparison, traffic on CDS typically reaches around 500 requests per minute, with peaks of 1,000 requests per minute.

Despite implementing counter-measures such as blocking numerous IPs or scaling up our infrastructure to accommodate more traffic, the number of requests continued to escalate during the attack. It appeared that the attacker was capable of increasing the size of the attack.

A sophisticated attack

Implementing counter-measures for attacks of this scale is challenging. The attack vector exhibited a high level of sophistication.

Here is an example of a single HTTP request (with the IP address masked):

<masked ip> - - [23/Apr/2024:12:59:59 +0200] "POST /6270607l7c07z7ldmt031x/6270607l7c07z7ldmt031x-6270607l7c07z7ldmt031x/ HTTP/1.1" 404 "-" "Mozilla/5.0 (Linux; U; Android 12; V2027 Build/SP1A.210812.003; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/123.0.6312.118 Mobile Safari/537.36 OPR/77.0.2254.69831" 0 1246 16756

As you can observe, identifying a consistent pattern to safely and effectively distinguish between legitimate and malicious traffic is challenging due to the following reasons:

  • The URL path and HTTP verb appeared to be entirely random, with most requests resulting in a 404 error.
  • The User-Agent was generated randomly.
  • We detected over 46,000 different IPs originating from various locations.
  • Each IP was responsible for a relatively low number of requests.

It was also unexpected to discover that attempting to block a large number of IPs could put pressure on many software components in the infrastructure.

Conclusion

Access to the website was restored within a few hours, successfully stopping the attack. However, this DDoS attack is still ongoing, and access to CDS from certain locations will remain blocked until it stops.

It is now even clearer to us that defending against such attacks requires a high level of expertise and investment in robust infrastructure and tools.

As service providers, we are grateful for the expertise and competence of the specialized teams at CERN. Their dedication ensures that we can effectively address challenges and maintain the reliability of our services.

Updates on the new CDS

Summer has already started 😎 and, in the previous months, we have worked hard to integrate the latest development in the new CDS platform.

The result looks beautiful!

The new CDS platform is the brand-new version of the current CERN institutional repository, a modern and easy-to-use website where CERN users can archive and share their research, multimedia content or departmental documents.

You can now preview and try out the latest features in our test instance https://sandbox-cds-rdm.web.cern.ch (reachable from inside CERN campus). Just to mention a few, we have integrated users and groups CERN databases; newly uploaded publications will now have a DOI out-of-the box, ready to be shared and cited; files are securely stored in EOS file system. And there is much more.

The “Browse” section contains links to collections and categories to the former CDS platform: we will slowly migrate data to this brand-new CDS.

The footer of the new CDS website contains useful links to make sure that you will find the information that you need.

The production instance https://new-cds.cern.ch will be soon start to be used by some selected communities at CERN, and we will gather feedback to continuously improve it and make it as easy as possible to use.

After summer, more features will be coming 🚀: we will make it very easy to restrict and share documents with other users, and we will work on the administration panel to fully manage records and users in the system.

This version is just the base for the future CDS. More features will be needed to support all current use cases. To that end, we will be contacting and working together all main users so that we can define together the plan for completion of this future Institutional Repository.

If you wish to, open the new CDS website, login, try it out and share feedback with us!

The new CDS, based on InvenioRDM

With the LTS release (v9) and the latest release (v10), InvenioRDM has reached the maturity needed for production-ready digital repository websites. InvenioRDM is a generic data management repository, developed by our team in collaboration with many partners all over the world. Free to use and open-source.

The InvenioRDM demo website.

As already done by several partners (e.g. Caltech University, TU Graz University, TU Wien University), our team worked hard to create a preview version of the future CDS, available at https://sandbox-cds-rdm.web.cern.ch.

The new CDS website, based on InvenioRDM.

As first milestone, we have created and deployed the new instance of CDS and also migrated a selected set of records, metadata-only. This initial setup will allow us to iterate with the process of data migration, expanding incrementally the number of records and improving the data quality.

In the first quarter of this year, we will continue working on the InvenioRDM product, adding more features and integrating them in the new CDS website.

We will also start an analysis of the feature-set available in the current CDS, but still missing in the new platform: thanks to this, we will be able to come up with a plan for the next steps.

We are very excited to finally see the new CDS taking shape! Stay tuned for future announcements!

Summer is over!

Good and resting holidays and… new features!

New CERN SSO – cds.cern.ch

In September 2022, we have changed the integration from the old CERN SSO login to the new one. This was not only needed in relation to the upcoming decommissioning of the old SSO, but it also brings more security (enabling for example Two-Factor Authentication), more performance and more login possibilities. The recurrent login issues with external accounts are also solved.

In the coming weeks, we will also work to perform the same migration on CDS Videos.

InvenioRDM v10

We have now released InvenioRDM v10. Why is this important? Well, simply because it will be the base and the core software of the future CDS platform!

InvenioRDM v10 comes with support for custom metadata (necessary to store CERN specific fields, such as report number, experiments, accelerators, etc.), a new administration panel to make it easy to manage the instance and support for the new search engine OpenSearch. The latter will be necessary to comply with the standard CERN IT infrastructure.

In this last part of the year, the team will focus on creating a showcase version of the new CDS website, including a new look and feel and automatic deployments. This demo website will be useful to demonstrate features, test user experience and perform dry-run data migration from the current CDS to the new one in an iterative and progressive manner, to make sure that migrated data is correct.

CERN Library Catalogue

During the summer, we have made a bunch of improvements and bug fixes to the CERN Library Catalogue as well! To mention a few, book covers are now beautifully aligned and styled and the search for periodical and serial volumes now shows search tips.

In the context of InvenioILS, we also made it easier to create your own library catalogue website: with a couple of commands, users can set up a new instance and have a running website in a few minutes!

Progress before summer holidays

These first 6 months of the year 2022 have been quite busy. The CDS team focused on the development of InvenioRDM, the future research data management digital repository platform that will be the core of the future CDS version.

At the same time, production services need to be taken care of. While performing maintenance operations, we also worked on bug fixes and some technical features.

Future CDS platform – cds.cern.ch

With a small break of the development activities, we took half a day to discuss and define how we will kick off the new CDS platform: we came up with what we think will be a good strategy, and we hope to have the first dev/test instance by the end of the year, including a small subset of data migrated from the current CDS to the new one.

Even if it will be a long journey, we are very excited to get started!

InvenioRDM v9 and 9.1

The team contributed to the development of the latest 2 releases: InvenioRDM v9.0 and v9.1. These new releases enable curators to create their own communities of documents, to better self-organize content and make it easier for users to submit and find content. Each community is clearly identified, thanks to the new header with the name and the logo.

The way new documents are added to a community is managed via requests: when submitting a new document to a community, a new request is created where the submitted and the community’s curator can have a conversation with commenting. The request can be approved or rejected.

CDS Videos just got faster – videos.cern.ch

We have implemented a new feature in CDS Videos: the post-processing of uploaded videos got 2x faster! This is particularly visible when large video files are uploaded: the first two tasks, the extraction of the metadata, embedded in the video file, and the extraction of the video’s thumbnails now take half of the time.

Maintenance…

Databases for all the services have been upgraded to the latest versions, to ensure safety and security. The transcoding software, used in CDS Videos, have also been upgraded by our colleagues from the Webcast team, so that we can take advantage of the latest fixes and features.

On the 5th of July 2022, CDS suffered some slowness due to a very high amount of traffic, requesting some photos embedded in the CERN Home website. While being slower than normal conditions, the CDS website managed to serve more than a million requests, with a pick of 800 requests/second (thanks to connection queues).

While we are happy to see that our platform was up and running in such unusual conditions, we have identified some improvements, and we are working to change our infrastructure to be more performant when serving files.

What’s next

During summer, many absences are foreseen. The team will continue working on InvenioRDM v10: its features are critical to the future version of CDS. We are also planning to upgrade the web lectures player for an improved user experience when watching recorded talks, seminar or events.

Happy summer vacations! 🏖 ⛰ 🏜

New year, new…

… features and improvements! In the last quarter of 2021 we have been busy as bees, preparing to kick off the 2022 with some big news. Read further to know about the details!

CDS Videos – videos.cern.ch 🎬

Following our plan for the Q4, we have introduced significant changes to the CDS Videos platform. You might not see it at the first glance, but the platform evolved “under the hood”.

Transcoding infrastructure and video processing 📺

Video upload processing view

Transcoding is one of the steps of the video processing performed after you upload your video file to the platform. The transcoding software is responsible for creating several predefined subformats for your video. These subformats are later used to provide improved streaming experience for anyone who is watching your video (after it is published).

The transcoding software previously used by CDS Videos was causing us and CERN users many headaches. Thanks to the collaboration with the CERN Webcast team, who provided the new video transcoding tool OpenCast as a service, we have worked very hard to integrate it. This new software is expected to have increased reliability and incomparable better performance.

We also took a good look into the rest of video processing steps and fixed the commonly reported issues. The processing will be now faster and more consistent – no more videos processing indefinitely in your “Upload” interface! 🎉

User experience tweaks

Last but not least, we have made a few improvements in the user interface, having in mind all the feedback we received from you. A few highlights:

  • We have reworked some explanation text to improve general understanding.
Project editor’s permissions panel
  • The authors and e-groups autocomplete has been improved, giving you access to powerful and more reliable users search – as broad as the CERN Phonebook!
  • The user and e-groups videos restrictions is now case-insensitive.
  • No more weird errors when publishing videos!

CERN Library Catalogue – catalogue.library.cern 📚

And what about the other applications? We worked on them as well!

CERN Library Catalogue’s latest version offers improvements in the user interface as well as librarian’s interface. We follow the latest standards on application security and data privacy, and as a result, we now provide user’s accounts and data anonymization. You can check details of our privacy policy here.

To make things easier, the identifiers in the book’s details page are now clickable.

Book page with external provider hyperlinks

Are you doing a lot of research? Are you in a dire need of articles and periodicals? Check the remodeled “Where to find” section of the periodical page.

Periodical page featuring physical volumes

Librarians’ tools

Our team has also worked on improving the librarians’ catalogue management tools. For example:

  • Better bulk import of e-books
  • Better export of catalogue’s object to CSV files
  • Search tweaks – case-insensitive searches and other

Web infrastructure upgrade 💻

Among of the many changes and enhancements, we had to upgrade the underlying web infrastructure for both services CDS Videos and CERN Library Catalogue.

Both platforms are now hosted on OKD4 cluster provided by our colleagues from Web Infrastructure team. The migration is a good news not only for our users, but also for our developers: the change helps us decrease the effort we have to make to maintain and deploy new code.

CERN Document Server – cds.cern.ch 📄

The last, but not least – CDS. Our biggest and oldest application is under heavy assessment process – we are reviving its features and conducting interviews with the main users to understand how you are using our services, and how we can evolve in the future.

What’s next?

In the first quarter, the whole team is joining the effort of developing InvenioRDM, which will be the foundation of the future CDS.

Plans for the last months of 2021

What’s happening on CDS until the end of the year?

Currently (summer 2021), the team is working on consolidating the recently released CERN Library Catalogue platform with bug fixes and some new features, such as bulk extensions for loans, improving e-mails and ad-hoc features for the daily operation of the CERN Library.

The CDS Videos platform is very much used at CERN during the past and current extended teleworking period due to the COVID-19 pandemic. It requires improvements and fixes. The team will be busy improving the upload experience of new videos to be much more smooth and fixing bugs.

At the same time, we will start a new analysis phase to identify the best approach to migrate the current CDS website to a new modern, user-friendly web platform based on InvenioRDM in collaboration with the CERN experiments. We will share more information on how the future platform will look like, stay tuned!

Powered by WordPress & Theme by Anders Norén