On the final Friday of March, a pivotal moment unfolded in the computing realm. The sound heard globally was software developers and cybersecurity experts in disbelief, their jaws hitting the floor. A Microsoft employee, driven by a mix of perseverance, expertise, and a considerable dose of luck, uncovered a backdoor—a sinister bypass embedded in the code that enabled attackers to connect to servers and execute remote code (RCE) anonymously. This vulnerability posed a potential catastrophic threat to numerous computer servers worldwide. An extraordinary, years-long proactive effort, the full extent of which is still being uncovered, was averted due to meticulous attention to detail, a sequence of fortunate events, and constant vigilance.
Should the hackers succeeded in fully executing their plan, they would have been able to freely access a substantial number of servers operating under the Linux system within just a few months. A code package known as xz Utils, routinely used across Linux servers for data compression, seemed benign and hardly the type to raise alarms. However, at least two updated versions of this package, previously certified as safe, harbored hidden malicious code. In the right circumstances, this code could enable an attacker who possesses the correct key to infiltrate a server compromised by this rogue version of the package and execute any command at will, bypassing the need for traditional authentication methods like usernames and passwords.
These malevolent versions had penetrated several Linux distributions, reaching advanced testing stages in two prominent distributions, Debian and Red Hat. Linux distributions, which are operating systems based on the Linux kernel, incorporate a variety of code libraries. Each distribution is distinct, varying in the composition of its libraries and the tools it provides, tailored to suit its intended users, and they periodically receive updates. Each distribution is exclusively managed and distributed by specific organizations or companies. Within both distributions, the malicious code had successfully infiltrated test versions and stood a high chance of passing through the testing phases unscathed, potentially becoming part of the stable release scheduled for the upcoming months.
Half a second
While working, software developer Anders Freund noticed unusual system behavior. He was utilizing a test version of the Debian Linux distribution and realized that a routine operation was taking longer than expected. The task involved making a remote connection via a commonly used protocol known as SSH. Delving into a particular aspect of the connection process, Freund discovered that in this new test version, the connection time had increased to 0.807 seconds from the previous 0.299 seconds, a significant delay of about half a second.
This seemingly minor delay, easily dismissed by many, led Freund to dig deeper. He observed an abnormal increase in CPU resource consumption during the failed SSH connection attempt, caused by an incorrect username. These resources were being consumed by a library named liblzma, part of the xz Utils package. Further investigation revealed that the link between the file compression library and the error in server connection was due to malicious code embedded within the library.
A complicated endeavor
Open source is a method of software development where the code is available to the public for review and is maintained transparently. In these environments, users can review code performance, propose modifications, and directly contribute under the oversight of project administrators. This structure generally supports transparency, accessibility, and collaborative engagement in software development. In the context of xz, the project is led by Lasse Collin, who also has the authority to update and alter the code.
In 2021, an individual using the alias JiaT75, real name Jia Tan, began offering code modifications for various open source initiatives. His initial submission, which was accepted, was not for the xz project and was also flagged as suspicious. Tan began submitting proposals to the xz project, and by early 2022, Collin started approving and incorporating them. Later that year, Collin received numerous complaints about the project's poor upkeep and sluggish procedures. Retrospectively, these complaints may not have come from genuinely concerned users but rather from "sock puppets," fictitious accounts created to exert pressure on Collin to recruit additional maintenance support to manage the growing demands of the project. In his communications, Collin defended his actions and revealed his struggle with a mental health condition that hindered his ability to manage stress. He later acknowledged that Tan was significantly helping with maintenance and suggested that Tan might play a more significant role in the project going forward.
Tan was eventually granted autonomous authority to implement changes in the project as an official maintainer. Over 2022 and 2023, he made several alterations that seemed harmless but were later understood to be part of a broader subversive strategy. To conceal the malicious code, Tan embedded it within test files—these are not the main software files used by end-users but are meant for developers to check software reliability. The benefit of hiding the code within these files is that they are not easily readable, making it less likely that they would be scrutinized.
As an additional layer of cover, Tan separately packaged these files from the open-source code for distribution as a new release of xz Utils. It's commonly presumed that the package content mirrors the open-source code, so any discrepancies might go unnoticed. By this point, Tan had become a trusted figure in maintaining the project, lending him the credibility needed when the files were distributed. In the package he assembled, Tan included a file designed to activate the scheme. Hidden malicious code was scattered throughout test files and various parts of the project, intended to remain inactive under normal circumstances. The crucial element, only present in the release package, was designed to activate the hidden malicious code in real-time as end-users operated the software.
In programming, it's typical to connect a software's executable file to the libraries that support it. Some Linux distributions, especially those that have been breached, contain a combined set of software and libraries known as systemd. This collection is essential for fundamental operations within the operating system. Notably, it includes a connection to the liblzma library, a component of the xz Utils package—the same package Freund noticed was causing slowdowns during SSH remote connections.
Within these vulnerable Linux versions, the SSH service is connected to systemd, which also interfaces with liblzma. Sometimes, just linking a software to a library might permit the library to impact the software’s functionality directly. As a result of this linkage between SSH and liblzma, along with the harmful code embedded in it, any SSH connection that is established allows compromised code from liblzma to grant unrestricted access to an intruder who possesses the correct key.
A bit of housekeeping
Currently, it appears that the security issue has been resolved. Tan's access rights have been revoked, and the altered versions of the software have been reverted to their original forms. Open-source software fundamentally relies on a foundation of trust and collaborative effort. Despite some projects being managed by individuals as a leisure activity, numerous major corporations depend on open-source initiatives. This event underscores the disparity between the contributions of individual volunteers, presumed to have good intentions, and their potential impact on a global user base, which may occasionally escape thorough monitoring. Furthermore, the open-source model was instrumental in allowing an outside developer to identify the issue, enabling a broad spectrum of security experts to promptly analyze the software from various perspectives once the breach came to light.
The open-source community, along with security researchers and major enterprises, is expected to persist in their efforts to track down these unidentified assailants and intensify their examination of other sectors to uncover more hidden vulnerabilities. The specifics of this case are still being analyzed, and both the extent of the breach and the identities of those responsible are still shrouded in mystery.
Yoav Talin and Tal Sokolov are from the Davidson Institute for Scientific Education