XZ backdoor: A Demonstration of Free Software Security

1 The XZ Library: a free, powerful and versatile data compressor.

Formerly known as LZMA Utils, the XZ library is a free and open source software library that provides a set of command-line utilities for losslessly compressing and decompressing data. It is based on the Lempel-Ziv-Markov chain compression algorithm (LZMA), which offers a high compression rate and a good balance between compression speed, decompression speed, lossless compression efficiency and compression rate (result size vs. original size).

1.a How to use the library

It is used in a wide variety of applications, including:

• File compression: to compress individual files or entire directories, reducing their size and saving storage space.

• Software packaging: For packaging Linux software and distributions, as it allows you to create compact and efficient files.

• Compression of data streams: Due to its speed, it is preferred for compressing data streams in real time, making it ideal for network and data transmission applications.

• Integration with other systems: such as databases and file systems, to provide transparent data compression.

All of this is achieved through the use of the LZMA algorithm, a lossless compression algorithm based on the identification and replacement of repetitive patterns in data that leverages a pattern dictionary and a statistical model to optimize compression.

Given the last mentioned feature “Integration with other systems”, there are hundreds of information systems and software, on Linux, UNIX and MacOS, that use the XZ library to solve their compression requirements, both for data storage and transmission. Additionally, the operating system kernel that runs when a device boots is typically XZ-compressed. Google, Oracle, IBM and many other megacorporations use the XZ library in their projects. Projects like 1password, a password manager, use XZ.

1.b Examples of using the XZ library on the command line:

• To compress a file named file.txt, you can use the following command: xz file.txt

• To unzip a file named file.txt, you can use the following command: unxz file.xz

• To pack and compress a directory called directory, you can use the following command: tar Jcvf directory.tar.xz directory

• To unzip and unpack a compressed packed file named directory.tar.xz, you can use the following command: tar Jxvf directory.tar.xz

• To compress a file named file.txt, using lzma algorithm, you can use the following command: lzma file.txt

• To unzip a lzma compressed file named file.txt.lzma, you can use the following command: unlzma file.txt.lzma

• To pack and compress a directory called directory, using lzma algorithm, you can use the following command: tar Ycvf directory.tar.lzma directory

• To unzip and unpack a compressed packed lzma file named directory.tar.lzma, you can use the following command: tar Yxvf directory.tar.lzma

2 Freedom is enough to guarantee quality?

The short answer is that freedom in software is no guarantee of security or quality.

Free software, despite its undeniable benefits, is not without risks. The idea that software freedom automatically implies its security and quality is a common misconception. As with private or proprietary software, there is bad and insecure software.

Free software licenses do not guarantee security or quality. Its primary function is to protect or restrict users' freedoms, such as the freedom to use, modify, and distribute the software. While some licenses, such as the GPL, include clauses that encourage collaboration and transparency, they do not guarantee that the software is secure or of high quality.

2.a So, how is the security of free software verified?

To evaluate the security and quality of free software, there are two main methods:

Source code analysis:

This method involves downloading the software's source code and reviewing it line by line to identify potential vulnerabilities or security issues. Although it is an effective method, it is extremely costly in time and resources, and requires a high level of technical knowledge.

Governments and megacorporations, such as the Russian government with Astra Linux and Novell with SuSE, have invested in this type of analysis to certify the security of their Linux distributions. However, this approach is not feasible for most individual users or small free software projects.

Community Analysis:

This method is based on the idea that free software with an active and engaged community is more likely to be secure and of high quality. The key indicators to consider are:

• Community size and activity: Software with a large, active community indicates greater interest and participation in its development, which increases the likelihood of detecting and fixing bugs or vulnerabilities.

• Discussion forums and communication channels: The existence of active discussion forums and communication channels where users can interact with developers and each other is an indication of an engaged community.

• Frequency of updates and new versions: Software with a constant rate of updates and new versions shows that developers are actively working on its improvement and maintenance.

• Complete and up-to-date documentation: The availability of generous, up-to-date documentation translated into multiple languages ​​makes the software easier to use and increases the likelihood that users will detect and report problems.

Although this method does not offer an absolute guarantee of safety and quality, it is more accessible and practical for most users. It allows you to evaluate the level of community commitment and the probability that the software remains updated and free of serious errors.

It is essential to evaluate each free software project individually, considering both the analysis of the source code and the evaluation of the community. The combination of these two methods allows you to make informed decisions about the security and quality of the free software used.

3 A security paradox in free software

In the world of free software, security has always been a topic of heated debate. Detractors of free software often point to the lack of centralized control as a weak point, while its defenders argue that the transparency and collaboration inherent in this model make it a more secure environment. The recent security breach in the XZ library, used to compress data across virtually all modern digital infrastructure, has reignited this debate with renewed momentum.

To understand how free software security works, you have to imagine a door with its completely transparent sheet metal. Anyone could see the lock mechanism, but without the corresponding key they will not be able to open the door. It is also true that expert locksmiths will be able to identify and warn of the vulnerabilities that the mechanism has and will be able to use their knowledge both to violate it and to correct it. With a big advantage, when any vulnerability is fixed, all doors are automatically improved at no cost to the owner.

The vulnerability in XZ, which allowed an attacker to execute arbitrary code on any system using the library, revealed the fragility of even the most popular and widely used free software projects. However, this gap also illustrates the paradox of security in free software: the openness and transparency that characterize this model, while they can be exploited by malicious actors, are also the key to its robustness and protection.

As legendary hacker and free software philosopher Eric Raymond said: "the more eyes, the faster bugs are detected." The decentralized nature of free software allows a large number of developers and users to examine the source code, significantly increasing the chances of identifying and fixing vulnerabilities before they can cause widespread damage.

4 An exhausted leader

Lasse Collin is a Finnish software engineer living in Nebraska, USA, best known for being the main maintainer of the XZ library. He has actively contributed to the XZ project since 2002 and has played a key role in its development and maintenance.

The exact reasons why Lasse Collin attempted to delegate his role as lead maintainer of XZ are not public. However, there are some clues to suggest that exhaustion, health problems and work overload could have been important factors.

• Long-term maintenance: Lasse Collin has been involved in the XZ project for more than two decades. This type of long-term commitment to a software project can be exhausting, especially considering the complexity and responsibility inherent in maintaining a critical library.

• Lack of support: While XZ is a popular project, it is clear that Lasse Collin has not received the level of support he needed to maintain it sustainably. The lack of resources and collaborators could have contributed to their feeling of exhaustion.

• Pressure and responsibility: Being the main person in charge of a library as important as XZ comes with great pressure and responsibility. This pressure can be overwhelming and negatively impact personal well-being.

Perhaps, this message sent by Lasse Collin is evidence of the pressure he was receiving from Jigar Kumar, which led him to give up the role of maintainer to Jia Tan. Within the message it is evident that for him to be a maintainer of the XZ project It is a hobby for which he receives no remuneration and, at the time of the attack, he had long-standing mental health problems. (The most important emails in this thread are in the bibliography.)

Re: [xz-devel] XZ for Java

Lasse Collin Wed, 08 Jun 2022 03:28:08 -0700

On 2022-06-07 Jigar Kumar wrote:
> Progress will not happen until there is new maintainer. XZ for C has
> sparse commit log too. Dennis you are better off waiting until new
> maintainer happens or fork yourself. Submitting patches here has no
> purpose these days. The current maintainer lost interest or doesn't
> care to maintain anymore. It is sad to see for a repo like this.

I haven't lost interest but my ability to care has been fairly limited
mostly due to longterm mental health issues but also due to some other
things. Recently I've worked off-list a bit with Jia Tan on XZ Utils and
perhaps he will have a bigger role in the future, we'll see.

It's also good to keep in mind that this is an unpaid hobby project.

Anyway, I assure you that I know far too well about the problem that
not much progress has been made. The thought of finding new maintainers
has existed for a long time too as the current situation is obviously
bad and sad for the project.

A new XZ Utils stable branch should get released this year withth
readed decoder etc. and a few alpha/beta releases before that.

Perhaps the moment after the 5.4.0 release would be a convenient moment
to make changes in the list of project maintainer(s).

Forks are obviously another possibility and I cannot control that. If
those happen, I hope that file format changes are done so that no
silly problems occur (like using the same ID for different things in
two projects). 7-Zip supports .xz and keeping its developer Igor Pavlov
informed about format changes (including new filters) is important too.

--Lasse Collin

 

5 A stealthy and bold attack

The breach at XZ was not a conventional attack. It was a carefully planned operation that took place over several years. A cunning criminal infiltrated the bookstore's development team, gaining the trust of its members and eventually replacing the chief engineer, who, tired and looking to retire, was looking for a successor. Once in place, the attacker embedded malicious code into the library, taking advantage of their knowledge of the code and the development process.

5.a Attack vector

This timeline reveals a social engineering attack carefully orchestrated over many years and with much patience on the part of the perpetrator.

2021:

In 2021, a user calling himself Jia Tan creates a GitHub account called JiaT75.

His first commits are not directed at XZ, but are deeply suspicious, as he reported an error unzipping tar files and proposed his solution. A solution that involved potentially introducing vulnerabilities into the bdstar project.

2022:

Jia Tan sends a patch via email. After this, a new person emerges, Jigar Kumar, who pressures the project director, Lasse Collin, to merge the patch and add another director to the XZ project.

Two apparently fake accounts, Jigar Kumar and Dennis Ens, are involved in pressuring Lasse Collin to add the new maintainer.

JiaT75 starts contributing regularly to the XZ project and eventually becomes the second most active contributor. It is unclear when they gained full trust within the community.

2023:

JiaT75 earns full trust by merging his first commitment in January. Jia Tan replaces Lasse Collin as the primary contact for Google's XZ-related oss-fuzz project and the infrastructure for the exploit is compromised, with Jia Tan credited despite it being written by Hans Jansen (another apparently fake account).

A pull request is made to oss-fuzz to disable functionality that could expose malicious changes. That is, code was introduced that would allow, in the future, to hide the introduction of malicious changes. Additionally, JiaT75 raises a seemingly unrelated issue that could divert attention.

2024:

A pull request changes the project URL on Google's oss-fuzz, which could give Jia more control.

The final steps for the backdoor are implemented with seemingly harmless commit messages ("Tests: Add some test files").

In addition, fake accounts are starting to pressure different distributions to include the latest version of xz in Debian, Fedora, Ubuntu and other distributions..

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

...This new version (5.6.1) fixes a valgrind bug with liblzma 
that outputs a false warning that could affect existing testing 
frameworks for packages that test with valgrind requiring a 
specific output. This release only fixes bugs.Regards,

-- Hans Jansen

 

5.b Vulnerability and its impact

The backdoor built into the code in versions 5.6.0 and 5.6.1 allowed an attacker to execute arbitrary code on any system that used the XZ library. Like Linux, MacOS or systems that base their connectivity on OpenSSH like many Google and Oracle projects. This means that the attacker could have taken control of servers, networks and devices around the world. The potential impact of the incident could have been enormous, as XZ is used in a wide range of systems, from smartphones to supercomputers.

5.c Detection and response: a community effort

Fortunately, the gap was detected in time thanks to the attention of an OpenSSH user: Andrés Freund, who when performing a microbenchmark noticed an unusual CPU consumption and a delay of 500 milliseconds in the connection process of the liblzma library (part of the XZ library). This small suspicion triggered an investigation that led to the discovery of the malicious code and the identity of the attacker. This research is evident in the email message. https://www.openwall.com/lists/oss-security/2024/03/29/4:

Date: Fri, 29 Mar 2024 08:51:26 -0700
From: Andres Freund 
To: oss-security@...ts.openwall.com
Subject: backdoor in upstream xz/liblzma leading to ssh server compromise

 

5.d The quick and effective response of the free software community

The free software community reacted quickly to the breach. Although the confirmed version of XZ was in a Debian PRE-RELEASE, it was never released in an official version. Users who had upgraded to versions 5.6.0 and 5.6.1 were notified to revert to version 5.4.0 or 5.2.5, which had not been compromised. The XZ developers worked tirelessly to fix the vulnerability and release a new version of the library. Linux distributions and other software projects quickly updated their systems to include the new version. Within days, the threat had been mitigated.

5.e Lessons learned

The gap in XZ leaves us with several important lessons.

• First of all, it shows that the security of free software is not a myth. The transparency and collaboration inherent in this model allow vulnerabilities to be detected and remediated quickly and effectively. Secondly, it highlights the importance of constant surveillance and monitoring. A small detail, such as an unusual delay in transferring files, can be a sign of a serious problem. Finally, the gap reminds us that security is an ongoing process that requires the participation of the entire community.

• The XZ breach was not an attack on the security of free software, but rather an attack on trust. The attacker took advantage of the community's trust to infiltrate the development team and carry out his plan. However, the community's rapid and effective response demonstrates that free software is a robust and secure model that has mechanisms to detect and correct vulnerabilities. The XZ breach reminds us that security is a shared responsibility and that we must all contribute to maintaining a safe and reliable free software ecosystem.

6 Other incidents in free software

Code transparency in free software, although it has advantages, has also given rise to notable security gaps that have put users and organizations in check. Among the best known examples are:

• Heartbleed: This vulnerability, discovered in 2014, affected the OpenSSL library used in numerous websites and online services. It allowed attackers to steal sensitive information, such as passwords and credit card data, silently. However, even though it was in the code for almost 2 years, a developer spotted it and fixed it before anyone exploited the breach. No incidents of vulnerability exploitation were reported.

• Logjam: This flaw, revealed in 2015, affected the TLS/SSL protocol, used to encrypt communications on the Internet. It allowed attackers to intercept and decrypt traffic using the Man In The Middle mechanism, putting sensitive data at risk when weak 512-bit Diffie-Hellman keys were used.

• KRACK: This breach, discovered in 2017, affected the WPA2 protocol, used to protect Wi-Fi networks. It allowed attackers to intercept and decrypt data traffic, compromising the privacy and security of communications.

• GhostShell: This vulnerability, revealed in 2019, affected the Ghost library, used to manage and automate tasks on Linux servers. It allowed attackers to execute arbitrary code on affected systems, taking full control of them.

It is important to note that, despite these security gaps, free software remains a solid and reliable option for many users and organizations. The transparency of the code, together with the collaborative approach to its development, allows for rapid detection and correction of vulnerabilities, which in turn contributes to continually improving the overall security of the software.

7 Conclusions

The XZ breach is a reminder that security in free software is not a guarantee, but rather an ongoing process that requires the active participation of the entire community. It is essential that free software developers maintain a high level of commitment to security, but it is also crucial that users contribute by reporting bugs and potential vulnerabilities.

It is important that users who use free software are aware that they do not have to kill the goose that lays the golden eggs. Free software provides them with multiple benefits, such as cost savings, greater freedom and access to an innovative and constantly evolving software ecosystem. It is the responsibility of users to give back to the free software community by donating part of the profits or savings generated by the use of this software. These donations are essential to support developers and maintain healthy and sustainable free software projects.

A committed and well-paid team of developers facilitates the processes aimed at guaranteeing the security and quality of free software. When developers have the necessary resources and support, they can spend more time and effort detecting and remediating vulnerabilities, implementing strong security practices, and improving overall software.

The fact that the breach was detected so quickly with a detail as small as the 500 millisecond delay in communications detected and reported by a user demonstrates the success of the free software model in keeping systems secure when the community is active. Constant surveillance and collaboration between developers and users are fundamental pillars for the security of free software.

The breach in XZ has also highlighted the importance of strong and vibrant free software communities. An engaged community is more likely to detect and fix vulnerabilities quickly and effectively, and is also more likely to develop strong security practices and foster a security culture within the project.

In conclusion, the XZ breach is not a sign of the failure of free software, but rather a reminder that security is a constant challenge that requires vigilance and continuous effort. The transparency and collaboration inherent in free software, along with active community participation, user responsibility, and the importance of an active community, are powerful tools for building a more secure and reliable software ecosystem.

Bibliography

• Official XZ website: https://en.wikipedia.org/wiki/XZ_Utils

• XZ Documentation: https://www.sony-asia.com/electronics/support/mobile-phones-tablets-mobile-phones/xperia-xz/manuals

• XZ tutorials: https://www.baeldung.com/linux/xz-compression

• https://boehs.org/node/everything-i-know-about-the-xz-backdoor

• Added error text to warning when untaring with bsdtar #1609 https://github.com/libarchive/libarchive/pull/1609/files

• https://www.mail-archive.com/xz-devel@tukaani.org/msg00566.html

• https://www.mail-archive.com/xz-devel@tukaani.org/msg00567.html

• https://www.mail-archive.com/xz-devel@tukaani.org/msg00568.html

• https://www.mail-archive.com/xz-devel@tukaani.org/msg00569.html

• Change of website and maintainer anouncement: https://github.com/google/oss-fuzz/pull/11587

• JiaT75’s Github account: https://github.com/JiaT75

• https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067708

• https://mastodon.social/@AndresFreundTec/112180406142695845

• https://www.openwall.com/lists/oss-security/2024/03/29/4

• https://tukaani.org/xz-backdoor/

• Infography: https://infosec.exchange/@fr0gger/112189232773640259

• Backdoor’s technical explanation: https://gist.github.com/smx-smx/a6112d54777845d389bd7126d6e9f504

Licence

XZ Gap: A Demonstration of Free Software Security is under a licencia de Creative Commons Reconocimiento-CompartirIgual 4.0 Internacional.