Bombshell in SSH servers! What CVE-2024-3094 means for Kubernetes users

Read our update: Yet another reason why the xz backdoor is a sneaky b@$tard

On March 29, 2024, Red Hat disclosed CVE-2024-3094 (a.k.a XZ vulnerability) scoring a critical CVSS rating of 10. Stemming from a supply chain compromise it affects the latest iterations of XZ tools and libraries. The CVE was identified by a software engineer following the discovery of performance issues in SSH connections. This led to the exposure of a major supply chain attack where a compromised library was inserted into sshd and exploited during the authentication process. 

In this article, we will explore the technical implications of this attack and explain its effects on Kubernetes infrastructure and users.

Technical Overview

Andres Freund, who discovered this backdoor, wrote a detailed report with his findings.

He found that the attacker pushed obfuscated code that seems like tests into the build process of liblzma, a popular data compression library. During the XZ build process, a critical script, Build-to-Host.m4, is invoked. Within this script, there lies a crucial line:

gl_[$1]config='sed "r\n" $gl_am_configmake | eval $gl_path_map | $gl[$1]_prefix -d 2>/dev/null'

which strategically embeds an obfuscated script to execute alongside the configured script’s conclusion. This injected script plays a major role in generating the MakeFiles for xz-utils and liblzma.

While researchers around the world are still analyzing and reverse engineering the obfuscated code, it seems like the malicious code eventually hooks the PLT (Procedure Linkage Table), specifically the function “RSA_public_[email protected]” to point to its own code.

This is dramatic and hard to detect because not only was the user who pushed the malicious code an open-source contributor for almost 2 years, the malicious code is not even in the “source code tar”. It can only be found in the distributed tar.

To paraphrase Andres Freund’s report:

Notably liblzma’s symbols are resolved before many of the other libraries, including the symbols in the main sshd binary. This is important because after symbols are resolved, the GOT (Global Offset Table) gets remapped with read-only page permissions thanks to the compile flags -Wl,-z,relro. In order for the malicious code to be able to resolve symbols in libraries that have not yet loaded, the backdoor installs an audit hook into the dynamic linker.

That hook gets called, from _dl_audit_symbind, for numerous symbols in the main binary. It appears to wait for “RSA_public_[email protected]” to be resolved.  When calling that symbol, the backdoor changes the value of RSA_public_[email protected] to point to its own code. It is possible to change the got.plt contents at this stage because it has not been (and can’t yet be) remapped to be read-only.”

The attacker wanted to make it even harder for researchers to detect as the obfuscated code that installs the backdoor runs only under certain conditions:

• The targeted operating system must be x86-64 Linux. Failure to meet this condition prevents the installation of the backdoor.

if ! (echo "$build" | grep -Eq "^x86_64" > /dev/null 2>&1) && (echo "$build" | grep -Eq "linux-gnu$" > /dev/null 2>&1);

• The XZ build process must be part of a Debian or an RPM package build, complicating replication as the backdoor avoids installation during manual XZ package builds.

if test -f "$srcdir/debian/rules" || test "x$RPM_ARCH" = "xx86_64";

Furthermore, various runtime requirements for the exploit have been identified:

• The TERM environment variable must remain unset, indicating that the SSH client-server communication hasn’t started, which aligns with the exploit’s targeted stage.
• The process executing the malicious code must have a binary path of /usr/sbin/sshd, ensuring activation solely when sshd utilizes the libzlma library.
• LD_DEBUG and LD_PROFILE environment variables must remain unset to prevent exposure to symbol resolution interference and other linker/loader manipulations.
• The LANG environment variable must be set, consistent with sshd’s behavior of setting LANG.
• The exploit detects debugging tools such as rr and gdb; if detected, it abstains from execution, employing a classic anti-debugging technique.

Impact on sshd

During authentication to the ssh service the user exchanges RSA keys with the server, the malicious code gets invoked during the verification process on the server side code: 

sshd 1736357 [010] 714318.734008: 1 branches:uH: 5555555ded8c ssh_rsa_verify+0x49c (/usr/sbin/sshd) => 5555555612d0 RSA_public_decrypt@...+0x0 (/usr/sbin/sshd)

The backdoor then calls back into libcrypto, presumably to perform normal authentication

sshd 1736357 [010] 714318.734009: 1 branches:uH: 7ffff7c137cd [unknown] (/usr/lib/x86_64-linux-gnu/liblzma.so.5.6.0) => 7ffff792a2b0 RSA_get0_key+0x0 (/usr/lib/x86_64-linux-gnu/libcrypto.so.3)

We can see this is the same function that was hooked in the build process of liblzma.

It looks like the malicious code is some sort of backdoor that allows the attacker to bypass authentication, but as the reverse engineering continues it seems that it is more of an RCE (remote code execution) and it calls the libc system() function as well.

Impact on Kubernetes 

For Kubernetes users, this critical vulnerability may put your entire cluster at risk. 

The potential risk here is mainly nodes or public-facing workloads that are running an SSH server. If they use a vulnerable version of “liblzma” at either level, the nodes can be exploited by anyone who has network access to the SSH port.

How to stay safe?

1. Check if your nodes are vulnerable and patch them before patching the rest of the cluster.
2. Scan for vulnerabilities, using a Kubernetes security solution, such as Kubescape or ARMO Platform, to see which images are vulnerable and patch them.
   • Prioritize patching external facing workloads loaded into memory first.
3. Make sure you are following Kubernetes security best practices like:
   • Implement the Principle of Least Privilege, with correct RBAC permissions – don’t give workloads more access then they need.
   • Don’t run pods as root user and or with the privileged setting.
   • Apply network policies to limit your workloads’ communications to those necessary for their function.
   • Apply seccomp profiles to limit the syscalls a workload can make in order to decrease the attack surface.
   • Use a KDR (Kubernetes Detection and Response) tool which can protect from runtime attacks.

Who is affected by this

All major Linux distributions advise either reverting to versions released before the introduction of XZ Utils 5.6.0 and 5.6.1, or upgrading to Red Hat Linux versions 6 and up, as they are reported by Red Hat as not affected.

Distro	Affected Version
Red Hat	Fedora Linux 40 and Fedora Rawhide
Debian	Debian stable versions remain unaffected. The compromised packages were found in the Debian testing, unstable, and experimental distributions, spanning from version 5.5.1alpha-0.1 (uploaded on 2024-02-01) up to and including version 5.6.1-1.
Kali	The vulnerability’s impact was felt on Kali systems between March 26 and March 29. If you updated your Kali installation within this time frame it’s vital to apply the latest updates immediately to mitigate this issue.
OpenSUSE	The vulnerability’s impact was felt onOpenSUSE Tumbleweed and OpenSUSE Micro OS between March 7th and March 28th 2024. If you updated your OpenSUSE installation within this time frame it’s vital to apply the latest updates immediately to mitigate this issue.
Alpine	5.6 versions prior to 5.6.1-r2
Arch	Installation medium 2024.03.01Virtual machine images 20240301.218094 and 20240315.221711 Container images created between and including 2024-02-24 and 2024-03-28