Exploits & Vulnerabilities
An already revealed weakness in the NVIDIA Container Toolkit has a deficient patch, which, once abused, might endanger a broad range of AI infrastructure and confidential information.
Read time: ( words)
Summary:
- Trend Research identified that NVIDIA’s September 2024 security update for a critical vulnerability (CVE-2024-0132) in the NVIDIA Container Toolkit was incomplete, leaving systems potentially vulnerable to container escape attacks. Additionally, researchers discovered a denial-of-service (DoS) vulnerability affecting Docker on Linux.
- Exploiting these vulnerabilities could enable attackers to access sensitive host data or cause significant operational disruption by exhausting host resources. Successful exploitation could lead to unauthorized access to sensitive host data, theft of proprietary AI models or intellectual property, severe operational disruptions, and prolonged downtime due to resource exhaustion or system inaccessibility.
- Organizations utilizing the NVIDIA Container Toolkit or Docker in AI, cloud, or containerized environments are directly affected, particularly those using default configurations or specific toolkit features introduced in recent versions. Companies deploying AI workloads or Docker-based container infrastructure are potentially at risk.
- Trend Vision One™ provides visibility and detection capabilities for potential attacks that can take advantage of the vulnerability. For additional best practices and detailed recommendations, see the mitigation guidance provided below.
In September 2024, NVIDIA released several updates to address a critical vulnerability (CVE-2024-0132) in its NVIDIA Container Toolkit. If exploited,The security flaw might expose the AI setup, data, or confidential details. Having a rating of 9.0 in CVSS v3.1, all clients were urged to promptly update the impacted software.
After conducting further investigation, it was discovered that the fix was inadequate. While analyzing the update in October 2024, we came across a performance glitch impacting Docker on Linux. These problems could allow hackers to evade container isolation, access crucial host resources, and trigger major operational disruptions.
Evaluation of CVE-2024-0132 disclosed an issue that could result in a denial of service
A time-of-check time-of-use (TOCTOU) flaw persists within the NVIDIA Container Toolkit, enabling a specially designed container to reach the host file system. The default configurations remain weak for versions 1.17.3 and earlier, while version 1.17.4 mandates the activation of the feature allow-cuda-compat-libs-from-container.
This weakness was identified during the examination of updates for CVE-2024-0132 and has been disclosed under ZDI-25-087.
| Product | Affected Versions |
| nvidia_container_toolkit | · 1.17.3 and earlier |
| · 1.17.4 require a specific feature to be activated |
Table 1. While older versions of the NVIDIA Container Toolkit are susceptible, version 1.17.4 necessitates the activation of a feature for it to be vulnerable.
There is also a potential performance drawback that could result in a denial-of-service (DoS) susceptibility on the host system. This problem impacts Docker on Linux systems. As per the Docker security team:
The Docker API serves as a privileged interface. Consequently, any user with API accessibility essentially possesses root-level privileges on the host system. It is currently uncertain if this problem stems from Docker’s runtime or the Linux kernel’s management of mount entries.
Exploitation process for the DoS-binding concern
The same performance dilemma has also been independently reported by moby and NVIDIA:
- Upon creating a new container with multiple mounts configured using (bind-propagation=shared), numerous parent/child paths are established. Nonetheless, the associated entries are not cleared from the Linux mount table upon container termination.
- This leads to swift and uncontrollable expansion of the mount table, depleting available file descriptors (fd). Ultimately, Docker is incapable of creating new containers due to fd exhaustion.
- This excessively large mount table induces a significant performance hindrance, preventing users from connecting to the host (e.g., through SSH).
A case study illustrating the potential exploitation of ZDI-25-087
The subsequent steps outline the potential progression of an attack:
- An attacker devises two malevolent container images interconnected via volume symlink.
- The attacker executes the images on the target’s platform, either directly or indirectly (e.g., via supply chain and social engineering methods).
- This gives the attacker access to the host file system through a race condition.
- With this access, the attacker can then reach the Container Runtime Unix sockets to execute arbitrary commands with administrative privileges, effectively gaining complete remote control of the compromised system.
Security measures for mitigating the vulnerability
To effectively mitigate vulnerabilities associated with the NVIDIA Container Toolkit (CVE-2024-0132 and the related Docker file system binding issue), we recommend the following security best practices:
- Constrain Docker API access and privileges. Control API access to authorized personnel exclusively. Avoid providing unnecessary root-level permissions or privilege elevation to reduce potential exposure risks.
- Deactivate non-essential functionalities. To diminish the attack surface, explicitly turn off optional features introduced in NVIDIA Container Toolkit 1.17.4 unless essential for operational purposes.
- Enforce container image admission controls. Enact stringent admission control policies in CI/CD pipelines. Automatically scan and block container images flagged as vulnerable before deploying them in production environments.
- Monitor the Linux mount table. Regularly check the Linux mount table for abnormal expansion as a sudden rise in entries can indicate active exploitation attempts or preparation for DOS attacks.
- Perform routine audits of container-to-host interactions. Conduct scheduled audits of container-to-host filesystem bindings, volume mounts, and socket connections. Restrict these interactions to essential scenarios only, employing strong isolation techniques to minimize risks.
- Implement runtime anomaly detection. Enforce runtime monitoring tools capable of detecting anomalous activities suggestive of exploitation, such as unauthorized host filesystem binding or unusual container behaviors.
- Validate security patch applications. Immediately confirm the adequacy of all applied security patches. Given past incomplete fixes, comprehensive post-patch verification is crucial to ensure effective vulnerability mitigation.
Proactive security with Trend Vision One™
Trend Vision One™ is the sole AI-powered enterprise cybersecurity platform that consolidates cyber risk exposure management, security operations, and robust layered protection. This holistic approach assists in anticipating and forestalling threats, expediting proactive security results throughout your entire digital landscape. Backed by extensive cybersecurity expertise and Trend Cybertron, the premiere proactive cybersecurity AI in the industry, it delivers proven outcomes: a 92% decrease in ransomware threats and a 99% reduction in detection time. Security leaders can evaluate their stance and exhibit continuous enhancement to stakeholders. With Trend Vision One, you’re empowered to eradicate security blind spots, focus on critical areas, and raise security to a strategic partner for innovation.
Trend Vision One renders protection and detection capabilities through the following:
- Observed Attack Techniques (OAT): XSAE.F8306
- Docker Root Filesystem Binding and XSAE.F11714
- Docker Root Filesystem Binding via docker.sock Workload Behaviors (WB):
- Suspicious Container Creation via Root Filesystem Binding
- Docker Root Filesystem Binding
- Suspicious Container Creation With Root Filesystem Binding via Socket
Trend Micro has also integrated a Time-Critical Vulnerability alert in the Trend Vision One Executive Dashboard, which will be continually updated with additional information concerning prevention and detection as it becomes available.
Swift patching remains the most efficient mitigation strategy, but it may not always be feasible, particularly in intricate or critical production environments. Trend Vision One™ Cloud Workload Security furnishes crucial visibility and detection capabilities, including identifying host file system binding to containers and detecting malicious containers attempting to access the host file system.
Moreover, Trend Vision One™ Container Security proactively pinpoints vulnerabilities, malware, and compliance breaches within container images. Detection capabilities for CVE-2024-0132 and the newly discovered vulnerability from the failed patch are already accessible and seamlessly integrate into Trend Vision One™ Cyber Risk Exposure Management.
Since the attacker can maliciously create an image with the exploit, Trend’s solutions aid in detecting this vulnerability in the pipeline before pushing the image to production. In case the vulnerability is identified, Container Security (admission control policy enforcement) can prevent the container image from being deployed in the production environment. Furthermore, we detect this vulnerability during runtime, ensuring customers have full visibility of this security concern across the entire ecosystem.
Tags
sXpIBdPeKzI9PC2p0SWMpUSM2NSxWzPyXTMLlbXmYa0R20xk
About Author
