Novel Attack Technique ‘Sleepy Pickle’ Aiming at Machine Learning Models
The potential security threats associated with the Pickle file format have once again been highlighted following the identification of a fresh “hybrid technique for exploiting machine learning (ML) models” referred to as Sleepy Pickle.
In line with Trail of Bits, this method of attack leverages the common format utilized for bundling and disseminating ML models to corrupt the model itself, creating a significant supply chain vulnerability for downstream consumers of an organization.
According to security researcher Boyan Milanov, “Sleepy Pickle is a sophisticated and innovative approach to attacking the ML model directly rather than the underlying system,” as noted in a statement.
Although pickle serves as a widely adopted serialization format within ML libraries like PyTorch, it can potentially facilitate the initiation of executable code execution attacks by simply loading a pickle file (i.e., during deserialization).
“We recommend retrieving models exclusively from trusted users and organizations, emphasizing signed commits, and/or opting to load models from [TensorFlow] or Jax formats with the from_tf=True auto-conversion mechanism,” as highlighted by Hugging Face in its documentation.
Sleepy Pickle operates by embedding a payload into a pickle file via open-source solutions like Fickling, and subsequently delivering it to a target host using one of four methods, namely an adversary-in-the-middle (AitM) attack, phishing, supply chain compromise, or exploiting a system vulnerability.
“Upon deserialization on the victim’s system, the payload gets executed, modifying the model contained within to introduce backdoors, manipulate outputs, or tamper with processed data before returning it to the user,” mentioned Milanov.
In other words, the payload injected into the pickle file housing the serialized ML model can be misused to adjust the model’s behavior by interfering with the model’s weights or altering the input and output data processed by the model.
In a hypothetical attack scenario, this method could result in generating harmful outputs or misinformation leading to hazardous consequences for user safety (e.g., recommending dangerous practices to cure illnesses), stealing user data upon fulfilling specific conditions, and indirectly targeting users by producing manipulated summaries of news stories with embedded links to phishing sites.
Trail of Bits emphasized that Sleepy Pickle can be exploited by threat actors to maintain concealed access on ML systems in a manner that avoids detection, given that the model becomes compromised when the pickle file is loaded within the Python process.
Furthermore, this approach proves more potent than directly uploading a malicious model to Hugging Face, as it enables the dynamic modification of model behavior or outputs without necessitating the persuasion of targets to download and run them.
“Through Sleepy Pickle, attackers can create pickle files that are not necessarily ML models but can still disrupt local models upon loading,” explained Milanov. “Consequently, the attack surface broadens significantly, as control over any pickle file within the supply chain of the target organization is adequate to jeopardize their models.”
“Sleepy Pickle illustrates how advanced attacks at the model level can exploit weaknesses in lower-level supply chains by exploiting the relationships between groundwork software components and the final application.”
About Author
