Issue with Prompt Injection in Vanna AI Exposes Databases to Remote Code Execution Attacks
A vulnerability that has been uncovered in the Vanna.AI library by cybersecurity researchers could potentially lead to remote code execution due to a flaw that allows prompt injection techniques to be exploited.
Referred to as CVE-2024-5565 with a CVSS score of 8.1, this security issue is linked to prompt injection within the “ask” function, enabling the execution of unauthorized commands within the library, as highlighted by supply chain security firm JFrog.
Vanna represents a Python-based tool for machine learning that permits users to interact with their SQL database through conversation-based querying, a process known as prompting that transforms queries into SQL language via a large language model (LLM).
The proliferation of generative artificial intelligence (AI) models has accentuated the threat landscape, allowing threat actors to exploit these tools through adversarial inputs that circumvent inherent safety measures.
Among the notable attacks is prompt injection, a type of AI jailbreak used to subvert safeguards put in place by LLM providers to prevent the production of inappropriate or illegal content or perform actions against the intended application purpose.
Such attacks can take different forms, including indirect methods that manipulate data processed by third-party systems to execute harmful payloads fostering AI jailbreaks.
Additionally, scenarios like multi-turn jailbreaks or Crescendo can be employed, gradually steering dialogues towards prohibited goals in a sequential manner to achieve malicious intent.
This methodology can be extended to introduce a novel jailbreak known as Skeleton Key.
“The Skeleton Key approach involves utilizing a multi-turn strategy to coerce a model into bypassing its in-built guardrails,” remarked Mark Russinovich, chief technology officer of Microsoft Azure, stating. “Once these guardrails are circumvented, the model becomes blind to distinguishing between legitimate and malicious requests.”
Skeleton Key differentiates itself from Crescendo in that following a successful jailbreak and alteration of system rules, the model can produce responses that defy ethical boundaries and safety concerns.
“Once the Skeleton Key exploit is successful, the model acknowledges the modification of its guidelines and proceeds to generate content without any regard to violating its original responsible AI guidelines,” Russinovich added.
“Unlike Crescendo and similar strategies that necessitate indirect or encoded task requests, Skeleton Key grants direct control for users to order tasks, resulting in unfiltered model outputs that showcase the scope of knowledge or content production capabilities,” explained in a statement.
Recent disclosures from JFrog, also brought to light independently by Tong Liu, illustrate the gravity of prompt injections, particularly when intertwined with command execution.
The exploit leverages Vanna’s feature for generating text-to-SQL queries, subsequently executing these queries and visually representing results using the Plotly graphing tool.
This loophole is mediated through a function called “ask”, such as vn.ask(“What are the top 10 customers by sales?”), a principal API endpoint enabling the formulation and execution of SQL queries against the database.
When combined with the dynamic Plotly code generation, this vulnerability allows a malicious actor to introduce a specially crafted prompt containing commands to be executed on the system.
JFrog highlighted that due to the usage of the prompt function in Vanna to furnish visual results to users, it is plausible to manipulate prompts using prompt injection, thereby executing arbitrary Python code instead of the intended visualization code.
“By permitting external input to the ‘ask’ method of the library with ‘visualize’ set to True (default behavior), the potential for remote code execution arises,” noted JFrog.
Following responsible disclosure, Vanna has released a security hardening guide advising users on the risks associated with the Plotly integration and urging those exposing this function to do so within a confined environment.
“This discovery underscores the perils of deploying GenAI/LLMs without adequate governance and security measures, posing significant repercussions for organizations,” expressed Shachar Menashe, the senior director of security research at JFrog.
“The awareness surrounding prompt injection risks remains limited, yet its adept exploitation is a concern. Companies should not solely rely on pre-prompting as a foolproof defense mechanism but rather implement robust safeguards when LLMs interact with critical assets like databases or during dynamic code generation.”
About Author
