Prompt injection

Prompt injection exploits the fact that a model can't reliably tell instructions from data. Anything you place in its context, such as a customer message, a retrieved help article, an uploaded PDF, or a fetched web page, can contain text telling it to ignore previous instructions and do something else, and the model may comply. Direct injection comes from the user typing it; indirect injection hides in content the feature pulls in.

The impact depends on what the feature can do. For a summarizer, the worst case may be a misleading summary. For features that call tools, write data, or render links and images, injection can leak private data, trigger actions, or exfiltrate information through a crafted URL. The dangerous combination, which Simon Willison named the lethal trifecta, is private data the model can reach, untrusted content it reads, and any channel through which data can leave.

No prompt reliably prevents injection, so defend with architecture. Give the model only the tools and data the current user is allowed to access, require confirmation for consequential actions, mark untrusted content clearly with delimiters, allowlist links and images in output, and avoid completing the trifecta in any single call. Then red-team the feature with planted instructions to check the defenses hold.