Jailbreak

A jailbreak targets the model's own guardrails. Where prompt injection smuggles instructions in through data, a jailbreak is usually the user directly trying to get the model to produce something it, or your product, shouldn't: harmful instructions, abusive content, or promises your business never made. Techniques include role-play framing, hypothetical scenarios, encoding tricks, and building up over many turns until a refused request finally succeeds.

For product builders, jailbreaks cause two kinds of damage. The first is reputational: a screenshot of your branded assistant saying something offensive or promising a refund you don't offer. The second is your provider account: when someone uses your feature to generate content that breaks the provider's usage policy, the request comes from your API key, and repeated violations can lead to restrictions on your whole organization.

You can't prompt your way to full immunity, so layer defenses. Keep the feature narrowly scoped so off-topic requests are refused early, moderate inputs and outputs, send a stable anonymous user identifier where your provider supports one, count refusals and moderation flags per account, and escalate from throttling to review to suspension for repeat offenders. Test multi-turn attacks specifically, since many defenses only look at one message at a time.