Red-teaming

Red-teaming means trying to make your feature misbehave the way a motivated user or attacker would. For a product team, it's a focused exercise before launch and after significant changes, not a research program. The goal is a list of concrete failures, each with a severity and a fix.

Start from your risk assessment and turn each risk into an attack category: off-scope use, harmful or brand-damaging output, unauthorized commitments such as promised refunds, system prompt leakage, cross-tenant data leakage, indirect injection through documents or tickets, unsafe links or markup in output, cost attacks, and multi-turn escalation. Write probes for each, and plant canary strings in system prompts and other tenants' test data so leaks can be detected in code rather than by eye.

Combine automated probes, which you can rerun in CI as regression tests, with a session where people try things your scripts didn't think of. Testing the bare model in a playground misses the point: test the deployed feature with its real retrieval, tools, and output rendering, because that's where defenses interact. Triage honestly. Some findings block launch, some need monitoring, and some are accepted risks you write down.