Input and Output Moderation
Screen what goes in and what comes out: moderation APIs, classifier prompts, policy definitions, and handling false positives without frustrating users.
Deskmate launches a customer-facing help chatbot. Within the first week, someone spends an hour trying to make it write insults about your competitors, an angry customer pastes a threat aimed at a named support agent, and a user in distress mentions they don't want to be alive anymore. Each of those needs a different response, and "the model will probably handle it" is not a plan for any of them.
Moderation is checking content against a policy you define: what users send in (input moderation) and what the model sends back (output moderation). In this lesson you'll write a moderation policy before choosing tools, compare dedicated moderation APIs with an LLM classifier that knows your rules, wire checks into a chat feature without adding latency, and handle false positives so legitimate users don't feel accused.
What moderation does, and what it doesn't
Three layers decide what your feature will and won't do, and it helps to keep them separate:
- The model's own safety training. Frontier models decline clearly harmful requests on their own. With Claude, a declined response arrives with
stop_reason: "refusal", which your code should check. This is a baseline, not your policy: it doesn't know your product, your users, or your rules. - Your policy checks. Classifiers that screen inputs and outputs against rules you wrote for this product. This is what this lesson is about.
- Human review. People who look at flagged content, handle appeals, and update the policy.
Moderation also differs from injection defense. Moderation asks "is this content acceptable here?" Prompt injection asks "is this content trying to control the model?" A perfectly polite ticket can carry an injection, and an abusive message can contain no instructions at all. You need both, and a moderation classifier doesn't substitute for the other.
Finally, your policy is usually narrower than the model's built-in limits. A help-desk bot shouldn't give legal advice or discuss a customer's medical condition, even though a general-purpose model might happily do both. Those scope rules belong in the same policy as the safety rules, because they're enforced the same way.