Content moderation

Moderation in an AI feature runs on both sides of the model. Input moderation screens what users send, such as harassment, sexual content involving minors, or attempts to misuse the feature. Output moderation screens what the model produces before anyone sees it. Image and audio features need the same checks on media.

Start with a written policy: what's allowed, what's blocked, and what gets flagged for review, specific to your product and audience. Then pick tools. Dedicated moderation APIs are fast and cheap for broad harm categories. An LLM classifier given your own policy handles product-specific rules, such as no legal advice or no discussion of competitors, that generic APIs know nothing about. Many products use both.

Moderation adds latency, so run checks in parallel with generation where you can, or scan output as it streams, and decide what happens if a flag arrives late. False positives are a real cost: blocking a legitimate support request about a medical device or a security incident is a product failure. Measure precision and recall on labeled examples, show users a clear message and a way forward, and review what gets flagged. Log decisions per account, because repeated flags are an abuse signal.