Validation, Retries, and Repair
Schema-valid isn't the same as correct. Add business-rule validation, targeted retries with feedback, and graceful fallbacks when output still fails.
Structured outputs end parse errors. They don't end wrong answers. A refund extraction can match your schema perfectly and still quote a sentence the customer never wrote, name an order that belongs to someone else, or request more money than the order cost. Your schema can't catch any of that, because the schema only knows about shapes. Your application knows about facts.
This lesson builds the layer between "the model returned valid JSON" and "my code acts on it": validators that check business rules, cheap deterministic fixes, retries that tell the model exactly what was wrong, an escalation path to a stronger model, and a graceful fallback when nothing works. You'll finish with a reusable pattern and the metrics that tell you whether it's earning its keep.
Four layers of checking
Think of validation as a stack. Each layer catches different failures at a different cost:
| Layer | Catches | Cost | Where it runs |
|---|---|---|---|
| Parse | Truncated or broken JSON | Free | Your code |
| Schema | Wrong types, missing keys, out-of-list values the grammar didn't enforce | Free | Zod or Pydantic |
| Business rules | Ungrounded quotes, cross-field contradictions, IDs that don't exist, impossible amounts | A few lines, maybe a database read | Your code |
| Judgment | Subtle wrongness: right shape, real quotes, wrong conclusion | Another model call, or a human | A grader or review queue |
Most teams stop after the second layer because structured outputs make the first two feel solved. The third layer is where the valuable checks live: it's cheap, deterministic, and catches the errors that actually hurt users. The fourth layer is for things code can't decide; you'll build model graders properly in Model-Graded Evals You Can Trust. This lesson focuses on the third layer and what happens when it fails.
We'll use a Deskmate refund extraction throughout. Here's the schema; note the evidence field first, a pattern from the previous lesson:
import { z } from "zod";
export const RefundRequest = z.object({
evidence: z.array(z.string()).describe("Exact sentences from the ticket that state the refund request."),
orderNumber: z.string().nullable().describe('Like "DM-10442". null if not given.'),
amountCents: z.number().int().nullable().describe("Amount requested, in cents. null if not stated."),
reason: z.enum(["double_charge", "not_as_described", "cancelled_too_late", "service_outage", "other"]),
reasonDetail: z.string().nullable().describe('Required when reason is "other"; otherwise null.'),
});
export type RefundRequest = z.infer<typeof RefundRequest>;