Skip to content
47% off: the complete course for $79 (reg. $149)Order now →
AI Builder
Join
CurriculumPricingGlossaryFAQSign inAccount
Theme
Module 9Full course12 min read

Eval-Driven Development and CI

Make evals part of every change: a fast local loop while iterating, a regression suite in CI, and release gates for prompt and model updates.

A support lead reports that Deskmate's drafted replies keep promising refunds. You add one sentence to the prompt, try it on the ticket from the bug report, and the draft looks right. You ship. Two days later someone notices that drafts for German-language tickets now come back in English, because your new sentence was the only English instruction near the end of the prompt. Nothing was wrong with your fix. What was wrong was checking it against one ticket.

You already have the pieces: a quality spec, a golden dataset, code checks, and calibrated graders. This lesson wires them into how you work. You'll build a fast local eval loop for iterating on prompts, a regression gate that runs in CI on every prompt or model change, and a release checklist for the riskiest change of all: swapping the model underneath a feature.

The eval-driven loop

Eval-driven development borrows its shape from test-driven development: you write down what "fixed" means as an eval case before you change anything, then change things until the evals say you're done. It turns "I think this prompt is better" into a claim you can check.

✕ Before
  1. Read the bug report.
  2. Edit the prompt.
  3. Try it on the reported ticket in a playground. Looks good.
  4. Open a pull request titled "fix refund promises."
✓ After
  1. Read the bug report. Add the ticket (and two or three similar ones from your logs) to the golden set, tagged refunds.
  2. Run the eval on the refunds tag and watch the new cases fail. Now you know the eval can see the bug.
  3. Edit the prompt. Rerun the tag until the new cases pass.
  4. Run the full suite. Read any case that newly fails.
  5. Open a pull request with the before/after numbers. CI runs the gate again.

Step 2 is the one people skip, and it's the one that keeps the eval honest. If the new case passes before you've changed anything, either your check can't detect the failure or you reproduced the wrong thing. Fix the check first.

The loop only works if running evals is fast and cheap enough to do twenty times an hour. That's an engineering problem, and the next section solves it.