Code-Based Checks and Assertions
The cheap, deterministic evals to write first — format, schema, length, required facts, forbidden content — and a tiny eval runner in TypeScript.
The first evals you write should be the boring ones. Does the output parse? Is it under the length limit? Does it mention the fact it was supposed to mention? Does it contain an order number that isn't in the ticket? None of this needs a model to judge. A few lines of code answer each question instantly, for free, with the same verdict every time.
These code-based checks (also called assertions) catch a surprising share of real failures, including several of the most expensive ones. In this lesson you'll write a set of them for Deskmate's drafted-reply feature, learn where string matching goes wrong and how to avoid it, and build a small eval runner in TypeScript that runs your feature over a test set, applies every check, and tells you exactly what broke.
Why deterministic checks come first
Model graders get the attention, but code checks deserve to be written first, for practical reasons:
- They're free and fast. You can run hundreds of cases through them in the time one grader call takes, which means you'll run them on every change instead of before releases only.
- They're perfectly consistent. The same output always gets the same verdict, so when a pass rate moves, the feature changed, not the grader.
- They catch severe failures well. An invented order number, a leaked email address, a malformed response that crashes the UI: these have crisp signatures, and they're often exactly the failures you can't afford.
- They double as production guards. The same function that fails a test case can stop a bad draft from reaching a user.
The limit is just as clear: code can check form and specific content, not meaning. "Does the reply answer the question the customer actually asked?" needs a model grader. A good eval suite is mostly code checks with a few targeted graders on top.