Evals

An eval runs your feature over a fixed set of inputs and scores what comes out. Scoring can be code (does the JSON validate, is the label right, does the reply include a link), a model grader for criteria that depend on meaning, or human review. The result is a number you can track over time, plus the failing cases you can read.

Evals replace the habit of trying a few examples and eyeballing them, which breaks down as soon as a feature has real users. With them, you can change a prompt, switch models, or add retrieval and know whether things got better or worse. Different feature types need different scoring: accuracy, precision, and recall for classifiers, field-level checks for extraction, rubrics and pairwise comparison for generated text.

The common mistake is starting from generic metrics or a public benchmark. Useful evals come from your product: read real outputs, find how they fail, and write checks for those failures. A hundred well-chosen cases you actually run beat a huge dataset nobody looks at. Run them on every prompt or model change, and refresh them from production samples so the test set keeps matching reality.