Building a Golden Dataset from Real Usage

Every eval is only as good as the inputs you run it on. A perfect grader scoring twenty tidy examples you typed on a Friday afternoon tells you how your feature handles tidy examples you typed on a Friday afternoon. Real users send three questions in one message, paste stack traces, write in Portuguese, and reply "still broken" to a thread from last month.

A golden dataset is the fixed set of test inputs, with expectations attached, that you run every prompt and model change against. In this lesson you'll build one for Deskmate's drafted-reply feature from real usage: sampling production traffic so the set reflects reality, filling coverage gaps on purpose, labeling efficiently without fooling yourself, and keeping the whole thing fresh as the product changes underneath it.

What a golden dataset is (and isn't)#

A golden dataset is a list of cases. Each case has an input (exactly what the feature receives), expectations (what a good output must and must not do), and metadata (where the case came from and what kind of input it is). "Golden" means the expectations have been checked by someone who knows what good looks like.

It is not:

• A benchmark. Public benchmarks measure general model ability. Your golden set measures whether your feature does its job on your traffic.
• Training data. You don't fine-tune on it. If you ever do, it stops being a fair test.
• A list of perfect answers. For open-ended features, many replies are acceptable. Instead of one "correct" output, most cases store the facts a reply must contain and the things it must avoid.

Write the case format down as a schema so every tool in your eval setup agrees on it:

import { z } from "zod";
 
export const GoldenCase = z.object({
  id: z.string(),
  input: z.object({
    subject: z.string(),
    body: z.string(),
    customerName: z.string().nullable(),
  }),
  expect: z.object({
    requiredFacts: z
      .array(z.object({ fact: z.string(), anyOf: z.array(z.string()).min(1) }))
      .describe("Facts a good reply must convey, each with acceptable phrasings."),
    forbidden: z.array(z.string()).describe("Phrases or claims that must not appear."),
    shouldEscalate: z.boolean().describe("True if the right reply hands off to a human specialist."),
    referenceReply: z.string().optional().describe("An example of a great reply, if one was written."),
  }),
  tags: z.object({
    source: z.enum(["production", "failure", "expert", "synthetic"]),
    category: z.enum(["billing", "login", "bug", "how-to", "account", "other"]),
    traits: z.array(z.string()).describe('For example "multi-request", "angry", "non-english".'),
  }),
  provenance: z.object({
    traceId: z.string().nullable(),
    addedAt: z.string(),
    addedBecause: z.string(),
  }),
});
 
export type GoldenCase = z.infer<typeof GoldenCase>;

A few of these fields earn their place later. requiredFacts and forbidden feed the code-based checks in the next lesson. tags let you report pass rates per slice instead of one blended number. provenance.addedBecause stops a future teammate from deleting the one case that guards against last spring's incident.