Golden dataset

A golden dataset is the test set behind your evals. Each case is an input the feature will see, such as a ticket, a document, or a question, paired with what a correct output looks like: an exact label, required fields, a reference answer, or a list of things the output must and must not do.

The best golden datasets come from real usage, not invented examples. Sample production traffic, deliberately include the hard and rare cases error analysis surfaced, and add every bug users report. Label carefully, ideally with someone who knows the domain, and note why each expected output is right. Keep a development split for iterating on prompts and a held-out split you check less often, so you don't overfit to the cases you stare at every day.

Golden datasets go stale. Your users, product, and knowledge base change, and a set built at launch gradually stops representing traffic. Version it like code, refresh it from production samples on a schedule, and retire cases whose expected answer is no longer true. Mind privacy too: real inputs contain personal data, so redact it or get permission before copying it into test fixtures.