Debugging Prompts Systematically

A support lead forwards you a drafted reply that told a customer their data export "will be ready within 24 hours." Nothing in Deskmate's help articles promises that. You open the prompt, add "Never promise timelines" in capital letters, run the ticket once, get a good answer, and ship. A week later, drafts have started refusing to tell customers that password resets are instant, because that's a timeline too.

That loop of patching a sentence, eyeballing one output, and shipping is how most prompts get debugged, and it's why so many prompts end up as a pile of contradictory rules. This lesson gives you a method instead: reproduce the failure exactly, isolate where it comes from, form one hypothesis, change one thing, and confirm the fix on a test set with enough runs to tell signal from noise.

Why one good output proves nothing#

Two properties of model calls make casual debugging unreliable.

Outputs vary between runs. Current Claude models don't accept sampling parameters like temperature, so you can't pin outputs to be identical (see Output Controls). The same request can succeed nine times and fail on the tenth. If the original failure happens one time in five, a single passing run after your change tells you almost nothing: you'd see a pass 80% of the time even if the change did nothing at all.

Prompts are global. A sentence you add to fix one input applies to every input. Software engineers are used to fixes with local effects; a prompt fix is more like changing a default that every caller inherits. That's how "never promise timelines" breaks password-reset replies.

So the method has two non-negotiable parts: measure failure rates over multiple runs, and check fixes against a set of cases that includes the ones that were already working.