Prototyping AI Features Fast
Test whether an AI feature is worth building in days: prompt prototypes, Wizard-of-Oz tests, and a prototype-to-production checklist.
The expensive way to learn that an AI feature isn't worth building is to build it: design the UI, wire up the backend, add streaming, write the evals, ship it, and then watch usage flatten out after the first week. Two different things can kill an AI feature, and you can test both of them in days, before writing any production code. Either the model can't do the task well enough on your real inputs, or users don't value the result enough to change how they work.
This lesson covers fast ways to test each risk: a prompt prototype that runs your idea against dozens of real inputs in an afternoon, Wizard-of-Oz and clickable prototypes that test whether people want the output, and a checklist for turning a prototype that passed into a feature you can ship. The examples use a proposed Deskmate feature: drafting a first reply to every new ticket.
The two risks a prototype has to retire
Every AI feature idea carries two separate risks, and they need different tests:
| Risk | The question | How to test it | Who's involved |
|---|---|---|---|
| Capability risk | Can a model do this well enough on our real inputs? | Run prompts over real examples and grade the outputs | You, plus someone who knows what good looks like |
| Value risk | If it worked, would people use it and change their behavior? | Put realistic outputs in front of users in their real context | Real users |
Teams usually test neither properly. They demo the idea on three hand-picked inputs (which tells you little about capability) and ask colleagues whether they like it (which tells you little about value). The methods below are almost as fast and far more informative.
Test capability first when you're not sure the model can do the task at all. Test value first when the task is clearly within reach but you're not sure anyone needs it. For most features, you can run both in the same week.