Fine-Tuning and Distillation for Product Features
Hands-on distillation: use a large model to label data, fine-tune a small one for a narrow feature, and prove with evals that it was worth it.
Deskmate's ticket triage works. A mid-tier hosted model reads every incoming ticket, picks a category, and flags the urgent ones, and your evals say it's right almost every time. Then you try the obvious cost-saving move: point the same prompt at a small open-weight model you can run on one modest GPU, or on a customer's own server. It's fast and cheap, and it gets noticeably more tickets wrong, including some urgent ones.
That gap is what distillation closes: using a large model's outputs to train a small model on one narrow task. In this lesson you'll do it end to end: label real tickets with a large "teacher" model, fine-tune a small open-weight "student" with LoRA, serve it behind an OpenAI-compatible endpoint, and run the evals that tell you, with numbers, whether it was worth it.
When distillation is the right move
Fine-tuning means continuing to train an existing model on your own examples so its weights change. Distillation is fine-tuning where the examples come from a stronger model: the teacher shows the right answer, the student learns to imitate it. For product features, it's the most practical way to make a small model good at one thing.
It earns its cost when all of these are true:
- The task is narrow and stable. Triage into five categories, extracting the same fields, tagging with a fixed taxonomy. Not open-ended chat.
- A large model already does it well. You have a working prompt and an eval set proving it. Distillation copies a capability; it can't create one.
- The output is checkable. Labels and structured fields can be scored automatically. Free-form prose is much harder to verify.
- Something forces you off the big model. Volume that makes per-token pricing hurt, latency a hosted call can't meet, or data that can't leave your infrastructure.
If the last condition isn't true, stop. Try, in order: a better prompt, a smaller hosted model, routing easy requests to it (Model Routing and Cascades), and caching. Run the arithmetic honestly, too. Suppose a triage call uses about 1,000 input tokens and 300 output tokens (including thinking). On Sonnet 5.5 at the time of writing (November 2026), that's about half a cent per ticket. On Haiku 5.5 it's about a fortieth of a cent, or roughly $250 per million tickets. A self-hosted student has to beat that after paying for a GPU that runs all month and the engineering time to maintain it. The Agent Builder Course has a broader take on that decision in When to Fine-Tune.