Distillation

Distillation is fine-tuning where the training examples come from a stronger model. A large teacher model processes real inputs from your feature, and its answers become the labels; a small student model is then trained to reproduce them. Done well, the student handles that one task nearly as well as the teacher, at a fraction of the cost and latency.

The product recipe: start from a task the large model already does well, proven by an eval set. Run the teacher over a few thousand real inputs, filter or correct its labels, fine-tune the student (usually with LoRA), and evaluate on held-out cases the student never trained on. Serve it behind the same interface as the original so you can switch back instantly.

Distillation copies a capability; it doesn't create one, so the teacher's mistakes become the student's habits. It suits narrow tasks with checkable outputs, such as classification and field extraction, far better than open-ended writing. Before you start, do the arithmetic against a small hosted model and routing, because a self-hosted student has to beat those after paying for GPUs and upkeep. And read your provider's terms: some restrict using model output to build competing models.