Choosing, Routing & Running Models
The right model for each job — hosted, open, or local
Pick models feature by feature, route between them, add gateways and fallbacks, and run open-weight models locally, on your own GPUs, or in the browser — plus fine-tuning and distillation, and staying portable.
- Lessons
- 9
- Reading time
- 94 min
- Level
- Intermediate
- Access
- Full course
Lessons in this module
- 6.111 min
The Model Landscape for Builders
Frontier, mid-tier, and small models; closed and open-weight; general and specialized — how the market is shaped and how to keep up without chasing every release.
- 6.211 min
Choosing a Model per Feature
A practical selection process: define the bar, shortlist, test on your own examples, and weigh quality against latency and cost for each feature.
- 6.310 min
Model Routing and Cascades
Send easy requests to small models and hard ones to big models — rule-based routers, classifier routers, and cascades that escalate on low confidence.
- 6.411 min
LLM Gateways and Provider Fallbacks
Put a gateway between your app and providers for retries, fallbacks, keys, and spend tracking — and know what it costs you in complexity.
- 6.511 min
Building with Open-Weight Models
When open-weight models make sense, how licenses differ, where to run them (hosted inference vs. your own GPUs), and how to serve them with vLLM.
- 6.610 min
Running Models Locally with Ollama and llama.cpp
Run capable models on a laptop for development, privacy, or offline features — quantization explained, and an OpenAI-compatible local endpoint in minutes.
- 6.710 min
On-Device and In-Browser AI
Run small models in the browser and on phones with Transformers.js, WebGPU, and built-in platform models — and what they're genuinely good for.
- 6.810 min
Fine-Tuning and Distillation for Product Features
Hands-on distillation: use a large model to label data, fine-tune a small one for a narrow feature, and prove with evals that it was worth it.
- 6.910 min
Model Portability and Avoiding Lock-In
Keep the freedom to switch models: thin adapters, provider-neutral prompts and schemas, portable evals, and a migration checklist for model upgrades.
About this module
Every AI feature runs on a model, and the model you pick decides most of what users experience: how good the answers are, how fast they arrive, and what each request costs you. Yet many products choose one model early and never revisit it, so a simple tagging job pays frontier prices while the feature that needs a stronger model is stuck on the default.
This module treats model choice as an engineering decision you make per feature and revisit on a schedule. It starts with a stable map of the market: frontier, mid-tier, and small models; closed and open-weight; general-purpose and specialized. Then it turns that map into a repeatable selection process built on a quality bar, a shortlist, and a small harness that measures quality, latency, and cost on your own examples.
From there you go deeper. You will route individual requests to cheaper or stronger models with rules, classifiers, and cascades; put a gateway between your app and its providers for retries, fallbacks, keys, and spend tracking; and decide when open-weight models make sense, including licenses and serving them with vLLM. Later lessons cover running models locally with Ollama and llama.cpp, running small models in the browser and on phones, fine-tuning and distilling a small model for a narrow feature, and keeping the freedom to switch providers without rewriting your product.
Examples use TypeScript, the Anthropic SDK, the OpenAI SDK for compatible endpoints, and a running help-desk app, with code you can adapt directly. By the end of the module you will have a model inventory for your product, evidence behind each choice, and a plan for outages, price changes, and new releases.
Questions about Choosing, Routing & Running Models
How do I choose the right LLM for my app?
Choose per feature rather than per app. Write down a quality bar, a latency budget, and a cost ceiling for the feature, shortlist a few models across tiers that meet your hard constraints, and test them on 30 to 100 real examples. Pick the cheapest option that clears every bar.
What is LLM routing?
LLM routing means choosing a model for each request instead of using one model for everything. Easy requests go to a small, cheap model and hard ones go to a stronger model, decided by rules, a small classifier model, or a cascade that escalates when the cheap model's answer fails a check.
Do I need an LLM gateway?
Not always. Provider SDKs already retry transient errors, which is enough for many apps. A gateway becomes worth it when several apps or teams share provider keys and budgets, or when you need cross-provider fallbacks and spend tracking everywhere, but it adds a network hop, another dependency, and another place your data passes through.
Are open-weight models cheaper than API models?
Sometimes. The weights are free, but running them costs GPU time whether or not the GPUs are busy, plus engineering effort. At high, steady utilization a small open model on a narrow task can be cheaper per request; at low or spiky traffic a per-token API is usually cheaper.
Can I use open-weight models commercially?
Often, but it depends on the license. Some models use permissive licenses like Apache 2.0 or MIT, while others use custom community licenses with conditions such as acceptable-use policies or user thresholds. Read the license for the exact version you deploy and talk to a lawyer for your situation.