Cutting Latency: Time to First Token and Beyond
Measure where time goes, then cut it: smaller models, lower effort, shorter outputs, parallel calls, speculative work, and regional endpoints.
A support agent clicks "Draft reply" and watches a spinner for nine seconds. The draft is good, but by second five they've started typing their own. Your feature works, your evals pass, and usage is flat, because slow AI features don't get used. Latency is a quality metric.
The fix is rarely one big trick. Model latency comes from several distinct stages, and each has its own levers: a smaller model, a lower effort level, a shorter output, a cached prompt, two calls in parallel instead of in sequence, work started before the user asks. In this lesson you'll measure where the time actually goes in a feature, then cut it stage by stage, and set a latency budget that keeps it fast as the feature grows. Making waits feel shorter is a design problem covered in Latency UX; this lesson is about making them actually shorter.
Where the time goes
A single model-backed request has more stages than it looks like from the outside:
| Stage | What happens | What makes it slow |
|---|---|---|
| Prep | Your code: auth, database reads, retrieval, building the prompt | Sequential queries, slow vector search, an embedding call |
| Network and queueing | The request travels to the provider and waits for capacity | Distance to the provider, cold connections, busy periods |
| Input processing | The model reads the whole prompt | Long prompts, especially uncached ones |
| Thinking | The model reasons before writing visible output | Higher effort levels, hard problems |
| Generation | Output tokens stream out one at a time | Long outputs, slower models |
| Post-processing | Validation, parsing, retries, follow-up calls | Retries on bad output, chained calls |
Two numbers summarize most of this. Time to first token (TTFT) is how long until the first visible output arrives: prep, network, input processing, and thinking. Total time adds generation and post-processing. Which one matters depends on who consumes the output:
- If the output streams to a person, TTFT dominates the experience. A reply that starts in under a second and finishes in eight often feels faster than one that appears all at once after four.
- If code consumes the output (a classification that routes a ticket, a JSON object a UI needs complete), only total time matters, because nothing is usable until the end.
Generation time scales almost linearly with output length. A useful approximation: total time ≈ TTFT + output tokens ÷ output speed. If a model streams, say, 80 tokens per second, a 600-token answer takes about 7.5 seconds after the first token, and a 150-token answer takes about 2. That's why output length is usually the biggest lever you have.