Time to first token (TTFT)

Time to first token measures how long the user stares at nothing. With streaming, once the first token arrives the response starts appearing, so TTFT often matters more to perceived speed than total generation time. It is distinct from output speed (tokens per second after the stream starts) and from end-to-end latency.

Several things drive it: the size of the input the model must process, the model you chose, provider load and region, and, on reasoning models, thinking, which happens before any visible answer text. Your own code adds to it too, through retrieval, database lookups, and other calls you run before the model call even starts.

To cut it, measure each stage first. Common fixes include prompt caching for long stable prefixes, smaller or faster models for latency-sensitive routes, lower effort, trimming context, running retrieval in parallel with other setup work, and starting the stream before slow, non-essential work finishes. Showing an immediate status in the interface helps perception even when the number can't drop further.

Track TTFT per route as a percentile, not an average; the slow tail is what users remember and what complaints are about.