LLM observability (tracing)

LLM observability is logging and tracing designed around model calls. A useful record captures the feature, model and prompt version, inputs and outputs (or references to them), token counts, cost, latency, stop reason, errors, and the user and account involved. Tracing links related steps, such as retrieval, a model call, a tool call, and a follow-up call, under the request that caused them, so you can see the whole chain.

These records feed almost everything else in running an AI product. Error analysis reads them, golden datasets are sampled from them, and cost per user, drift detection, quality monitoring, incident response, and answering a customer's "why did the AI say that?" all depend on them. You can write a thin logging wrapper around your model client and store rows in your own database, emit OpenTelemetry traces to an existing backend, or adopt a dedicated LLM observability platform.

The catch is that full prompts and outputs contain user data. Decide what you store, for how long, and who can see it; redact where you can; and make sure retention matches what you've promised customers. Always record prompt and model versions, or you won't be able to tell which change caused a regression.