Logging and Tracing LLM Calls
Capture every model call with inputs, outputs, tokens, latency, and cost — with OpenTelemetry or an LLM observability tool — while respecting privacy.
A customer writes in: "Your AI summary told my agent I'd asked for a refund. I didn't." To answer, you need to know which prompt version produced that summary, which model ran it, what the model was actually sent, what came back, and whether it was a one-off or the hundredth time this week. Without a record of the call, you're left reproducing it by hand and hoping the model does the same thing twice. It usually won't.
In this lesson you'll give every model call in Deskmate a durable record: one row per call with feature, prompt version, model, tokens, cost, latency, and outcome, tied into a trace for the request that caused it. You'll decide where that data should live, how to log prompt and response content without creating a privacy problem, and which queries make the logs pay for themselves.
What every AI call record needs
Regular application logs answer "did the request succeed?" AI call logs have to answer three harder questions:
- Why did this output happen? You need the exact inputs, the prompt version, the model, and the settings.
- What does this cost, and who is spending it? You need tokens broken down by type, a cost estimate, and the feature and customer responsible.
- Is it getting worse? You need consistent fields over time so you can compare this week to last week, and version 7 of a prompt to version 6.
That means a fixed set of fields for every call. Treat it as a schema: if one code path skips promptVersion, every dashboard built on top silently lies.
| Field | Example | What it answers |
|---|---|---|
| Call ID, trace ID | uuid, 4bf92f35… | Find this call; find everything else in the same user request |
| Provider request ID | req_011C… | What the provider needs if you open a support case |
| Feature | ticket-summary | Cost and quality per feature |
| Prompt version | ticket-summary@7 | Did the last prompt change cause this? |
| Model and effort | claude-sonnet-5-5, low | Did a model change cause this? |
| Tenant and user | acct_42, usr_9 | Cost per customer; support lookups |
| Status and stop reason | ok, max_tokens | Truncations, refusals, errors |
| Tokens | input, output, thinking, cache read, cache write | The bill, broken down |
| Estimated cost | 0.0041 | Spend per feature and per customer |
| Latency | total ms, time to first token for streams | Where slowness lives |
| Content (by policy) | prompt and response, or a redacted copy | Debugging and error analysis |
The provider request ID is the first thing a provider asks for when you report a problem. The prompt version only exists if you version prompts; Versioning and Managing Prompts shows how.