Shipping reliable AI apps, part 3 of 7
Observability for LLM Apps
How to catch the failures your dashboard says aren't happening: traces, spans, what to log on every model call, privacy, sampling, quality scores and alerts.
After this video
You can instrument an LLM app with traces and spans, log the right fields on every model call, keep it private and affordable, and turn bad traces into eval cases.
The short version
After this you can see inside your LLM app while it runs, instead of trusting a dashboard full of 200s. You can record each request as a trace of nested spans, log the model, settings, prompt version, token counts, finish reason, errors, latency and cost on every model call, keep message content private and sampling smart, score a sample of live traffic and user feedback, alert on cost, speed, errors and quality, and replay bad traces as regression tests.
A 200 hides it
Normal failures are loud: crashes, timeouts, stack traces. Model failures are quiet: the request succeeds and the model says something wrong, gets cut off, or answers from the wrong document. Hit the max token limit and the response still comes back successful; the only clue is the stop reason. Status code, latency and error rate tell you the call happened, not that it was good.
Truncation behind a 200. Check the stop reason on every successful response; its names differ by provider (end_turn or max_tokens on one, stop or length on another).
Traces and spans
A trace is the full path of one request through your app. It's made of spans: one unit of work each, with a name, a start time, an end time and some notes. Spans nest; the root span is the whole request and has no parent, and every span in a trace shares one trace ID. The common standard is OpenTelemetry, whose generative AI conventions name steps like chat, retrieval and execute tool (they are marked Development, so names may change).
A support bot's trace. The root span answers the customer's question; its children are retrieval, a model call, a tool call to look up the order, and another model call that writes the reply. When an answer is bad, the trace shows which step went wrong.
What to log
On every model call, record the model name, the settings you chose (like temperature and max tokens), a prompt version label, input and output token counts (providers return them with the response), and the finish reason. Record errors with a type: rate limited, timed out, bad request. Retrieval spans get the query and which documents came back; tool spans get the tool, a short summary of the arguments, the duration and whether it failed.
A tidy model span. Attribute names below follow the OpenTelemetry generative AI conventions; the prompt version is your own custom attribute, not an OpenTelemetry field. Values are placeholders.
Speed and cost
Latency has two flavours: total time and, if you stream, time to first token, which is how long the user stares at a blank screen (OpenTelemetry calls it time to first chunk). They fail for different reasons, so record both. Providers bill by tokens and the response carries token counts, not dollars, so compute cost as tokens times your price list when you log.
Cost on the span. Store the cost and group it by feature, prompt version and customer to find the prompt eating the budget. Watch retries too: each one is another span and another bill.
Privacy and sampling
Prompts and answers can hold names, emails and pasted secrets; OpenTelemetry treats message content as opt-in and warns it likely contains personal data. Always log metadata, log content only with a reason, redact before it leaves your app, and set retention and access limits. For volume, head sampling decides at the start and can drop the failed trace; tail sampling decides when the trace is done and can keep every error and slow trace, but is harder to run.
Keep the traces you'd open. Sample the detailed traces, not your counters: totals should still count every request.
Quality scores
Nothing in a span says whether the answer was good. Sample live traffic and score the sample: cheap checks first (valid JSON, cited a source, refused when it shouldn't), then a judge model with a rubric for the squishy parts. User signals like a thumbs-down, a regenerate click or a rephrased question are imperfect, but a rising rate is a smoke alarm.
Scores with an address. Attach every score and every piece of feedback to the trace ID, so a low score points straight at the steps that caused it.
Dashboards and alerts
Four dashboards matter: cost (per day, by feature and prompt version), latency (percentiles, not the average), error rate (rate limits, timeouts, and finish reasons like length and refusal), and quality (sampled scores and thumbs-down rate per prompt version). Page a human only for a cost spike, error jump or quality drop, compared against what's normal for that time of week.
Averages hide unlucky users. A prompt is a deploy: when a score drops after a prompt update, line it up with the prompt version on every span and roll back. Made-up latencies below show why you track the slowest five percent.
Replay failures
Every bad trace is a free test case: it already has the input, the retrieved documents, the tool results and the output. Copy it into your eval dataset with what a good answer should have been, and it becomes a regression test. Scrub personal data before it goes into the dataset.
From trace to test. Replay the case against your fixed prompt: if it passes, ship it. Observe, find a failure, save it as a test, fix it, and watch the score.
Check yourself
Your dashboard shows only 200 OK and zero errors, but users get cut-off answers. Which logged field would have caught it?
The finish (stop) reason. Hitting the max token limit still returns a successful response; the stop reason is the only clue, so record it on every model span.
Why can head sampling lose the trace you most need, and what does tail sampling do instead?
Head sampling decides at the start, before it knows how the request ends, so it can drop the one failed trace. Tail sampling waits until the trace is done, keeps every error and slow trace, and samples the rest.
A quality score dropped this afternoon. How do you find the cause and stop it happening again?
Line up the drop with the prompt version logged on every span and roll back if a prompt change caused it. Open the bad traces to see which step failed, copy them into your eval dataset with a good answer, and replay them against the fix.