Shipping reliable AI apps
Everything that keeps an LLM feature working in production, in watch order: reliable JSON, retries and timeouts, observability, and agents that don't loop. Hand-drawn doodle explainers, theory then a practical example.
1. Structured Outputs: Reliable JSON from a Model
Why please return JSON breaks in production, and the ladder that fixes it: JSON mode, constrained decoding, tool calling, then validate and retry.
2. Retries, Timeouts and Fallbacks for LLM Calls
Why instant retries make an outage worse and why 200 OK can lie: backoff with jitter, Retry-After, timeout budgets, idempotency keys and tested fallbacks.
3. Observability for LLM Apps
How to catch the failures your dashboard says aren't happening: traces, spans, what to log on every model call, privacy, sampling, quality scores and alerts.
4. Agents in Production: Why They Loop
Why your agent can loop all weekend, and the guardrails that make it stop: step caps, budgets, timeouts, loop detection, idempotent tools, a kill switch and traces.
5. Rate Limiting LLM APIs: Token Buckets, Queues and 429s
How OpenAI and Anthropic rate limit you (requests and tokens, token buckets, 429s) and how to stay under it: your own bucket, a queue, the batch API, one shared limiter.
6. What an Agent Harness Is, and a Tiny One You Can Build
An agent is a model plus a harness: the loop, tools, stop rules, context trimming and permissions around it. Here's each part, and a tiny harness in Python.
7. Context Rot Explained
Models read long prompts worse than short ones. Why context rot happens, what the research shows, how to measure it in your app, and what to trim.
play with it
What retries buy you
Each try fails some of the time. How often do all of them fail, and how long can a user wait?
All 3 tries fail: 1.6% of requests
Worst-case waiting between tries: 3.0 s (1s + 2s)