Retries, Timeouts and Fallbacks for LLM Calls

200 isn't OK

Theory

With streaming, the 200 is sent first and the words after, so if the stream breaks midway the error shows up inside the stream. A reply can also be cut off at your token limit, be JSON that doesn't parse, or be confident nonsense. A failure is anything the user can't use; the status code is just a suggestion.

Practical example: Check before you trust

Check the stop reason, validate the output, and treat a bad answer like any other failed call.

import json

def usable(stop_reason, text):  # stop reason names differ by API
    if stop_reason in ("max_tokens", "length"):  # cut off at the token limit
        return False
    try:
        json.loads(text)  # the answer must parse
    except ValueError:
        return False
    return True

print(usable("max_tokens", '{"answer": "half a sent'))  # False: treat as a failed call

Retry or not

Theory

Transient errors fix themselves: a dropped connection, a server error, the provider saying it's overloaded. Permanent errors never do: a bad request, a wrong API key, no permission, a model name with a typo. Status codes give the first hint: 400s mean your request is wrong, 500s mean their day is wrong, and 429 means your pace is wrong.

Practical example: A tiny classifier

Retry transient errors later, fix permanent ones, and always cap the number of tries. A 429 can also mean a spending cap: those carry no Retry-After and keep failing until access resumes.

MAX_ATTEMPTS = 4  # always cap the tries

def is_transient(status):
    # 5xx: their day is wrong. 429: your pace is wrong (slow down, then retry).
    # Other 4xx (bad request, wrong key, no permission): fix the request instead.
    # Careful: a 429 from a spending cap will not fix itself.
    return status == 429 or status >= 500

print([is_transient(s) for s in (400, 401, 429, 500, 503)])

Backoff and jitter

Theory

Waiting zero seconds between tries makes an outage worse. Exponential backoff waits one second, then two, four, eight; but if many clients fail at once, they retry together and fail together. Jitter adds randomness so the crowd spreads into a trickle; a cloud provider measured that backoff alone still clusters calls, and jitter spreads them out.

Practical example: Full jitter

Sleep a random time between zero and min(cap, base times 2 to the attempt). Exponential, capped, jittered, limited.

import random

def backoff_delay(attempt, base=1.0, cap=8.0):
    # full jitter: a random wait between 0 and the capped exponential delay
    return random.uniform(0, min(cap, base * 2 ** attempt))

for attempt in range(4):  # ceilings: 1, 2, 4, 8 seconds
    print(round(backoff_delay(attempt), 2))

Rate limits

Theory

Providers cap how fast you can call, in requests per minute and tokens per minute; go over and you get a 429. Good providers send a Retry-After header, a number of seconds or an HTTP date; when it's there, trust it over your own guess, and without it fall back to backoff with jitter. Many providers count failed requests against your limit too, so hammering a 429 keeps you locked out longer.

Practical example: Honor Retry-After

Read the header if present (seconds or a date), otherwise use your jittered backoff delay.

from datetime import datetime, timezone
from email.utils import parsedate_to_datetime

def wait_seconds(retry_after, fallback):
    if retry_after is None:  # no header: back off with jitter instead
        return fallback
    if retry_after.isdigit():  # a number of seconds
        return int(retry_after)
    when = parsedate_to_datetime(retry_after)  # or an HTTP date
    return max(0.0, (when - datetime.now(timezone.utc)).total_seconds())

print(wait_seconds("20", fallback=1.3))

Timeout budgets

Theory

A model can legitimately think and write for tens of seconds, so a timeout that is too short kills good answers you still pay for, and one too long leaves users staring at a spinner. One way to choose: pick a false-timeout rate you can live with and check latency at that percentile. Start from how long the user will wait, spend that budget across attempts, and pass the deadline down the stack.

Practical example: One deadline, shared

With a thirty-second budget, a retry can't also take thirty. With streaming, watch time to first token and the gap between chunks, and give up on an attempt early if no first token arrives. SDKs generally don't retry a half-delivered stream for you: text can sometimes be resumed, a half-done tool call can't.

import time

BUDGET = 30.0  # seconds the user will wait, end to end
deadline = time.monotonic() + BUDGET

def time_left():
    return max(0.0, deadline - time.monotonic())

# each attempt gets only what is left, and passes the deadline down:
# call_model(prompt, timeout=time_left())
print(round(time_left(), 1))

Safe retries

Theory

A request can succeed while its response is lost, so a retry does the work twice. For a plain model call that mostly costs tokens (you're billed per attempt), but if the answer triggers an action like sending an email or charging a card, twice is a real problem. The HTTP spec calls a request idempotent when repeating it has the same effect as doing it once: reads and deletes are, creating or charging usually is not.

Practical example: Idempotency key on the side effect

Send a unique id per action with every attempt; the server sees it already did the work and returns the saved result. Retry the model call freely, but put the key on the side effects.

import uuid

sent = {}  # server side: idempotency key -> saved result

def send_email(key, to):
    if key in sent:  # already done: hand back the saved result
        return sent[key]
    sent[key] = "sent to " + to
    return sent[key]

key = str(uuid.uuid4())  # one key per action, reused on every attempt
for attempt in range(3):
    send_email(key, "customer")
print(len(sent))  # 1: the email went out once

Fallbacks

Theory

Retrying only helps if the same call might work next time. Plan B is a different model or provider, but it isn't a clone: it can be worse at your task, format tool calls differently, or have less context. A circuit breaker opens after enough failures, sends calls straight to the fallback, later lets one test call through, and closes if it works.

Practical example: Test the spare tire

Run your fallback through the same evals as your primary. A fallback you've never tested is just a hope.

Retry storms

Theory

Retries at several layers multiply. If the frontend, backend and SDK each make up to four attempts, one user click becomes sixty-four calls to a provider that is already struggling (the example is from Google's SRE book). Everybody retries, nobody recovers.

Practical example: Retry at one layer

Pick one layer to retry and let the others pass the error up. Add a retry budget: only so many retries per minute per process, then fail fast.