Rate Limiting LLM APIs: Token Buckets, Queues and 429s

What your limit counts

Theory

OpenAI counts requests and tokens per minute (and per day); Anthropic counts requests, input tokens and output tokens per minute. Limits belong to your organization or project, not to a user or a server, and the first one to run out blocks you.

Practical example: Which limit runs out first

500 requests and 200,000 tokens a minute, about 1,500 tokens per call: 200,000 / 1,500 = about 133 calls a minute. Tokens run out first.

The token bucket

Theory

A bucket holds permits up to a maximum and refills at a steady rate; each request takes permits, and an empty bucket means the request is rejected. For a requests limit a call costs one permit; for a tokens limit it costs one permit per LLM token. The per-minute number is the refill rate, and Anthropic notes 60 a minute might be enforced as 1 a second, so bursts can trip it.

Practical example: A burst of twenty

A bucket that holds 5 and refills 1 a second: fire 20 requests at once and 5 go through, 15 are rejected, even though 20 is far under 60 a minute.

429s and retries

Theory

429 means too many requests in a given time. Wait at least retry-after plus a small random delay; on OpenAI, failed requests still count toward your per-minute limit, so hammering makes it worse. On Anthropic, a spend-cap 429 has no retry-after and won't clear by retrying, and a fast traffic ramp can 429 even under your limits.

Practical example: retry-after: 56

Wait at least 56 seconds, plus a little random delay. No retry-after on a spend-cap 429? Stop retrying and fix the billing.

Your own token bucket

Theory

Run a token bucket in your app, a little under the provider's limit, and take each call's estimated tokens (input plus expected answer) before sending; read the remaining-tokens headers to stay in sync. On OpenAI each call counts as the larger of max_tokens and its estimated size, so set max_tokens close to the answer you expect; Anthropic's output limit counts only generated tokens.

Practical example: Refill 3,000 a second, hold 15,000

For a 200,000 tokens/min limit, refill at 180,000 a minute (3,000 a second) and hold only a few seconds' worth (15,000), so the bucket can't fire a big burst; each 1,500-token review waits at most half a second when it's short. With max_tokens 8,000 on OpenAI, the same limit allows only 25 calls a minute.

import time

class TokenBucket:
    def __init__(self, capacity, per_second):
        self.capacity, self.rate = capacity, per_second
        self.tokens, self.last = capacity, time.monotonic()

    def take(self, n):
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= n:
            self.tokens -= n
            return 0.0
        return (n - self.tokens) / self.rate  # seconds to wait

bucket = TokenBucket(15_000, 180_000 / 60)  # refills 3,000/s, under a 200,000 TPM limit; holds 5 s
wait = bucket.take(1_500)
print(wait)  # 0.0: send now

Queues, backpressure and batches

Theory

A bounded queue in front of the bucket (say 1,000 waiting requests) absorbs bursts while the bucket sets the pace; when the queue is full, push back quickly instead of hanging, and if you run your own API answer with 429 or 503. Put live users ahead of background jobs, and send work nobody waits for to the batch API.

Practical example: 50,000 reviews

50,000 reviews at about 133 a minute take over six hours anyway, so send them to OpenAI's Batch API: half the price, a separate pool of higher limits, a 24-hour turnaround, and the live chatbot's limits untouched.

Many servers, one limit

Theory

Each server running its own full-size bucket multiplies your traffic by the number of servers. Either split the limit per server, which wastes idle shares, or share one bucket in Redis, taken through a Lua script that Redis runs atomically, and fail open if Redis is down, since the provider's limit is still the backstop.

Practical example: Four servers

Four servers with 200,000-token buckets can send 800,000 tokens a minute. Split: 50,000 each. Shared: one Redis bucket for all four.