Rate Limits and Retries

The Anthropic SDK already retries 408, 409, 429 and 5xx errors with exponential backoff (2 retries by default). Change it with anthropic.Anthropic(max_retries=5). Write your own loop only when you need different behaviour, such as falling back to another model.

An LLM API is a shared service with finite GPUs, so providers set rate limits per organisation: requests per minute, input tokens per minute and output tokens per minute. They protect the service from one customer's traffic spike, and protect you from a runaway bug that would otherwise burn your budget. Errors are a normal part of operating at scale, not a sign that something is broken.

The question is how to retry. Retrying immediately makes things worse: if 1,000 requests fail together and all retry at once, they hit the API together again (a thundering herd). Exponential backoff waits longer after each failure, for example 1 s, 2 s, 4 s, 8 s, so the load spreads out and the service has time to recover. Jitter multiplies each wait by a random factor so that clients that failed at the same moment do not retry at the same moment. It is like a crowded railway ticket counter: if everyone who was turned away comes back at exactly 10:05, the queue forms again; if they come back at random times over the next few minutes, everyone gets served.

• Retry only transient errors (429, 5xx, 529, timeouts). A 400 will fail the same way every time.
• Respect the retry-after header when the API sends one.
• Cap the number of retries and the maximum wait, then fail clearly or fall back.
• Make retried actions idempotent (safe to repeat), so a retried "create return" does not create two returns.

import random
rng = random.Random(42)

class RateLimited(Exception): pass
class BadRequest(Exception): pass

responses = iter([RateLimited(), RateLimited(), RateLimited(), "ok: answer text"])
def flaky_api():  # stand-in for client.messages.create
    r = next(responses)
    if isinstance(r, Exception):
        raise r
    return r

def call_with_backoff(fn, max_retries=5, base=1.0, cap=30.0, sleep=None):
    for attempt in range(max_retries + 1):
        try:
            return fn()
        except BadRequest:
            raise                                   # never retry client errors
        except RateLimited:
            if attempt == max_retries:
                raise
            delay = min(cap, base * 2 ** attempt) * (0.5 + rng.random() / 2)   # jitter
            sleep(delay)

waits = []
print(call_with_backoff(flaky_api, sleep=waits.append))
print("waited:", [round(w, 2) for w in waits], "seconds")

ok: answer text
waited: [0.82, 1.03, 2.55] seconds

Three rate-limit errors, then success. The base waits would be 1, 2 and 4 seconds; jitter scales each to between 50% and 100%, giving about 0.82, 1.03 and 2.55 seconds. A client error (BadRequest) would be re-raised at once without waiting.

class TokenBucket:
    def __init__(self, rate_per_sec, capacity):
        self.rate, self.capacity, self.tokens, self.last = rate_per_sec, capacity, capacity, 0.0
    def allow(self, now):
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket(rate_per_sec=2, capacity=3)
for t in [0.0, 0.1, 0.2, 0.3, 0.4, 1.0, 1.1, 2.5]:
    print(f"t={t:.1f}s -> {'send' if bucket.allow(t) else 'wait (queue it)'}")

t=0.0s -> send
t=0.1s -> send
t=0.2s -> send
t=0.3s -> wait (queue it)
t=0.4s -> wait (queue it)
t=1.0s -> send
t=1.1s -> send
t=2.5s -> send

A token bucket holds up to capacity tokens and refills at rate tokens per second; each request spends one. The bucket starts full, so the first three requests go at once (a burst). At 0.3 s and 0.4 s the bucket is empty and requests must wait. By 1.0 s it has refilled enough to send again. Limiting yourself like this is smoother than hitting the provider's limit and handling 429s.

• Summary: rate limits are normal; plan for 429 and 5xx errors.
• Retry transient errors with exponential backoff and jitter; never retry 4xx client errors.
• The SDK retries for you; customise only when you need fallbacks.
• Throttle on your side with a token bucket and make retried actions idempotent.