Skip to content

Search articles, repositories, topics and pages.

Infrastructure

Designing Retries for LLM APIs

Timeouts, exponential backoff with jitter, circuit breakers and fallbacks — a small, testable retry policy for calling model APIs.

TIAN NEWS AI

Published Updated 2 min read

A request passing through a retry policy with timeout, backoff and jitter, a circuit breaker and a fallback model before reaching the model API

Why model APIs need a policy

Model APIs fail in more ways than a typical REST service: rate limits, overloaded capacity, slow first tokens and long generations that time out. Retrying blindly makes things worse, because every retry adds load at the moment the provider is already struggling.

Decide what to retry

FailureRetry?Why
429 rate limitUsually, within a bounded budgetHonour Retry-After and provider-specific quota errors
5xx or overloadedUsually, with backoffOften temporary, but retries must still be capped
TimeoutOnly when duplicate execution is safeThe original request may still be running or billable
400 invalid requestNoRetrying cannot fix the input

Backoff with jitter

Exponential backoff spaces retries out; jitter spreads clients apart so they do not all retry at the same moment.

retry.ts, ts
type Attempt<T> = () => Promise<T>;

const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));

export const withRetry = async <T>(
  attempt: Attempt<T>,
  isRetryable: (error: unknown) => boolean,
  { retries = 3, baseMs = 500, maxMs = 8_000 } = {},
): Promise<T> => {
  for (let index = 0; ; index += 1) {
    try {
      return await attempt();
    } catch (error) {
      if (index >= retries || !isRetryable(error)) throw error;
      const ceiling = Math.min(maxMs, baseMs * 2 ** index);
      await sleep(Math.random() * ceiling);
    }
  }
};

This is "full jitter": the delay is a random value between zero and the exponential ceiling. It is simple and spreads load well.

Here retries = 3 means three retries after the initial call, so one request can produce at most four attempts. Name and document that distinction explicitly in production code.

If you are replacing a fixed delay, the change is small:

ts
const ceiling = Math.min(maxMs, baseMs * 2 ** index);
await sleep(ceiling); 
await sleep(Math.random() * ceiling); 

Circuit breaker and fallback

When most recent calls fail, stop calling for a short cool-down instead of retrying every request. During the cool-down, either fail fast with a clear message or send the request to a compatible fallback model. Compatibility is a policy decision: context limits, tool support, structured-output behaviour, safety settings and cost can all differ between models.

For timed-out requests, retry only when the operation is idempotent or the provider gives you a request identifier that supports reconciliation or deduplication. Otherwise you can pay for two generations and receive two side effects.

Respect Retry-After

If the provider returns a Retry-After header, wait at least that long. Ignoring it can get your key throttled for longer.

Test the policy

Inject the clock and the random source so tests can check the delays without waiting. Test the decision table above directly: each failure type should map to exactly one behaviour.

Key takeaway

Retry only what can succeed, back off with jitter, and stop calling a provider that is clearly down.

  1. Exponential Backoff and Jitter — the full-jitter strategy and its load-spreading rationale.