Model APIs fail in more ways than a typical REST service: rate limits, overloaded capacity, slow first tokens and long generations that time out. Retrying blindly makes things worse, because every retry adds load at the moment the provider is already struggling.
| Failure | Retry? | Why |
|---|---|---|
| 429 rate limit | Usually, within a bounded budget | Honour Retry-After and provider-specific quota errors |
| 5xx or overloaded | Usually, with backoff | Often temporary, but retries must still be capped |
| Timeout | Only when duplicate execution is safe | The original request may still be running or billable |
| 400 invalid request | No | Retrying cannot fix the input |
Exponential backoff spaces retries out; jitter spreads clients apart so they do not all retry at the same moment.
type Attempt<T> = () => Promise<T>;
const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms));
export const withRetry = async <T>(
attempt: Attempt<T>,
isRetryable: (error: unknown) => boolean,
{ retries = 3, baseMs = 500, maxMs = 8_000 } = {},
): Promise<T> => {
for (let index = 0; ; index += 1) {
try {
return await attempt();
} catch (error) {
if (index >= retries || !isRetryable(error)) throw error;
const ceiling = Math.min(maxMs, baseMs * 2 ** index);
await sleep(Math.random() * ceiling);
}
}
};This is "full jitter": the delay is a random value between zero and the exponential ceiling. It is simple and spreads load well.
Here retries = 3 means three retries after the initial call, so one request can produce at most four attempts. Name and document that distinction explicitly in production code.
If you are replacing a fixed delay, the change is small:
const ceiling = Math.min(maxMs, baseMs * 2 ** index);
await sleep(ceiling);
await sleep(Math.random() * ceiling); When most recent calls fail, stop calling for a short cool-down instead of retrying every request. During the cool-down, either fail fast with a clear message or send the request to a compatible fallback model. Compatibility is a policy decision: context limits, tool support, structured-output behaviour, safety settings and cost can all differ between models.
For timed-out requests, retry only when the operation is idempotent or the provider gives you a request identifier that supports reconciliation or deduplication. Otherwise you can pay for two generations and receive two side effects.
Respect Retry-After
If the provider returns a Retry-After header, wait at least that long. Ignoring it can get your key throttled for longer.
Inject the clock and the random source so tests can check the delays without waiting. Test the decision table above directly: each failure type should map to exactly one behaviour.
Key takeaway
Retry only what can succeed, back off with jitter, and stop calling a provider that is clearly down.
- Exponential Backoff and Jitter — the full-jitter strategy and its load-spreading rationale.