Exponential Backoff for OpenAI API Calls

429. Then a 500. Then, twenty minutes later, a connection reset that isn't even in the docs. The call succeeds nine times out of ten and then just doesn't, and if you're firing these requests from a queue worker or a webhook handler, one unhandled failure means a dropped job, a confused user, or a silent gap in your data pipeline.

Exponential backoff for OpenAI API calls is the fix, and it's less code than most people expect — but there are three or four details in the actual implementation that separate "retries that work" from "retries that make the outage worse."

Why a flat retry makes things worse, not better

Retrying immediately on failure feels intuitive, but it's exactly the wrong move when the failure is a rate limit. If ten workers all hit a 429 at the same second and all retry in the same second, you've just resent the same burst that got rejected the first time. I've seen this turn a brief rate-limit blip into a five-minute outage because every retry collided with the next one. The fix isn't "retry," it's "retry with increasing delay, plus a bit of randomness so requests don't all land on the same millisecond."

What's actually worth retrying

Not every failure deserves a retry. A 400 means your request body is malformed — retrying sends the same broken payload again. A 401 means your key is wrong or revoked. Retrying those just burns time and makes your error logs harder to read. What you want to retry:

• 429 (rate limited)
• 500, 502, 503 (server-side failures)
• Network-level errors — timeouts, ECONNRESET, ETIMEDOUT

Everything else should fail fast and bubble up immediately. I'd actually argue most retry wrappers I've reviewed get this backwards — they retry on any thrown error, which means a genuine bug in your request payload gets silently retried three times before anyone notices it's not a transient issue at all.

The actual backoff formula

Three things in there matter more than the loop itself. First, the 30_000 cap — uncapped exponential growth means attempt six is waiting eight minutes, which is useless in a request/response cycle. Second, the jitter — without it, a batch of workers that all started their backoff at the same moment stay synchronized through every retry, which defeats the point. Third, and this is the one people skip: OpenAI's 429 responses include a retry-after header. If it's present, use it instead of your own calculated delay — OpenAI's rate limiter knows its own reset window better than your guess does.

Wiring it into a real API call

import OpenAI from 'openai';
const client = new OpenAI();

const completion = await withBackoff(() =>
  client.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: prompt }],
  })
);

That's the whole integration. The wrapper doesn't care what fn does — it works the same way around embeddings calls, function-calling requests, or a streaming call wrapped to resolve on first chunk.

Where this breaks in production

A plain retry wrapper on a single request is the easy 80%. The part that actually causes incidents is when you have hundreds of these calls fanning out from a queue (BullMQ, SQS, whatever) at once. If every job in the queue independently backs off and retries on its own schedule, you can still create a thundering-herd pattern at the queue level even with per-call jitter, because queue concurrency settings and backoff timing aren't talking to each other.

On a project last year, we had a Redis-backed queue processing AI summarization jobs, and under load the queue's own concurrency limit kept re-triggering retries faster than OpenAI's rate limit reset — so the backoff delay was correct per-call but the system-level throughput still exceeded the actual rate limit. The real fix wasn't a smarter backoff formula, it was lowering queue concurrency so fewer requests were in flight at once, letting the simple backoff above actually do its job instead of fighting a flood. If you're retrying a lot and it's still not settling down, check your concurrency setting before you touch the backoff math again.

Logging retries without flooding your error tracker

Don't console.error on every retry attempt — a transient 429 that resolves on attempt 2 isn't an error, it's normal operation, and logging it as one trains everyone to ignore your alerts. Log retry attempts at debug or info level, and only escalate to an actual alert if maxRetries is exhausted and the call fails for real. That's the signal that actually means something's wrong.

Frequently Asked Questions

Should I use the OpenAI SDK's built-in retry option instead of writing my own? The official openai Node SDK does have a maxRetries client option, and it's a reasonable default for simple scripts. The reason to roll your own is control: the built-in version doesn't expose hooks for your own logging, doesn't let you tune behavior per call type (a background job can tolerate a longer backoff than a user-facing chat response), and won't help at all with the queue-level thundering-herd problem above.

Is it safe to retry a request that might have already partially succeeded, like a function call that triggers a side effect? No — this is the one case where blind retries are actually dangerous. If the model's response triggers something non-idempotent (sending an email, charging a card, writing a row), wrap that side effect with its own idempotency key, not just the OpenAI call itself. Retrying the API call is safe; retrying the action the response causes usually isn't, unless you've explicitly made it safe to repeat.

Key Takeaway

Exponential backoff for OpenAI API calls is a five-line wrapper with three details that matter — a capped delay, randomized jitter, and respecting the retry-after header when it's present. Get those right at the single-call level first; if retries are still piling up under load, the next place to look is your queue's concurrency setting, not a more aggressive backoff curve.