OpenAI API Rate Limit: Fix 429 Errors with Retry, Backoff, and Model Routing

An openai api rate limit error is not solved by adding an infinite retry loop. A production client needs to identify which limit was reached, wait for the right amount of time, reduce pressure on the endpoint, and decide whether the request should be retried at all. Otherwise, retries become extra traffic, increase latency, and can make the limit harder to recover from.

The most visible symptom is HTTP 429 Too Many Requests. It can mean temporary requests-per-minute or tokens-per-minute pressure, but it can also indicate that an account has run out of quota or needs a billing change. Those cases need different actions. A short backoff can recover a burst; it cannot create more account balance.

This guide builds a rate-limit strategy for Python applications. It covers the official SDK retry behavior, manual handling for custom HTTP clients, concurrency and token budgeting, multi-key and multi-model routing, observability, testing, and the differences between OpenAI direct access and an OpenAI-compatible gateway such as OurToken.

Verification note: The retry guidance, 429 behavior, Retry-After, SDK retry behavior, and exponential-backoff examples were checked against the official OpenAI rate limits documentation on 2026-09-02. OurToken endpoint examples use the public OurToken configuration docs. Account-specific limits and model availability must be verified in the relevant live dashboard or model catalog.

What an OpenAI API Rate Limit Means

An API limit is a control on how much traffic a project, organization, key, model, or endpoint can process during a time window. Limits are commonly expressed as requests per minute (RPM) and tokens per minute (TPM), but a provider can also enforce daily quotas, concurrent-request limits, or spend limits.

Think of every request as consuming two resources:

request rate  -> how many calls start in a minute
 token rate   -> how much input + output capacity those calls consume

A small classification request may consume one request and a few dozen tokens. A long-context coding request consumes the same one request but many more tokens. A client that controls only request count can still hit a token limit.

The 429 decision tree

Start with the response status and message, but do not treat every 429 as transient:

The official OpenAI documentation states that temporary rate limits return 429 and may include a Retry-After header. Treat that value as a minimum wait, then add a small random delay so many workers do not retry simultaneously. Unsuccessful requests still contribute to the per-minute limit, so sending the same request continuously is counterproductive.

An application should expose the reason and the next action in its metrics. A generic retry_count counter is not enough. Record whether the response was classified as temporary throttling, quota exhaustion, authentication failure, or a route error.

Measure Limits Before Changing Code

Before adding a queue or rotating keys, measure the traffic your application actually sends. Capture at least:

• Requests started and completed per minute
• Estimated input tokens and actual output tokens
• Concurrent in-flight requests
• 429 responses by endpoint and model
• Retry attempts and retry wait time
• 401, 404, 5xx, and timeout counts
• Successful answers after retry
• Cost per successful request

A simple dashboard can reveal the shape of the problem:

Do not infer a limit from one failed request. Run a controlled load test with synthetic data, increase concurrency gradually, and stop when the first sustained errors appear. Keep the test below an approved budget, and never use customer data to discover a limit.

Requests per minute and tokens per minute

RPM and TPM interact. If a worker sends 60 requests per minute and every request uses 20,000 input tokens, the token rate is already 1.2 million input tokens per minute before output tokens are counted. Conversely, very high RPM of tiny requests may be limited by request count first.

Use a conservative local budget rather than aiming at the provider ceiling:

allowed_rpm = provider_rpm * safety_factor
allowed_tpm = provider_tpm * safety_factor
safety_factor = 0.70 to 0.85 for a shared production project

The factor is a starting policy, not a provider rule. Leave room for retries, health checks, and traffic from other services that share the same project or key. If you cannot see a reliable provider limit, start lower and tune from observed success rates.

Retry 429 Responses Correctly

The official OpenAI SDK automatically retries eligible rate-limit errors and honors Retry-After when it is present. If you use the standard SDK, do not wrap every call in a second unbounded retry decorator. Double retries can multiply traffic and make latency unpredictable.

A manual retry loop is appropriate when you use a custom HTTP client, need a global retry budget, or want to coordinate retries across a worker pool. It should have four properties:

1. It retries only errors classified as transient.
2. It honors Retry-After when the value is valid.
3. It adds exponential backoff and jitter when the header is absent.
4. It stops after a maximum attempt count or total elapsed time.

Python with the OpenAI SDK

For a normal OpenAI Python SDK call, start by configuring the SDK built-in retry and timeout settings rather than adding another loop:

import os

from openai import OpenAI


client = OpenAI(
    api_key=os.environ["OPENAI_API_KEY"],
    max_retries=2,
    timeout=30.0,
)

response = client.chat.completions.create(
    model="your-verified-model-id",
    messages=[{"role": "user", "content": "Return one short sentence."}],
    max_tokens=80,
)
print(response.choices[0].message.content)

The model ID is intentionally a placeholder. Use the model list and account configuration for the endpoint you are calling. Set max_retries to a small, explicit value and measure the resulting latency and success rate.

The same client shape can point at an OpenAI-compatible gateway. For OurToken, the SDK base URL ends at /v1:

import os

from openai import OpenAI


client = OpenAI(
    api_key=os.environ["OURTOKEN_API_KEY"],
    base_url="https://api.ourtoken.ai/v1",
    max_retries=2,
    timeout=30.0,
)

Do not append /chat/completions to base_url when the SDK constructs that path for you. For current model IDs and route capabilities, use the live OurToken model catalog. A gateway may apply its own project, key, model, or concurrency limits, so monitor gateway responses rather than assuming that direct-provider limits transfer unchanged.

Manual HTTP backoff with Retry-After

If you use httpx, requests, or another HTTP client, parse Retry-After defensively. The header can be a number of seconds; if it is absent or invalid, use exponential backoff with jitter. The example below retries only 429 responses that are marked temporary by the caller:

import random
import time
from collections.abc import Callable

import httpx


def retry_after_seconds(response: httpx.Response) -> float | None:
    value = response.headers.get("Retry-After")
    if value is None:
        return None
    try:
        seconds = float(value)
    except ValueError:
        return None
    return max(0.0, seconds)


def post_with_backoff(
    send: Callable[[], httpx.Response],
    *,
    max_attempts: int = 5,
    max_elapsed_seconds: float = 90.0,
) -> httpx.Response:
    started = time.monotonic()
    delay = 1.0

    for attempt in range(1, max_attempts + 1):
        response = send()
        if response.status_code < 400:
            return response

        if response.status_code != 429:
            response.raise_for_status()

        retry_after = retry_after_seconds(response)
        wait_seconds = retry_after if retry_after is not None else delay
        wait_seconds += random.uniform(0.0, 0.25)

        if attempt == max_attempts:
            response.raise_for_status()
        if time.monotonic() - started + wait_seconds > max_elapsed_seconds:
            raise TimeoutError("retry budget exhausted")

        time.sleep(wait_seconds)
        delay = min(delay * 2.0, 60.0)

    raise RuntimeError("unreachable")

This function deliberately does not retry 401, 404, quota errors, or arbitrary 4xx responses. It also stops on a total elapsed-time budget, which is important for an interactive request. For a background job, use a queue deadline instead of letting a worker sleep indefinitely.

The official documentation also shows Tenacity, the backoff library, and a manual implementation as possible patterns. They are third-party or illustrative helpers, not guarantees from OpenAI. Whichever implementation you choose, make the retry policy visible in configuration and metrics.

Jitter and backoff math

A common fallback schedule is:

base delay: 1 second
attempt 1:  1 + jitter
attempt 2:  2 + jitter
attempt 3:  4 + jitter
attempt 4:  8 + jitter
cap each wait at 60 seconds

A full-jitter variant chooses a random value between zero and the exponential cap. Equal jitter waits for half the cap plus a random half. The exact variant matters less than avoiding synchronized retries and enforcing a cap. Do not increase the delay after a valid Retry-After header as if the header were an error; wait at least the header value, then add a small random amount.

Control Concurrency Before Rotating Keys

Key rotation is not a substitute for backpressure. If ten workers share one project and each rotates among five keys, the application can still exceed an organization-level limit, spend too quickly, or violate provider terms. First control the number of in-flight requests and the amount of work admitted into the queue.

A semaphore is a simple local guard:

import asyncio


MAX_IN_FLIGHT = 8
semaphore = asyncio.Semaphore(MAX_IN_FLIGHT)


async def bounded_call(make_request):
    async with semaphore:
        return await make_request()

A production limiter usually needs more than a semaphore:

If the API returns usage data, update your token estimate with actual input and output counts after each successful response. Before the call, estimate the worst case from prompt length and max_tokens; reserve that amount in the local budget, then release the unused portion after the response.

Prompt size is part of rate-limit handling

Reducing input tokens can be more effective than reducing requests. Common levers include:

• Remove repeated instructions from dynamically generated content
• Truncate irrelevant retrieval chunks
• Use a smaller max_tokens for classification and extraction
• Summarize conversation history before it grows without bound
• Cache stable prefixes when the selected route supports prompt caching
• Send large offline workloads through a batch workflow when available

These changes improve TPM headroom and often reduce cost and latency. Do not silently truncate content that changes the meaning of a legal, financial, or safety-critical request; log the transformation and test it.

Multi-Key and Multi-Model Routing

Once concurrency and queueing are controlled, you can add routing for availability, workload fit, or cost. The routing decision should be explicit and observable. Do not randomly switch models after every error; that makes behavior and debugging opaque.

A practical policy has three stages:

1. Choose a primary route by task type.
2. Retry transient throttling on that route within a small budget.
3. Use an approved fallback only when the error is safe to fail over.

Failover is usually reasonable for a temporary provider or route failure. It may be wrong for a quota or billing error shared by all routes, and it may change output quality or tool behavior. Record both the original route and the fallback route in the request trace.

Key rotation without accidental overload

If your provider permits multiple keys, keep a per-key limiter and a shared project limiter:

from dataclasses import dataclass


@dataclass
class KeyState:
    value: str
    in_flight: int = 0
    recent_requests: int = 0
    disabled_until: float = 0.0


class KeyPool:
    def __init__(self, keys: list[str], per_key_limit: int = 4):
        self.states = [KeyState(value=key) for key in keys]
        self.per_key_limit = per_key_limit

    def choose(self, now: float) -> KeyState:
        available = [
            state
            for state in self.states
            if state.disabled_until <= now
            and state.in_flight < self.per_key_limit
        ]
        if not available:
            raise RuntimeError("all API keys are busy or temporarily disabled")
        return min(available, key=lambda state: (state.in_flight, state.recent_requests))

This is only the selection layer. In production, protect state changes with an async lock or shared store, reset request counters on a defined window, and enforce a project-wide budget. Never put keys in logs, error messages, request metadata, or client-visible responses.

Route by workload, not by hope

A model router can classify work before the expensive call:

The OurToken LLM model routing guide covers the broader routing pattern. For rate-limit handling, route before the request starts and keep a bounded fallback after a transient failure. Do not use routing to hide a malformed model ID or invalid endpoint; fix those configuration errors first.

Observability and Alerting

A rate-limit policy is incomplete if you cannot tell whether it is working. Emit structured events for every attempt, but redact secrets and sensitive prompt content.

Recommended fields:

request_id
provider_or_gateway
model_id
endpoint
attempt_number
status_code
error_class
retry_after_seconds
backoff_seconds
queue_wait_ms
provider_latency_ms
input_tokens
output_tokens
fallback_used
final_status

Useful derived metrics include:

• rate_limit_rate = 429 responses / total attempts
• retry_success_rate = successful retries / retry attempts
• retry_amplification = total attempts / original requests
• queue_wait_p95
• cost_per_successful_request
• fallback_rate_by_model

Set alerts on sustained 429s, rising retry amplification, and a drop in retry success. A single 429 is often normal in a bursty system; a retry amplification of 3.0 means you are doing three attempts per original request on average and may be turning a small limit issue into a large cost issue.

Use a request ID that is safe to share with support, but do not log the API key or the complete authorization header. Store prompt and completion samples separately with the retention and access controls required by your application.

Test the policy with failure injection

Do not wait for a real incident to discover that the worker pool has no timeout. Add tests that simulate:

1. A 429 with Retry-After: 2.
2. A 429 without a valid Retry-After header.
3. A quota or billing message that must not retry.
4. A 401 and a 404.
5. A 503 followed by a successful response.
6. A request that exceeds the total retry deadline.
7. All keys being temporarily disabled.
8. A fallback model returning a different but valid schema.

Assert both behavior and accounting: number of attempts, total sleep time, final error class, selected route, and emitted metrics. Use a deterministic random source in unit tests so jitter does not make tests flaky.

OpenAI Direct API vs OpenAI-Compatible Gateways

An OpenAI-compatible API lets you reuse an OpenAI SDK or HTTP client while changing the base URL and model ID. That is useful when an application needs multiple model providers, a consolidated budget, or a gateway-level routing policy.

The compatibility layer does not mean every operational behavior is identical. Compare these properties before production rollout:

For OurToken, use https://api.ourtoken.ai/v1 as the OpenAI SDK base URL and consult the OurToken model catalog for current route IDs. Send a raw HTTP smoke test to the complete operation path only when testing HTTP directly. The live gateway response is the authority for what your key can access at runtime.

A gateway can provide a useful place to centralize budgets and fallback rules. Keep the client-side limiter anyway. Local backpressure protects your application from queue explosions and protects the provider from avoidable retry storms.

Production Checklist

Before shipping an OpenAI API rate-limit policy, verify each layer:

[ ] 429 responses are classified as temporary throttling or quota/billing action
[ ] Official SDK retries are not wrapped in an unbounded second retry loop
[ ] Custom HTTP clients honor Retry-After when valid
[ ] Exponential backoff includes jitter and a maximum attempt count
[ ] Total retry time has a deadline appropriate to the request SLA
[ ] 401, 404, model-not-found, and quota errors fail fast
[ ] Concurrency is bounded before keys are rotated
[ ] RPM and TPM budgets leave room for shared traffic and retries
[ ] Prompt and output sizes are measured and capped where safe
[ ] Fallback models are approved and tested for schema/tool compatibility
[ ] Secrets never appear in logs or request metadata
[ ] Rate-limit, retry, queue, latency, and cost metrics are emitted
[ ] Failure-injection tests cover headers, quota errors, and deadlines
[ ] Live model IDs and endpoint behavior are checked before deployment

Treat this checklist as a release gate, not a troubleshooting note. A retry loop that works in a notebook can still overload a shared production project when 50 workers start together.

Conclusion and FAQ

Handling an OpenAI API rate limit is a control-system problem. Classify the error, honor Retry-After, use bounded exponential backoff with jitter, control concurrency, account for both RPM and TPM, and fail fast on quota, authentication, and configuration errors. Then add routing only after the basic request path is correct.

The OpenAI SDK already retries eligible temporary rate-limit errors, so configure a small retry count instead of adding a second blind loop. If you use a custom HTTP client, implement a deadline-aware policy and test it with injected 429 responses. When using OurToken or another OpenAI-compatible gateway, keep the SDK base URL at the API root, verify exact model IDs from the live catalog, and measure gateway-specific limits rather than assuming direct-provider behavior.

FAQ

What does an OpenAI API 429 error mean?

It means the request was rejected with HTTP 429. The cause may be temporary RPM or TPM pressure, or it may be an account quota or billing issue. Inspect the response and classify the cause before retrying.

How long should I wait after a 429?

If the response contains a valid Retry-After value, wait at least that long and add a small random delay. If the header is absent or invalid, use capped exponential backoff with jitter and a total retry deadline.

Should I retry every 429 response?

No. Do not retry quota, billing, or other errors that require account action. Retry temporary throttling only within a bounded attempt and time budget.

Does the OpenAI SDK retry rate-limit errors automatically?

The official OpenAI SDK automatically retries eligible rate-limit errors and honors Retry-After when present. Check your SDK version and configure its retry count explicitly. Avoid adding an unbounded application-level loop around it.

What are requests per minute and tokens per minute?

Requests per minute limits how many calls start in a time window. Tokens per minute limits the input and output token throughput. A client can stay below RPM and still exceed TPM with long prompts.

Will rotating API keys remove a rate limit?

Not necessarily. Limits may apply at the project, organization, model, or gateway level. Rotate keys only when your provider permits it, and keep a shared limiter so rotation does not create a larger burst.

How should I use model fallback after a rate limit?

Retry the primary route briefly for transient throttling, then use an approved fallback if the task can tolerate a different model. Record the original and fallback model IDs, and do not use fallback to hide a bad endpoint or invalid model ID.

What base URL should I use with OurToken and the OpenAI SDK?

Use https://api.ourtoken.ai/v1 as the SDK base URL. For current model IDs, limits, and route behavior, check the live OurToken model catalog. Do not append /chat/completions when the SDK adds the operation path.

How do I test rate-limit handling safely?

Inject synthetic 429, 401, 404, 5xx, timeout, and quota responses in a test client. Assert retry count, wait budget, final classification, fallback behavior, and emitted metrics without sending a high-volume live load.