OpenAI API Migration Guide for Multi-Model Endpoints

An openai api migration guide should begin with a distinction that is easy to miss: moving from Chat Completions to the Responses API is one migration, while moving from a direct OpenAI integration to an OpenAI-compatible multi-model endpoint is another. They may happen in the same quarter, but they change different parts of your system and should not be bundled into one risky release.

This guide covers the second path. It shows how to keep an application's core request contract stable while moving the provider boundary behind one configured OpenAI-compatible endpoint. The result is not a promise that all models behave identically. It is a controlled way to retain an OpenAI SDK-shaped client while making model choice, fallback, cost policy, and provider routing explicit.

For a team with a working OpenAI integration, the goal is modest: change one integration boundary, test the new route against real workloads, send a small percentage of traffic through it, and keep a one-control rollback. That is safer and more useful than a big-bang rewrite.

Scope note: This article is about an OpenAI-compatible API migration, not a Chat Completions-to-Responses migration. If your immediate task is migrating OpenAI API surfaces, read the separate Responses API migration guide and OpenAI's official migration documentation.

Define the Migration Target Before Changing the Endpoint

The phrase "migrate from OpenAI" can describe several very different projects. If the team does not name the target precisely, a base-URL configuration change turns into an accidental rewrite of prompts, tools, state management, and model policy.

This article assumes the second and third rows. Your application already creates completions or responses through the OpenAI SDK. You want a provider boundary that can use the live models available to your account, rather than hard-coding a single upstream provider in feature code.

A Compatible Endpoint Is an Interface, Not a Behavioral Guarantee

OpenAI compatibility is valuable because it preserves familiar authentication and request patterns. It does not mean every endpoint, parameter, streaming event, tool-call field, rate limit, or model capability is identical across routes. A safe migration starts from the portable subset your application genuinely needs.

For each workload, write down these requirements before you route any production traffic:

• Required API surface: Chat Completions, Responses, embeddings, or another endpoint.
• Required response shape: text, JSON Schema output, streaming, or tool calls.
• Maximum acceptable latency and error rate.
• Required context length and multilingual behavior.
• Whether a fallback is allowed to return a different model's answer.
• Which usage fields you need for billing and product analytics.

Treat anything not in this list as optional during the first rollout. This protects the migration from becoming a catch-all platform project.

Keep the Existing OpenAI Surface Migration Separate

OpenAI's current migration guidance covers moving application logic from older surfaces to the Responses API. That work can be worthwhile for tools, durable state, and newer API capabilities, but it has a separate response contract and test plan.

Do not change all three of these in one release:

API surface:          Chat Completions -> Responses
Provider boundary:    direct OpenAI -> compatible endpoint
Model policy:         one fixed model -> routed model set

Change one dimension, verify it, then change the next. If a regression occurs, this gives your on-call engineer a concrete explanation and an easy rollback path.

Put a Stable Model Adapter at the Application Boundary

The most important code change is usually not the base_url. It is putting all model calls behind a small internal adapter. Controllers, jobs, and agent loops should ask for a normalized generation result. Only the adapter should know which SDK, endpoint, API key, model ID, and route policy are active.

Application route / worker
          |
          v
    Model adapter -----------------------> request metrics and evaluation log
          |
          +--> primary compatible route
          |          |
          |          v
          |      selected model
          |
          +--> approved fallback route
                     |
                     v
               rollback-safe response

This design gives you one place to add a compatible endpoint, one place to record usage, and one place to make provider-specific behavior visible. It also stops model IDs from leaking into every feature file, which is the usual reason a small migration becomes expensive.

Normalize the Result Your Product Actually Uses

Most product code does not need a full SDK response object. It needs text, a request identifier, token usage, selected model, and an indication of whether a fallback was used. Define that contract once.

from dataclasses import dataclass


@dataclass
class GenerationResult:
    text: str
    request_id: str
    model: str
    input_tokens: int | None
    output_tokens: int | None
    fallback_used: bool = False

Do not return a raw ChatCompletion object from the adapter if the rest of the application only reads choices[0].message.content. That forces every caller to understand provider response details and makes a future API-surface change much harder.

The adapter can be deliberately small at first:

import os

from openai import OpenAI


def make_client() -> OpenAI:
    return OpenAI(
        api_key=os.environ["OURTOKEN_API_KEY"],
        base_url="https://api.ourtoken.ai/v1",
        timeout=30.0,
        max_retries=2,
    )


def generate_text(*, model: str, system: str, user: str) -> GenerationResult:
    completion = make_client().chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user},
        ],
        temperature=0.2,
    )

    usage = completion.usage
    return GenerationResult(
        text=completion.choices[0].message.content or "",
        request_id=completion.id,
        model=completion.model,
        input_tokens=getattr(usage, "prompt_tokens", None),
        output_tokens=getattr(usage, "completion_tokens", None),
    )

The example uses the OpenAI SDK's Chat Completions shape because it is a clear compatibility baseline. Keep the base URL at the API root; do not append /chat/completions when the SDK builds the operation path itself. Before deploying, verify model IDs and route support against the live OurToken model catalog, not a model display name copied from a marketing page.

Make Configuration Explicit and Secret-Safe

The migration should change environment configuration, not place a new key in source code or in a frontend bundle. Keep credentials out of logs and accept model IDs through trusted server-side configuration.

OURTOKEN_API_KEY=stored-in-secret-manager
LLM_PRIMARY_MODEL=verified-model-id
LLM_FALLBACK_MODEL=verified-fallback-model-id
LLM_ROLLOUT_PERCENT=5

LLM_ROLLOUT_PERCENT is intentionally not a provider setting. It is your application's release control. Keeping it separate means you can stop new traffic from using the migrated path without modifying an upstream configuration or shipping an emergency build.

For a broader explanation of why this boundary matters, see what an OpenAI-compatible API changes for application architecture. The practical rule is simpler: one code path owns transport configuration; product code owns product behavior.

OpenAI API Migration Guide: Migrate in Four Releases, Not One Cutover

A migration is ready when it can be measured and reversed. The following four releases are intentionally boring. They avoid the two common failures: changing too much at once and declaring success because a single manual prompt worked.

1. Inventory Calls and Create Contract Tests

Start by listing every model call, then group them by behavior rather than by repository folder. A support-ticket classifier, a document summarizer, and an autonomous coding agent have radically different migration risk.

Build a fixed evaluation set for each class. Include ordinary examples, multilingual inputs if your product serves them, malformed inputs, long prompts, and known edge cases. Record the expected schema or grading rubric before changing the endpoint.

For structured outputs, test the exact schema your downstream systems consume. A response that looks plausible in a console but changes an enum, omits a field, or emits a tool call differently is still a production regression. Our Structured Outputs JSON Schema guide is useful when you need a schema contract before starting the migration.

2. Send Shadow Traffic Without Affecting Users

For medium- and high-risk routes, keep the direct OpenAI result as the user-facing response while sending a sampled copy of the request to the candidate compatible route in the background. Compare results asynchronously. Do not double-send requests that mutate external systems or trigger paid side effects.

import hashlib


def should_shadow(request_id: str) -> bool:
    # Stable sampling keeps retries on the same treatment.
    digest = hashlib.sha256(request_id.encode("utf-8")).hexdigest()
    return int(digest[:8], 16) % 100 < 5


def handle_summary(request_id: str, ticket: str) -> GenerationResult:
    primary = generate_with_current_production_path(ticket)

    if should_shadow(request_id):
        candidate = generate_with_compatible_path(ticket)
        write_evaluation_record(
            request_id=request_id,
            current=primary,
            candidate=candidate,
        )

    return primary

The comparison should include more than string similarity. Record schema validity, safety outcomes, tool-call sequence where relevant, latency, token usage, and a task-specific quality score. For subjective generation, use blinded human review on a representative sample rather than declaring a different writing style a failure by default.

3. Canary by Workload, Then by Traffic Percentage

After shadow traffic meets the contract, switch a low-risk workload to the compatible route for a small, stable percentage of requests. Route a given tenant or request hash consistently so one user does not receive alternating provider behavior in the same workflow.

Increase the rollout only after each threshold is met:

5% traffic:  verify authentication, route configuration, and basic error rate
25% traffic: verify p95 latency, schema validity, and token accounting
50% traffic: verify operational dashboards and fallback behavior
100% traffic: keep the old path available for the defined rollback window

Avoid switching global traffic merely because the new path is less expensive. A lower price cannot compensate for invalid tool arguments, missing fields, or a regression in a customer-critical language.

4. Keep Rollback a Configuration Change

Your rollback should answer three questions in less than a minute:

1. Which routes are currently migrated?
2. Can new requests return to the old route without a deployment?
3. Where are the logs needed to compare the two paths?

A minimal feature-flag policy can be enough:

def select_path(rollout_percent: int, request_bucket: int) -> str:
    return "compatible" if request_bucket < rollout_percent else "direct"

Keep the old route functional until the agreed observation window ends. Do not remove it on the same day that you reach 100% traffic; a rollback path that has never been exercised is mostly a story you tell yourself.

Add Multi-Model Routing Only After the Core Route Is Stable

Once the compatible endpoint is reliable, you can introduce controlled model selection. This is where a unified endpoint becomes commercially useful, but it must follow the same principle as the transport migration: define a policy, test it per workload, and log every decision.

Route by Task Requirements, Not by Brand Preference

Start with an explicit task policy. Exact available model IDs, context limits, prices, and feature support can change, so the live catalog should remain the source of truth.

For example, a team might evaluate GPT-5.6 Terra for complex reasoning, DeepSeek V4 Pro for selected code or analytical workloads, and GLM 5.2 for workloads where it meets the product's language and quality requirements. This is an evaluation plan, not a claim that one model is universally best. Verify the model IDs exposed to your account before using any route in production.

The LLM model routing guide explains the decision layer in more depth. In a migration, the important addition is traceability: every response should identify the selected model, whether fallback ran, and why the policy chose that route.

Fallback Is a Recovery Policy, Not a Random Retry

Only fail over when the error and workload make it safe. A temporary upstream 5xx or a bounded rate-limit condition may justify trying an approved backup route. A 401, malformed request, wrong base URL, or invalid model ID should fail fast because a fallback only conceals the real configuration problem.

from openai import APIStatusError


TRANSIENT_STATUS_CODES = {429, 500, 502, 503, 504}


def generate_with_fallback(request, primary_model: str, fallback_model: str):
    try:
        return generate_with_model(request, primary_model)
    except APIStatusError as error:
        if error.status_code not in TRANSIENT_STATUS_CODES:
            raise

        result = generate_with_model(request, fallback_model)
        result.fallback_used = True
        return result

In production, add a small retry budget on the primary path before fallback, idempotency protection for workflows with side effects, and a per-model circuit breaker. The OpenAI API rate-limit guide covers why unlimited retries make rate-limit incidents worse instead of better.

Make Capability Differences Visible

Keep a compact capability matrix alongside your route configuration. This prevents a team from selecting a cheap fallback for a request that requires streaming, strict structured output, image input, or a particular tool behavior.

route                  streaming  json_schema  tool_calls  long_context  approved
primary-support        yes        yes          yes         tested        yes
fallback-classifier    no         yes          no          limited       yes
experimental-route     unknown    unknown      unknown     unknown       no

An "unknown" capability should not be selected by automatic fallback. Turn unknown into tested or unavailable before it reaches production policy.

Measure Cost, Quality, and Reversibility Together

The point of an API migration is not merely to see a different endpoint return 200. It is to retain or improve the product's quality while gaining enough routing and cost control to justify the operational work. Measure three dimensions together.

At the task level, the simple cost calculation is:

cost per successful task =
  (input token cost + output token cost + retry cost)
  / successful completed tasks

Use successful task count as the denominator, not raw request count. An inexpensive model that causes schema failures, human rework, or repeated retries is often more expensive than it first appears.

Log enough metadata to explain an incident without leaking prompts or credentials:

request_id
route_id
model_id
endpoint
release_flag
attempt_count
fallback_used
status_code
input_tokens
output_tokens
latency_ms
schema_valid
evaluation_score

Keep prompt content and API keys out of these general operational logs. Store any sensitive trace data separately with the data-retention and access controls appropriate for your product.

The Migration Checklist

Before treating the compatible path as the default, verify the following:

[ ] Every model call goes through a small adapter or client boundary.
[ ] API keys are server-side secrets, never source code or browser configuration.
[ ] The SDK base URL ends at the API root, and the SDK owns operation paths.
[ ] Model IDs were copied from the live provider catalog and tested by route.
[ ] Contract tests cover text, JSON, streaming, tools, or long context as applicable.
[ ] Shadow comparisons are stored for representative, non-mutating workloads.
[ ] Canary traffic is stable by tenant or request hash.
[ ] Fallback is restricted to approved models and transient failures.
[ ] 401, 404, malformed requests, and model-not-found errors fail fast.
[ ] Metrics distinguish primary success, retry success, fallback success, and failure.
[ ] A configuration-only rollback has been exercised.
[ ] Cost reports use completed tasks and include retry cost.

Conclusion

An OpenAI-compatible API migration should be a boundary change, not a leap of faith. Keep application code on a normalized adapter, make the endpoint and model policy explicit, validate model capabilities against the workloads that matter, and move traffic gradually with a real rollback control.

This approach lets a team begin with one tested compatible route, then adopt deliberate model routing as requirements change. It also keeps the separate question of a Responses API migration where it belongs: in its own scoped project with its own contract tests. By separating transport, API surface, and model policy, you get a system that can evolve without forcing every feature team to rewrite its AI integration.

FAQ

Is an OpenAI-compatible API migration just changing base_url?

Changing base_url is often the first configuration step, but it is not the whole migration. You also need to verify model IDs, endpoint support, response fields, rate-limit behavior, streaming and tool capabilities, observability, and a rollback path. Keep the initial code change small, but treat operational verification as part of the migration.

Should I migrate to the Responses API and a compatible endpoint at the same time?

Usually no. A Responses API migration changes request and response semantics; a compatible endpoint migration changes the provider boundary and model policy. Migrating one dimension at a time makes test failures understandable and rollback practical.

How do I choose a model after moving to a multi-model endpoint?

Choose by evaluated task requirements: output schema reliability, tool behavior, context needs, latency, language coverage, and cost per successful task. Start with a small approved route set. Use the current model catalog for exact IDs and availability rather than copying an old example.

When should a request use fallback?

Use an approved fallback for temporary, safe-to-retry failures such as a bounded rate-limit event or selected upstream 5xx errors. Do not use fallback for authentication, malformed requests, wrong endpoints, or invalid model IDs. Those require configuration fixes.

How can I test the new route without changing user responses?

Use sampled shadow traffic for non-mutating workloads. Return the current production result to the user, send the same request to the candidate route in the background, then compare schema validity, quality, latency, tokens, and error behavior offline.

What should I measure to prove the migration was successful?

Measure quality, reliability, and cost together: task-specific evaluation score, schema-validity rate, latency, 429/5xx and fallback rates, token usage, retry cost, and cost per successfully completed task. A lower price is not a win if it creates retries or manual recovery work.