OpenAI-Compatible Prompt Caching: Reduce LLM Costs Across GPT and Claude

If you are evaluating openai compatible prompt caching, you are probably already past the “what is an API” stage. You have real traffic, long prompts, tool definitions, retrieval context, or agent memory that repeats across requests. At that point, the problem is no longer model access. The problem is paying to recompute the same prompt prefix over and over again.

That is exactly where prompt caching matters. OpenAI’s official guide says Prompt Caching can reduce latency by up to 80% and input token cost by up to 90% for supported workloads. Anthropic’s prompt caching docs describe the same core goal from a different interface direction: reuse stable prompt prefixes so large system prompts, tools, and long conversations are not reprocessed from scratch every time.

The catch is that “OpenAI-compatible” does not mean “cache-compatible.” A unified client can make requests look the same, but caching behavior still depends on the provider underneath. OpenAI enables prompt caching automatically. Anthropic exposes both automatic and explicit cache controls. Some OpenAI SDK compatibility layers smooth over request syntax, but they do not erase the provider-level differences that determine whether you actually get cache hits.

This article explains what OpenAI-compatible prompt caching should mean in practice, how OpenAI and Anthropic differ, what architecture keeps your app portable, and how to avoid the easy mistakes that erase the savings you expected.

Why OpenAI-Compatible Prompt Caching Matters

Most multi-model teams solve the first problem before they solve the second.

The first problem is access: one SDK, one auth pattern, one request shape, many providers. That is the value of an OpenAI-compatible layer such as the integration patterns described in OurToken Docs.

The second problem is much more expensive: repeated prompt prefixes. In production, the same request family often shares a large stable prefix:

• a 1,500-token system prompt
• tool definitions for search, CRM, billing, and retrieval
• few-shot examples
• policy instructions
• long retrieval context that changes slowly

If that prefix is recomputed on every request, your bill grows faster than your product complexity.

The Hidden Tax in “Unified” AI Infrastructure

A lot of teams assume that if they have a unified endpoint, optimization is already handled. It is not. A unified endpoint standardizes transport. It does not guarantee that repeated content is placed correctly for caching, that provider-specific cache controls are enabled, or that routing decisions keep similar requests landing where cache reuse is likely.

This is why prompt caching becomes especially important in three workloads:

1. RAG pipelines with large, repeated instructions and tool definitions
2. Agents that reuse the same tool stack across many turns
3. Support and operations workflows where requests differ at the tail, not the prefix

In each case, prompt caching is not a micro-optimization. It changes the economics of whether your app can scale without routing every request to the cheapest possible model.

What “Compatible” Should Mean

For engineering teams, a good OpenAI-compatible prompt caching design should mean:

• one application-facing interface
• provider-aware caching controls under the hood
• telemetry that shows cache hits and misses by provider
• prompt structure designed for shared prefixes
• the ability to switch models without rewriting feature code

That is a stronger standard than “the SDK call did not error.”

How OpenAI Prompt Caching Actually Works

OpenAI’s current prompt caching model is simple from the application side. It works automatically on API requests with large enough repeated prefixes, and the docs say no code changes are required to get baseline caching behavior.

Automatic by Default, but Not Magic

According to OpenAI’s official guide, Prompt Caching is enabled automatically for prompts that are 1024 tokens or longer. Cache hits require exact prefix matches. Static content should go first. Variable content should go last.

That sounds straightforward, but it has real architectural implications:

• your system prompt should stay stable
• tool definitions should not be regenerated unnecessarily
• per-user fields should be pushed later in the request
• retrieval context should be organized so the reusable part stays early

If your prompt starts with a timestamp, request ID, or constantly changing tool description, you have already made caching harder than it needs to be.

The Two OpenAI Details That Matter Most

Two details from the OpenAI docs matter far more than most teams realize.

First, OpenAI says exact prefix matching drives cache hits. That means semantic similarity is irrelevant. “Almost the same prompt” does not help.

Second, OpenAI documents an optional prompt_cache_key parameter that can improve routing for traffic sharing common prefixes. This matters because cache effectiveness is not only about prompt design. It is also about whether similar requests are likely to land where prior work can be reused.

Here is a minimal example using the Responses API:

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
  baseURL: process.env.OPENAI_BASE_URL
});

const response = await client.responses.create({
  model: "gpt-5.5",
  instructions:
    "You are a support triage assistant. Follow billing policy. Return concise answers.",
  input: [
    {
      role: "user",
      content: "Customer says invoice failed after card update."
    }
  ],
  prompt_cache_key: "support-triage-v3"
});

console.log(response.output_text);

The important idea is not the syntax. It is the prefix discipline. A shared prompt_cache_key helps only if the requests also share a real repeated prefix.

Retention Policy Changes the Strategy

OpenAI’s docs also distinguish between in-memory retention and extended prompt cache retention. In-memory retention generally keeps cached prefixes active for several minutes of inactivity, up to one hour. Extended retention is available on a specific set of newer models and can keep cached prefixes active for longer, up to 24 hours.

That means your architecture should not assume one universal cache lifetime. A bursty support queue and a once-per-hour document job may need different expectations. If your workload depends on long-lived reuse windows, model selection and cache retention policy become part of the design, not an afterthought. For OpenAI, set prompt_cache_retention: "24h" only on models and routes where extended retention is supported; otherwise omit the field and let the default in-memory behavior apply.

Why Anthropic Changes the Design

Anthropic also supports prompt caching, but the operating model is different enough that you should not hide it behind a fake uniform abstraction.

Anthropic Gives You More Explicit Control

Anthropic’s prompt caching docs support two approaches:

• automatic caching through a top-level cache_control
• explicit cache breakpoints placed on individual content blocks

That is a more visible model than OpenAI’s mostly automatic behavior. Anthropic also documents that cache prefixes cover tools, then system, then messages, in that order. The docs say the default cache lifetime is 5 minutes, with a 1-hour option available at additional cost.

This makes Anthropic especially powerful for long, structured prompts where you know exactly which part should stay stable. For example, if your tool definitions and system prompt remain fixed while user content changes every request, explicit breakpoints let you mark the last stable block rather than hoping a generic interface guesses correctly.

Here is the basic shape:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-8",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are a support operations assistant.",
    messages=[
        {"role": "user", "content": "Summarize recurring billing issues this week."}
    ],
)

print(response.usage)

If your production workload depends on long reusable prefixes, this explicit model can be easier to tune than a purely implicit one. That is one reason teams evaluating provider-specific behavior often compare the Claude path against a concrete model page such as Claude Opus 4.8 instead of treating all “compatible” backends as identical.

Compatibility Is Not Feature Parity

Anthropic’s OpenAI SDK compatibility docs are especially important here. They make clear that compatibility helps with request syntax, but not every OpenAI-specific setting maps cleanly. That matters because many teams hear “OpenAI-compatible” and infer “OpenAI semantics.”

That inference is wrong in caching.

The safe assumption is:

• compatibility may simplify client code
• compatibility does not erase provider-native cache behavior
• provider-native controls usually deserve a dedicated adapter layer

If you hide those differences too aggressively, you will ship a beautifully consistent SDK surface that quietly leaves money on the table.

The Anthropic Failure Mode Most Teams Miss

Anthropic documents a common mistake that is useful beyond Anthropic itself: setting the cache breakpoint on a block that changes every request. If the changing content is inside the cached prefix, the hash changes and the cache read disappears. You pay for cache writes without getting cache hits.

This is the key lesson for any provider: do not cache where the request ends. Cache where the stable part ends.

The Right Architecture for OpenAI-Compatible Prompt Caching

If you want one app-level interface and real savings underneath, the architecture should be provider-aware, not provider-blind.

A Good Adapter Has Two Jobs

Your prompt caching adapter should do two separate jobs:

1. Normalize the application contract
2. Specialize the provider behavior

The application contract is simple:

• here is the model
• here is the stable prefix
• here is the variable suffix
• here is the workload key
• here is whether this route is latency-sensitive or cost-sensitive

The provider specialization is where the real work happens:

• OpenAI path: stable prefix ordering, prompt_cache_key, retention choice
• Anthropic path: automatic or explicit cache_control, breakpoint placement, TTL choice

That separation keeps your feature code clean while preserving the controls that actually produce cache hits.

A Minimal Provider-Aware Adapter

type CacheablePrompt = {
  system: string;
  tools?: Array<Record<string, unknown>>;
  sharedContext?: string;
  userInput: string;
  workloadKey: string;
};

type Provider = "openai" | "anthropic";

const EXTENDED_RETENTION_MODELS = new Set(
  (process.env.OPENAI_EXTENDED_CACHE_MODELS ?? "")
    .split(",")
    .map(model => model.trim())
    .filter(Boolean)
);

function supportsExtendedRetention(model: string) {
  return EXTENDED_RETENTION_MODELS.has(model);
}

export async function generateWithCache(
  provider: Provider,
  model: string,
  prompt: CacheablePrompt
) {
  if (provider === "openai") {
    const request = {
      model,
      instructions: prompt.system,
      tools: prompt.tools,
      input: [
        {
          role: "user",
          content: `${prompt.sharedContext ?? ""}\n${prompt.userInput}`.trim()
        }
      ],
      prompt_cache_key: prompt.workloadKey,
      ...(supportsExtendedRetention(model)
        ? { prompt_cache_retention: "24h" }
        : {})
    };

    return openai.responses.create(request);
  }

  return anthropic.messages.create({
    model,
    max_tokens: 1024,
    cache_control: { type: "ephemeral" },
    system: prompt.system,
    tools: prompt.tools,
    messages: [
      {
        role: "user",
        content: [
          {
            type: "text",
            text: prompt.sharedContext ?? "",
            cache_control: { type: "ephemeral" }
          },
          {
            type: "text",
            text: prompt.userInput
          }
        ]
      }
    ]
  });
}

The exact SDK fields will evolve. The design principle will not. Your app should describe stable and unstable prompt sections once, and your adapter should translate that into the right provider-native caching behavior. In this example, extended OpenAI retention is controlled by an allow-list instead of being sent blindly on every request. That keeps the sample honest for mixed model fleets where not every route or model supports the same cache lifetime.

Where a Unified Endpoint Still Helps

None of this means OpenAI-compatible infrastructure is unhelpful. It helps a lot. A unified model catalog such as OurToken Models still reduces integration complexity, auth sprawl, and provider switching cost.

But the right way to use that compatibility layer is as a control plane, not as an excuse to pretend provider-specific runtime behavior does not exist.

That distinction becomes even more valuable when some routes do not benefit much from caching at all. If a workload has short prompts and low prefix reuse, a cheaper model such as GLM 5.2 may matter more than perfect caching strategy. Compatibility gives you the freedom to make that decision route by route.

OpenAI-Compatible Prompt Caching for RAG and Agents

This is where the keyword becomes truly practical. The biggest winners from prompt caching are not toy chat apps. They are systems with repeated structure and expensive prefixes.

RAG Pipelines

In a RAG pipeline, the stable prefix often includes:

• response policy
• citation rules
• retrieval instructions
• formatting rules
• tool definitions for retrieval or reranking

The variable suffix usually includes:

• the user question
• a fresh set of retrieved chunks
• per-request metadata

The design goal is to keep the reusable policy and tool layers early and stable. If every retrieval instruction block is rebuilt in a slightly different order, cache hits disappear even though the product behavior looks unchanged.

This is one reason prompt caching pairs naturally with a standardized gateway. If your RAG routes already use the same OpenAI-compatible infrastructure described at OurToken, then cache-aware prompt construction becomes a central optimization rather than a per-team habit.

Agents and Tool Stacks

Agents benefit even more because their prompts often include:

• large system instructions
• many tool schemas
• tool usage policy
• memory summaries
• safety rules

These are exactly the kinds of prompt prefixes that should be cached. But agents also create a trap: if your memory compaction or tool serialization changes every turn, the prefix stops being stable.

For agents, the best rule is simple:

• cache static tool definitions
• cache stable policy and role instructions
• keep dynamic memory summaries late unless they are reused broadly
• measure cache-hit rate separately from total token usage

If you only track total cost, you will not know whether a prompt refactor improved reasoning or silently destroyed your cache efficiency.

Cost Analysis: Where the Savings Actually Come From

Prompt caching is often pitched as a token discount. That is true, but incomplete.

The Direct Savings

OpenAI’s official guide says prompt caching can reduce latency by up to 80% and input token cost by up to 90%. Anthropic publishes a more explicit pricing model: 5-minute cache writes cost 1.25x base input price, 1-hour writes cost 2x, and cache reads cost 0.1x of base input price.

Here is the simple break-even math for the cached prefix. With Anthropic’s 5-minute cache, one write plus one read costs about 1.25x + 0.1x = 1.35x, compared with 2x if the same prefix is processed twice without caching. With the 1-hour cache, one write plus two reads costs about 2x + 0.2x = 2.2x, compared with 3x without caching. In plain terms: the short cache can pay off after one reuse, while the longer cache needs more reuse but is better for routes with wider spacing between requests.

The practical lesson is that caching pays off when:

• the stable prefix is large
• the prefix is reused frequently
• the reuse window fits the provider’s retention behavior

If any of those fail, the math weakens quickly.

The Bigger Savings Most Teams Ignore

The more durable savings usually come from architecture:

That fourth row matters. The real cost of multi-model AI is not only tokens. It is operational complexity. A good caching layer reduces repeated inference work and reduces the amount of special-case prompt plumbing each team has to maintain.

When Caching Is Not the Main Lever

There are cases where prompt caching is not the first optimization to reach for:

• very short prompts
• low request repetition
• jobs spaced too far apart for the retention window
• workloads whose variable content dominates the prompt

In those cases, model routing, cheaper models, batching, or better retrieval compression may matter more.

This is why the best multi-model stacks do not treat caching as a universal answer. They treat it as one route-level optimization among several. The product advantage of a unified layer is that you can combine caching, routing, and provider selection without rewriting application code every time.

The Mistakes That Kill Cache Hit Rates

Most failed prompt caching rollouts do not fail because the provider is bad. They fail because the prompt structure is sloppy.

Mistake 1: Putting Changing Content Too Early

If your first blocks include timestamps, per-user IDs, or changing memory summaries, you have destroyed the exact prefix match before the request really begins.

Mistake 2: Re-serializing Tools Differently Each Request

Tool definitions are part of the cacheable prefix for both OpenAI-style and Anthropic-style designs. If property order, optional descriptions, or function metadata drift between calls, the reusable prefix is no longer reusable.

Mistake 3: Treating Compatible Syntax as Compatible Semantics

This is the biggest product-level mistake. A request that succeeds through an OpenAI-compatible SDK is not proof that your provider is delivering the same cache behavior you designed for on another backend.

Mistake 4: Tracking Total Cost Without Tracking Cache Performance

You should log at least:

1. cache hit rate by route
2. average shared-prefix size
3. cost per request before and after caching changes
4. latency before and after caching changes

If you only watch aggregate spend, you will miss regressions caused by subtle prompt edits.

Cache Hit Debugging Checklist

When cache-hit rate drops after a prompt or routing change, check these before blaming the provider:

• Is the reusable prefix byte-for-byte identical across requests?
• Are timestamps, user IDs, request IDs, or memory summaries appearing before the stable prefix ends?
• Are tool schemas serialized in a deterministic order?
• Are cache metrics split by route, provider, model, and workload key?
• Did a model switch move traffic to a provider with different cache controls or retention behavior?

Conclusion

OpenAI-compatible prompt caching is worth doing, but only if you define it correctly. It does not mean pretending every provider caches the same way. It means keeping one application-facing interface while preserving the provider-specific controls that generate real cache hits underneath.

OpenAI gives you a mostly automatic path, with exact prefix matching, optional prompt_cache_key, and retention choices that matter for bursty versus long-lived workloads. Anthropic gives you more explicit control through cache_control, block-level breakpoints, and documented prefix ordering. Both are useful. Neither should be flattened into a fake universal behavior model.

The winning architecture is simple: separate stable prefix from variable suffix, keep provider-specific cache logic in the adapter layer, measure hit rates by route, and only optimize where reuse is real. For teams building on a unified API surface, that approach preserves the portability benefits of compatibility without losing the hard savings that prompt caching can unlock.

FAQ

Does OpenAI-compatible prompt caching mean every provider supports the same cache behavior?

No. Compatibility usually standardizes request syntax, not provider-native cache semantics. You still need provider-aware logic if you want predictable cache savings.

Is OpenAI prompt caching automatic?

Yes, according to OpenAI’s official prompt caching guide, baseline prompt caching works automatically on eligible requests with repeated prefixes. You can further influence routing with prompt_cache_key and choose retention policies on supported models.

Is Anthropic prompt caching automatic or manual?

It can be both. Anthropic supports automatic caching with top-level cache_control and explicit cache breakpoints for finer control.

What workloads benefit most from prompt caching?

RAG pipelines, tool-heavy agents, support automation, and any workflow with large repeated prefixes and frequent request reuse.

Should I always optimize caching before model routing?

No. If the workload has short prompts or weak prefix reuse, model routing or cheaper models may matter more than caching. The right stack uses both where they fit.