OpenAI Batch API: Cut Async LLM Costs 50% with Python and JSONL

The OpenAI Batch API is the practical answer to a common production problem: you need to run thousands of LLM requests, but users do not need the answers in real time. Instead of hammering the synchronous chat endpoint, you package requests into a JSONL file, upload that file, create a batch job, and retrieve the results when the job completes. OpenAI prices these asynchronous requests at 50% less than the equivalent synchronous API calls, gives batches a separate pool of significantly higher rate limits, and targets a 24-hour completion window.

That makes the Batch API a good fit for evaluation runs, classification, extraction, summarization, moderation, embedding repositories, and other offline workloads. It is a poor fit for interactive chat, tool-heavy agents that need immediate feedback, or any request where the user is actively waiting for the next token.

This guide walks through the complete workflow: how to structure the JSONL file, upload it with Python, create and monitor a batch, download results, recover from failed requests, and decide when to use Batch instead of prompt caching or model routing.

Verification status: The discount, completion window, endpoints, file format, statuses, and code patterns were checked against the official OpenAI Batch API guide on 2026-08-28. OpenAI's guide notes that markdown versions of documentation pages are available by appending .md to the page URL.

The Short Version

A batch job is five steps:

1. Build requests.jsonl
2. Upload the file with purpose="batch"
3. Create a batch for a supported endpoint
4. Poll the batch status
5. Download output_file_id and error_file_id

The request body is a normal API request body wrapped in four batch fields:

{
  "custom_id": "review-1001",
  "method": "POST",
  "url": "/v1/chat/completions",
  "body": {
    "model": "gpt-5.4-mini",
    "messages": [
      {"role": "system", "content": "Classify the sentiment as positive, neutral, or negative."},
      {"role": "user", "content": "The product arrived early and works perfectly."}
    ]
  }
}

Every line in the file must follow that shape. The custom_id is the value you use to join the result back to your source record after the batch completes.

What the OpenAI Batch API Is Good At

Batch processing works when latency is acceptable and throughput is the bottleneck. Typical workloads include:

• Classifying thousands of support tickets, product reviews, or sales notes
• Extracting structured fields from invoices, contracts, or forms
• Running an evaluation set against a new model
• Summarizing a document corpus
• Embedding content repositories
• Generating content variants for internal review
• Running moderation checks at scale

The economics are simple: if the work can wait up to 24 hours, every request in the batch costs half as much as the equivalent synchronous call. If the work cannot wait, the Batch API is the wrong tool.

There is also an operational benefit. Because batches use a separate rate-limit pool, a large offline job does not consume the same quota as your real-time application traffic. That separation can be more valuable than the discount itself when nightly evaluation jobs compete with production requests.

When Not to Use the Batch API

Do not use the Batch API for:

• Interactive chat or copilot responses
• Tool-calling loops where the next action depends on the previous response
• Streaming interfaces
• Time-sensitive workflows such as live support
• Requests that must complete in seconds
• Workflows that cannot tolerate an expired or partially failed batch

Batch jobs can complete faster than 24 hours, and often do, but your architecture should treat 24 hours as the contract, not the expectation. If your downstream process cannot handle that uncertainty, use the synchronous API or redesign the workflow before adopting batch processing.

Supported Endpoints and Request Limits

The official guide lists the endpoints Batch currently supports:

The parameters inside each line's body field match the parameters of the underlying endpoint. That means your existing request construction logic often carries over directly; only the wrapper changes.

Three structural rules matter most:

1. Each request needs a unique custom_id. This is your join key for output and errors.
2. One input file can only target a single model. If you need multiple models, create separate batches.
3. Input files are JSONL. Each line must be a complete, valid JSON object, with no commas between lines and no top-level array.

The upload limit is 200 MB. For image and multimodal workloads, OpenAI recommends referencing remote assets through image_url instead of embedding base64 blobs in the JSONL file. That keeps files well below the limit and makes failures easier to diagnose.

Model Naming and Provider Configuration

The examples in this guide use OpenAI-style model IDs. When running through OurToken or another OpenAI-compatible gateway, confirm the exact model ID from the provider's model catalog before generating the batch file. A display name, provider name, and model ID are different values, and mixing them is a common cause of model_not_found failures.

For current routes and pricing, check the OurToken model catalog. If you are comparing model costs before deciding which route belongs in a batch, the AI API pricing comparison guide shows how to think about input, output, and cached-token costs together.

Preparing the JSONL Input File

Suppose you have a CSV of customer reviews and want to classify each one. The source data might look like this:

review_id,review_text
1001,The product arrived early and works perfectly.
1002,The packaging was fine, but the app stopped syncing.
1003,Support answered in two minutes and fixed the issue.

The batch input file needs one JSON object per review:

{"custom_id":"1001","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.4-mini","messages":[{"role":"system","content":"Classify the sentiment as positive, neutral, or negative. Reply with only one word."},{"role":"user","content":"The product arrived early and works perfectly."}]}}
{"custom_id":"1002","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.4-mini","messages":[{"role":"system","content":"Classify the sentiment as positive, neutral, or negative. Reply with only one word."},{"role":"user","content":"The packaging was fine, but the app stopped syncing."}]}}
{"custom_id":"1003","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5.4-mini","messages":[{"role":"system","content":"Classify the sentiment as positive, neutral, or negative. Reply with only one word."},{"role":"user","content":"Support answered in two minutes and fixed the issue."}]}}

A small Python generator keeps this maintainable:

import csv
import json

SYSTEM_PROMPT = (
    "Classify the sentiment as positive, neutral, or negative. "
    "Reply with only one word."
)

with open("reviews.csv", newline="", encoding="utf-8") as source, open(
    "batch-input.jsonl", "w", encoding="utf-8"
) as target:
    for row in csv.DictReader(source):
        request = {
            "custom_id": row["review_id"],
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": {
                "model": "gpt-5.4-mini",
                "messages": [
                    {"role": "system", "content": SYSTEM_PROMPT},
                    {"role": "user", "content": row["review_text"]},
                ],
            },
        }
        target.write(json.dumps(request, ensure_ascii=False) + "\n")

This generator has three properties that matter at scale:

• It writes one JSON object per line, which is the exact format the Batch API expects.
• It uses the review ID as custom_id, so results can be joined back to the original row.
• It reads and writes UTF-8 explicitly, which prevents silent character corruption in multilingual datasets.

Creating and Monitoring a Batch in Python

After generating the JSONL file, upload it through the Files API with purpose="batch":

from openai import OpenAI

client = OpenAI()

input_file = client.files.create(
    file=open("batch-input.jsonl", "rb"),
    purpose="batch",
)

print(input_file.id)

Then create the batch:

batch = client.batches.create(
    input_file_id=input_file.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    metadata={"description": "nightly-review-classification"},
)

print(batch.id)

The only supported completion window is currently 24h. The metadata field is optional, but it is useful when your team runs multiple nightly jobs and needs to identify a batch months later.

To check status:

batch = client.batches.retrieve(batch.id)
print(batch.status)
print(batch.request_counts)

A production poller should sleep between checks rather than continuously hitting the API:

import time

while True:
    batch = client.batches.retrieve(batch.id)

    if batch.status in {"completed", "failed", "expired", "cancelled"}:
        break

    print(batch.status, batch.request_counts)
    time.sleep(300)

The Batch object exposes request_counts.total, request_counts.completed, and request_counts.failed, which gives you a progress signal before the job finishes.

Batch Statuses You Will Actually See

Do not treat completed as meaning every request succeeded. It means the batch finished; individual requests can still appear in the error file.

Retrieving Results and Handling Failures

When the batch is complete, download output_file_id and write it to a local JSONL file:

import json

result_content = client.files.content(batch.output_file_id).text

with open("batch-output.jsonl", "w", encoding="utf-8") as output:
    output.write(result_content)

for line in result_content.splitlines():
    result = json.loads(line)
    print(result["custom_id"], result["response"]["status"])

Each result line includes the custom_id, so joining it back to the source CSV is straightforward:

import csv
import json

with open("reviews.csv", newline="", encoding="utf-8") as source:
    reviews = {row["review_id"]: row for row in csv.DictReader(source)}

with open("batch-output.jsonl", encoding="utf-8") as output:
    for line in output:
        result = json.loads(line)
        review_id = result["custom_id"]
        review = reviews[review_id]

        message = result["response"]["body"]["choices"][0]["message"]
        print(review_id, message["content"], review["review_text"])

If error_file_id is set, download it as well. The error file tells you which requests failed and why, which is much more useful than rerunning the entire job blind.

if batch.error_file_id:
    errors = client.files.content(batch.error_file_id).text
    with open("batch-errors.jsonl", "w", encoding="utf-8") as output:
        output.write(errors)

A safe post-processing pattern is to store three groups separately:

records/
  source.csv
  batch-input.jsonl
  batch-output.jsonl
  batch-errors.jsonl
  processed.parquet
  failed.parquet

Then your downstream job can load successful records, retry failed records, and preserve the original input for auditability.

Cost Model: How the 50% Discount Actually Works

OpenAI's Batch API gives a 50% cost discount compared with synchronous API calls. The simplest way to model the saving is:

Batch cost = synchronous input cost * 0.5
           + synchronous output cost * 0.5

For example, if a synchronous workload would cost $200 in input tokens and $100 in output tokens, the equivalent batch workload is priced at $100 + $50, or $150 total, before any other discounts your provider applies.

The discount applies to the request, not to your engineering time. Before adopting batch processing, account for:

• JSONL generation and validation
• File upload and storage
• Polling and job orchestration
• Result joining
• Error retries
• Monitoring and alerting
• Data retention and audit requirements

For small datasets, those operational costs can exceed the API savings. Batch processing usually becomes worthwhile when you have hundreds or thousands of requests, a repeatable source of truth for each record, and a downstream process that can consume delayed results.

Cost Calculation in Python

A simple calculator makes the tradeoff explicit:

def batch_savings(
    input_tokens: int,
    output_tokens: int,
    input_price_per_million: float,
    output_price_per_million: float,
) -> tuple[float, float]:
    synchronous_cost = (
        input_tokens * input_price_per_million / 1_000_000
        + output_tokens * output_price_per_million / 1_000_000
    )
    batch_cost = synchronous_cost * 0.5
    return synchronous_cost, batch_cost


sync_cost, batch_cost = batch_savings(
    input_tokens=4_000_000,
    output_tokens=1_000_000,
    input_price_per_million=0.15,
    output_price_per_million=0.90,
)

print(f"Synchronous: ${sync_cost:.2f}")
print(f"Batch: ${batch_cost:.2f}")
print(f"Saved: ${sync_cost - batch_cost:.2f}")

Use the actual current prices for the model you select. Model catalogs change often enough that copying an old number into a production calculator is a common source of budget surprises.

Batch API vs Prompt Caching vs Model Routing

Batch processing is one cost lever, not the only one. It solves a different problem from prompt caching and model routing.

These techniques can overlap. A nightly job might use Batch API pricing, a cached system prompt, and a lower-cost model for routine classification. An interactive application might use prompt caching and model routing, but no batching because users need immediate responses.

Our OpenAI prompt caching guide covers the caching side, and the LLM model routing guide explains how to send easy work to cheaper models while reserving stronger models for complex requests.

Designing a Reliable Batch Pipeline

A production batch pipeline should be idempotent and observable. At minimum, persist:

• The source dataset version
• The generated JSONL file hash
• File ID and batch ID
• Model ID
• Prompt version
• Job metadata
• Start and completion timestamps
• Request counts
• Output and error file IDs

A useful job record looks like this:

{
  "batch_id": "batch_abc123",
  "source_version": "reviews-2026-08-28",
  "input_sha256": "9f2c...",
  "model": "gpt-5.4-mini",
  "prompt_version": "sentiment-v3",
  "status": "completed",
  "request_counts": {"total": 50000, "completed": 49982, "failed": 18},
  "output_file_id": "file-output",
  "error_file_id": "file-errors"
}

That record answers the question every data team eventually asks: "Which model and prompt produced this dataset?" Without it, a table full of generated labels becomes expensive to debug and unsafe to reproduce.

Retry Strategy

Do not retry the entire batch because a small number of requests failed. Instead:

1. Download the error file.
2. Parse failed custom_id values.
3. Join them back to the source records.
4. Generate a new JSONL file containing only the failed requests.
5. Submit a new batch.
6. Merge the retry results with the successful first run.

This approach avoids paying again for the requests that already succeeded and keeps your dataset lineage clean.

Conclusion

The OpenAI Batch API is a straightforward way to cut asynchronous LLM workloads in half: build a JSONL file, upload it, create a batch, poll until completion, and download the output and error files. The tradeoff is latency and orchestration. Batches target a 24-hour completion window, so they belong in offline pipelines rather than interactive products.

Use Batch API when you have many requests, a stable record ID for each request, and a downstream process that can consume delayed results. Use synchronous APIs when the user is waiting. Combine batching with prompt caching and model routing when the workload supports all three, but measure each layer separately so you know which optimization actually produced the saving.

FAQ

What is the OpenAI Batch API?

It is an asynchronous API for submitting large groups of requests in a JSONL file. OpenAI processes the batch within a 24-hour target window and charges 50% less than equivalent synchronous API calls.

How much does the OpenAI Batch API cost?

Batch requests are priced at 50% of the equivalent synchronous request cost. Calculate input and output token costs for your selected model, then halve the total. Check the current model pricing before running production jobs.

What file format does the OpenAI Batch API use?

JSON Lines. Each line is a complete JSON object containing custom_id, method, url, and body. There is no top-level array and no commas between lines.

How long does an OpenAI batch take?

Each batch targets completion within 24 hours, and OpenAI notes batches often finish faster. Design your system around the 24-hour window, not the fastest possible observed completion time.

Can one batch contain multiple models?

No. Each input file can contain requests for only one model. Create separate batch files for separate models.

What is the maximum OpenAI batch file size?

The documented upload limit is 200 MB. For multimodal requests, OpenAI recommends referencing remote assets with image_url rather than embedding large base64 payloads.

How do I handle failed requests?

When the batch completes, download both output_file_id and error_file_id. Retry only the failed custom_id values in a new batch, then merge the successful results.

When should I use the Batch API instead of prompt caching?

Use Batch API when requests can wait up to 24 hours and the workload is large. Use prompt caching when repeated prefixes or shared context appear in synchronous requests. The two techniques solve different problems and can be combined.