Batch processing guide
Batch API Cost Calculator Guide
Batch APIs are useful when a large AI job can wait. Estimate the normal interactive cost first, then apply the current batch pricing rules only to requests that can run asynchronously.
Good batch workloads
Batch processing fits non-urgent work such as document summarization, data extraction, offline classification, evaluation runs, translation backlogs, nightly enrichment, and synthetic test generation. It is a poor fit for chat, live search, user-facing agents, or any flow where the user expects an immediate answer.
The batch cost formula
Planning formula
Interactive baseline = input token cost + output token cost for every request
Batch estimate = eligible input token cost + eligible output token cost at the provider's current batch rate
Total project cost = batch estimate + real-time requests + validation reruns + storage or orchestration overhead
Use the calculator to estimate input and output tokens per item, then multiply by the number of records in the job. Keep input and output separate because output-heavy jobs can still be expensive even after a batch discount.
Collect these inputs
- Records per batch: documents, rows, tickets, images, or test cases processed together.
- Average input tokens: prompt instructions plus the record content sent for each request.
- Expected output tokens: summaries, JSON fields, labels, extracted evidence, or rewritten text.
- Failure and rerun rate: invalid JSON, policy refusals, low-confidence extraction, or duplicate jobs.
- Latency tolerance: whether the product can wait for the provider's batch turnaround window.
Batch vs. real-time cost: worked example
A support-ticket job processes 40,000 records, each with 1,200 input tokens and 180 output tokens: 48 million input tokens and 7.2 million output tokens. For this example only, assume standard rates of $2 per million input tokens and $8 per million output tokens, with batch rates of $1 and $4. These are illustrative rates, not a quote for any model.
Same workload, different processing choices
- All real time: (48 × $2) + (7.2 × $8) = $153.60.
- All batch, no reruns: (48 × $1) + (7.2 × $4) = $76.80.
- All batch, then 6% rerun once in batch: $76.80 × 1.06 = $81.408 (about $81.41).
- All batch, then 6% rerun once in real time: $76.80 + ($153.60 × 0.06) = $86.016 (about $86.02).
The rerun cases assume every original request completed and was billed, but 2,400 results need a quality correction. With unchanged token counts, those reruns add exactly 2.88 million input tokens and 432,000 output tokens. Longer correction prompts or repeated retries require a larger budget. For a fair comparison, include any quality reruns that the real-time baseline would also need.
What if only part of the workload can wait?
Let C be the standard cost of the full workload, f the fraction of that cost eligible for batch, and d the batch discount as a decimal. When the discount is the same for input and output:
Mixed cost before extra overhead = C × (1 − f × d)
Net savings = C × f × d − extra retry cost − extra operating cost
In the example above, if 75% of records can wait and every record has the same token mix, C = $153.60, f = 0.75, and d = 0.50. Batch costs $57.60 and the real-time portion costs $38.40, totaling $96.00. Savings are $57.60, or 37.5%, before extra costs. A 50% batch discount does not mean a 50% reduction in the whole bill.
Batch is cheaper in this comparison only while extra retries and operating costs stay below $57.60. Count only costs added by choosing batch; shared expenses belong on both sides. If large records are more likely to be batched, use their share of baseline token cost, not simply their share of request count. If input and output discounts differ, calculate each token bucket separately.
Do expired or failed batches cost money?
Do not treat every unsuccessful job as a fully billed second run. OpenAI's batch expiration documentation says completed requests remain billable when a batch expires; unfinished requests are cancelled. Reconcile output and error files by custom_id before resubmitting only the records that still need work. A completed answer rejected by your quality checks is different from a request that never executed.
OpenAI's Batch API guide, checked on August 27, 2026, describes a 50% discount and a 24-hour processing window. Verify eligibility, billing rules, and deadlines for your chosen provider and model; do not assume these terms apply to every batch service.
Compare batch, cache, and real time
Batch pricing and prompt caching solve different problems. Batch helps when the work can run later. Caching helps when many real-time requests reuse the same prefix. For a backlog with repeated instructions, estimate both options separately and confirm whether the provider allows any discounts to stack before assuming combined savings.
Some workflows should be split: run historical backfills through batch, then keep new user-facing traffic on a real-time endpoint. Track the two paths separately so launch traffic does not hide the true cost of the backlog.
Next, compare the prompt caching cost breakdown or turn the job into a monthly API budget. The main calculator estimates workload tokens and standard model costs; apply the separate batch rates and mixed-traffic formula above to that baseline.
Operational checklist
- Validate prompts on a small sample before submitting the full job.
- Store stable job IDs so the same records are not submitted twice.
- Keep enough source metadata to audit low-quality outputs later.
- Budget for a human or automated review pass when extraction accuracy matters.
- Check current provider docs before purchase decisions because supported models, rate limits, and pricing can change.
Official pricing references
Confirm the latest details in the OpenAI Batch API guide, Claude batch processing guide, and Gemini Batch API guide.