When to Use Batch Inference (and When Not To)

Batch inference gets you roughly half-price tokens from every major model provider. That's a big enough number that the real question isn't "should we use batch?" — it's "which of our workloads can tolerate it?" This post gives you a practical framework for making that call.

The core trade-off

Batch APIs trade latency for cost. Providers run batch jobs on spare capacity, so they can offer deep discounts, but completion times are measured in minutes to hours (with a typical SLA of 24 hours). Everything else follows from that one trade-off.

Dimension	Real-time API	Batch API
Cost per token	1x	~0.5x
Latency	Seconds	Minutes to hours
Throughput	Rate-limited (TPM)	Effectively unlimited
Failure surface	Per request	Per job + per request

Workloads that belong on batch

If a human isn't actively waiting on the response, the workload is probably a batch candidate:

• Data enrichment — classifying, tagging, or extracting structure from records in a database.
• Embeddings backfills — re-embedding a corpus after a model upgrade.
• Evaluation runs — scoring model outputs across a test set on every deploy.
• Content pipelines — summarization, translation, or moderation of accumulated content.
• Synthetic data generation — producing training or test data at scale.

A useful heuristic: look at your inference logs and ask, for each call site, "would anything break if this response arrived 30 minutes later?" Teams are usually surprised how much of their volume — often 40–70% — answers "no."

Workloads that don't

• Interactive chat and agents — a user is waiting; latency is the product.
• Request-path features — anything blocking a page render or API response.
• Tight feedback loops — multi-step agent chains where step N+1 depends on step N right now.

The cost math

Say you process 2B input tokens and 500M output tokens per month on a mid-tier model at $3/M input and $15/M output:

Real-time:  (2,000 × $3) + (500 × $15) = $13,500 / month
Batch:      (2,000 × $1.5) + (500 × $7.5) = $6,750 / month
Savings:    $6,750 / month (50%)

Even if only half your volume is batchable, that's still ~$3,400/month back — for workloads where nobody notices the difference.

The hidden costs of DIY batch

The discount is real, but so is the engineering effort if you build the plumbing yourself:

1. JSONL assembly and validation — malformed lines fail whole jobs on some providers.
2. Job orchestration — submit, poll, download, parse, on a schedule, with retries.
3. Partial failure handling — a 100k-request job with 200 failures needs surgical retry logic, not a full resubmit.
4. Minimum batch efficiency — small jobs waste the fixed overhead; you end up building an aggregation queue anyway.
5. Provider divergence — every provider has a different batch format, endpoint, and lifecycle.

This is exactly the layer Convoy provides: you send normal-looking API requests, and Convoy handles aggregation, submission, polling, retries, and result delivery across models. See the launch post for how it works.

Getting started checklist

1. Audit your inference call sites and label each one latency-sensitive or latency-tolerant.
2. Estimate monthly token volume for the tolerant set — that × 50% is your savings ceiling.
3. Move one pipeline to batch first (evaluations are a great low-risk start).
4. Measure results, then migrate the rest incrementally.

If you'd rather skip building the orchestration layer, Convoy gets you batch pricing with a real-time-style API — no queues, no minimum volumes, no JSONL.