Batch APIs and Async Processing
Process large volumes at roughly half price with batch APIs: when they fit, how to submit and collect results, and designing features around delay.
Some AI work has a user waiting on it. A lot of it doesn't. You ship a better tagging prompt and want it applied to every ticket from the last two years. Product wants a sentiment score on 400,000 support conversations. Your eval suite runs 2,000 cases every night. None of these need an answer in two seconds, yet if you run them through the normal API, you pay the same price as a live chat reply.
Batch APIs let you trade speed for money. You hand over thousands of requests at once, the provider processes them when it has spare capacity, and you pay roughly half price. In this lesson you'll learn which workloads fit, how to submit and collect a batch reliably with the Anthropic SDK, how to stack batching with prompt caching, and how to design features that are honest about the delay.
What a batch API trades away
A batch API accepts a large set of independent requests in one call and processes them asynchronously. Instead of getting a response per request, you get a batch ID, check on it later, and download all the results when it's done.
With Anthropic's Message Batches API, the trade looks like this:
| You get | You give up |
|---|---|
| 50% off standard prices for input and output | An immediate answer: most batches finish within an hour, but you're only promised up to 24 hours |
| Separate rate limits, so batch work doesn't eat into your live traffic's limits | Streaming: each result arrives whole, at the end |
| Up to 100,000 requests (or 256 MB) per batch | Ordering: results come back in any order |
| No charge for requests that error, get canceled, or expire | Simplicity: you now have a submit step, a collect step, and partial failures |
Requests that aren't processed within 24 hours expire, and results stay downloadable for 29 days after the batch is created. Every model is available through batches, and each request in a batch is a normal Messages API request: same model, system, messages, structured outputs, and tools. A few things aren't supported in batch requests, including streaming and the speed option.
The separate rate limits matter more than they first appear. A big backfill run through the normal API competes with your users for the same requests-per-minute and tokens-per-minute budget. Run it as a batch and your interactive features don't notice. (Rate limits get their own lesson: Rate Limits, Quotas, and Per-User Budgets.)
Which workloads fit
The test is simple: would anyone notice if this result arrived an hour from now? If not, it's a batch candidate.
| Workload | Batch? | Why |
|---|---|---|
| Re-tagging every historical ticket after a prompt change | Yes | Nobody is waiting; volume is huge |
| Nightly digest emails, weekly reports | Yes | Has a deadline, but hours away |
| Running your eval suite on a new prompt | Usually | Half-price evals mean you can afford bigger test sets; skip it when you need results in minutes |
| Generating embeddings-ready summaries for a new knowledge base import | Yes | One-time, bulk |
| Summarizing the ticket an agent just opened | No | Someone is looking at the screen |
| Classifying an incoming ticket for routing | No | Routing delay is user-visible delay |
Notice what's on the "no" list: anything in a request path. For those, the cost levers are prompt caching, smaller models, and fewer tokens, which other lessons in this module cover.