AI builder glossary
Batch API
An asynchronous API mode where you submit many requests at once and collect results later, usually at a large discount, for work nobody is waiting on.
A batch API accepts a set of requests, processes them on the provider's schedule, and makes results available when they're done. Anthropic's Message Batches API, for example, charges half the normal price for input and output tokens; most batches finish within an hour, and requests not processed within 24 hours expire. Other major providers offer similar discounted batch modes.
Any work where nobody is watching the screen is a candidate: backfilling summaries for old tickets, tagging a product catalog, reclassifying records after a prompt change, nightly evals, weekly reports. Batching also keeps bulk jobs from eating the rate limits your interactive features need, and the discount can combine with prompt caching when requests share a prefix.
Design for the asynchronous shape. Give every request a custom ID that maps back to your own records, because results can arrive in any order. Expect partial failure: some requests will error or expire, so record each result's status and resubmit the failures. Store the batch ID so a restarted worker can resume polling instead of resubmitting everything. And keep batch out of anything a user is waiting for; interactive features belong on the normal API.
After improving its triage prompt, Deskmate reclassifies tens of thousands of recent tickets overnight in one batch, keyed by ticket ID, and writes each new label back to the database as results come in.
Learn more in the course
Related terms
Go from definitions to doing
Learn to build AI-powered apps in 132 practical lessons.