Scaling AI Workloads
Queues, concurrency limits, backpressure, and capacity planning for AI traffic that spikes — without blowing through rate limits or budgets.
Monday, 9 a.m. A weekend's worth of support tickets lands in Deskmate, every agent opens their queue at once, and the AI features that felt instant on Friday start timing out. At the same moment, a customer kicks off an import of 30,000 old tickets, each one queuing a summary. Your servers autoscale fine. The model calls don't, because your provider's rate limit is the same at 9 a.m. Monday as it was at 3 a.m. Sunday.
AI traffic breaks the usual scaling instincts. Requests take seconds instead of milliseconds, capacity is capped by a provider you don't control, and every request costs real money. In this lesson you'll plan capacity with simple math, limit concurrency so you stay inside provider limits, split interactive and background work into priority lanes, and add backpressure so a spike degrades gracefully instead of collapsing.
Why AI traffic is different
Most web scaling advice assumes requests are short, servers are the bottleneck, and adding servers adds capacity. AI features violate all three:
- Requests are long. A typical web request finishes in tens of milliseconds. A model call takes seconds, and a long generation can take a minute. Long requests mean many are in flight at once, holding connections and memory.
- The bottleneck is upstream. Your provider caps requests and tokens per minute for your whole organization (see Rate Limits, Quotas, and Per-User Budgets). Autoscaling your app to 20 instances doesn't raise that cap; it just lets 20 instances hit it at once.
- Traffic is spiky in two ways. Interactive traffic follows your users' working hours. Background traffic arrives in huge bursts: an import, a backfill, a nightly job. When the two collide, the burst starves the interactive features users are watching.
- Every request costs money. Scaling up isn't just a capacity question; a retry storm or a runaway job is also a billing event.
The answer is to treat model capacity as a scarce shared resource and manage it deliberately: know how much you need, control how much you use at once, decide who gets it first, and decide what happens when demand exceeds supply.