Rate Limits, Quotas, and Per-User Budgets

Every AI feature sits between two kinds of limits. Above you, the provider caps how many requests and tokens your whole organization can send per minute, and when you hit that cap, every feature for every customer starts failing at once. Below you, your users can send as much as you let them, and without limits of your own, one enthusiastic customer, one runaway script, or one bad actor can burn through a month of margin in an afternoon.

In this lesson you'll handle both sides. You'll learn how provider rate limits work and how to respond to them without making things worse, then build your own controls for Deskmate: per-user request limits, per-organization spend budgets that can't be overrun, and plan-based limits that degrade gracefully instead of slamming a door on paying customers.

How provider rate limits work#

Provider rate limits protect shared capacity. Anthropic's are set per organization and per model class, measured in three units:

• Requests per minute (RPM)
• Input tokens per minute (ITPM)
• Output tokens per minute (OTPM)

You can hit any of the three first. A feature sending many short requests hits RPM; one stuffing 100,000-token documents into each call hits ITPM long before RPM. Your limits depend on your usage tier and are shown in the provider console, so look them up rather than guessing.

A few details change how you design around them:

• They're token buckets, not fixed windows. Capacity refills continuously up to the maximum rather than resetting on the minute. A limit of 60 RPM may be enforced as roughly one request per second, so a burst of 60 requests in the first second can be rejected even though you're "under 60 per minute."
• Cached reads don't count toward ITPM on current Claude models; only uncached input and cache writes do. A well-cached feature has far more input headroom than its raw token count suggests (see Prompt Caching in Practice).
• Sudden ramps can be throttled. Anthropic applies acceleration limits when an organization's usage spikes sharply, and recommends increasing traffic gradually. A launch day or a big backfill can trip this even below your nominal limits.
• Batches have separate limits. Work sent through the Message Batches API doesn't consume your Messages API limits, which is one more reason to move bulk work there (Batch APIs and Async Processing).
• Limits are shared across everything. Every feature, every customer, every background job, and every developer running scripts against the production key draws from the same bucket. That's the part that hurts: a backfill job can take down live chat.

When you exceed a limit, you get an HTTP 429 with a rate_limit_error and a retry-after header saying how many seconds to wait. Don't confuse it with 529 (overloaded_error), which means the provider itself is under heavy load across all customers. A 429 is about your usage; a 529 isn't, and no amount of throttling on your side fixes it.