Rate limiting

Rate limits show up on both sides of an AI feature. Providers limit your organization by requests and tokens per minute, varying by model and usage tier, and return 429 errors when you exceed them. Meanwhile, your product needs its own limits on users, because every user request spends your money.

On the provider side, handle 429s with retries and exponential backoff that respects any retry-after header; official SDKs already retry a couple of times by default. Run bulk jobs through a queue with controlled concurrency, or through a batch API, so they don't starve interactive traffic. A fallback provider or gateway can absorb spikes.

On your side, rate limiting is a cost and abuse control. Limit per user and per workspace, by requests and by tokens or spend, since one huge request can cost as much as hundreds of small ones. Pair short-window limits with daily or monthly budgets tied to the customer's plan, and tell users clearly when they've hit a limit instead of failing silently. The mistake is treating provider limits as your only ceiling: they protect the provider, not your bill.