LLM Gateways and Provider Fallbacks
Put a gateway between your app and providers for retries, fallbacks, keys, and spend tracking — and know what it costs you in complexity.
At some point your model provider will have a bad hour. Requests start returning overloaded errors, rate limits tighten during a launch, or an outage takes a region down. If every AI feature in your product calls the provider SDK directly, every one of them breaks at once, and fixing it means a deploy for each feature while users wait.
An LLM gateway is a layer between your app and the model providers that handles this kind of problem in one place: retries, fallbacks to other models or providers, API keys, spend tracking, and logging. In this lesson you'll build a provider fallback chain in plain TypeScript so you understand exactly what a gateway does, then look at the hosted and self-hosted gateway options, how to use them for keys and spend tracking, and the costs they add, so you can decide whether you need one at all.
What a gateway does, and what you already have
A gateway can be a library inside your app, a proxy you run, or a hosted service. Whatever its form, it takes on some combination of these jobs:
| Job | What it means in practice |
|---|---|
| Unified API | One request format for many providers and models |
| Retries | Re-sending a request after a transient error, with backoff |
| Fallbacks | Trying a different model, deployment, or provider when the primary fails |
| Key management | Provider keys live in one place; apps get scoped keys that can be revoked |
| Spend tracking and budgets | Cost attributed per feature, team, or customer, with limits |
| Logging | Every request and response recorded in one format |
| Caching and rate limiting | Optional response caching and request limits in front of the provider |
You already have one of these for free. The Anthropic TypeScript SDK retries connection errors, 408, 409, 429, and 5xx responses twice by default with backoff, and the OpenAI SDK does the same for its API. For many apps, that plus sensible timeouts is enough reliability. A gateway earns its place when you need the rest of the list: cross-provider fallbacks, central keys, and cost attribution across many features.