LLM gateway

An LLM gateway sits in the path of every model call. Your code talks to the gateway, usually through an OpenAI-compatible API, and the gateway forwards each request to the provider and model you named. Hosted gateways such as Vercel AI Gateway, OpenRouter, and Cloudflare AI Gateway run on the vendor's infrastructure; self-hosted proxies such as LiteLLM run on yours.

The value is in cross-cutting work: falling back to another provider when one is down or rate-limiting you, issuing separate keys per app or team, enforcing spend limits, and recording cost and latency for every call in one place. When several services share models, a gateway stops each one from reinventing that plumbing.

A gateway is also one more dependency. It adds a network hop, can become a single point of failure, and may lag behind providers on new features, since provider-specific options like caching controls, thinking settings, or new tool types don't always pass through cleanly. A hosted gateway also sees your prompts, so check its data terms. For one app with one or two providers, a small fallback wrapper in your own code is often enough. And treat fallback models as a quality decision: run your evals on the backup before an outage forces you onto it.