Deploying AI Features: Serverless, Containers, and Timeouts

AI features break deployment assumptions that held for a decade. A typical API request finishes in 200 milliseconds; a model call with thinking and a long answer can take two minutes. A typical response is sent in one piece; a chat reply streams for 40 seconds through a CDN, a load balancer, and a proxy, any of which may buffer it or cut it off. The feature that worked flawlessly on your laptop fails in production with a timeout error nobody on the team configured.

This lesson is about getting AI features through the infrastructure between your code and your users. You'll learn why AI routes hit platform limits, how the main hosting options compare for AI workloads, how to set timeouts as a stack so the right layer fails first, how to keep streams flowing through proxies, where to put your compute, and how to handle provider keys across environments.

Why AI routes break platform defaults#

Most platform defaults were tuned for short, cheap requests. AI routes differ in four ways:

• Duration. Output speed is limited by generation. A response with a few thousand output tokens plus thinking can run for minutes, especially at higher effort settings.
• Mostly waiting. An AI request spends nearly all its time waiting on the provider, using almost no CPU. Platforms that bill by wall-clock time charge you for that waiting; platforms that bill by CPU time mostly don't.
• Streaming. Long responses only feel fast if they stream (Streaming End to End), and every hop between your server and the browser has to pass bytes through as they arrive.
• Bursty concurrency. A hundred users chatting means a hundred long-lived open connections at once, each holding memory and, on some platforms, an instance.

None of this requires exotic infrastructure. It requires knowing your platform's limits and configuring them on purpose.