Streaming

Without streaming, your app waits for the entire response before showing anything, which can be many seconds for a long answer. With streaming, the provider sends small pieces as tokens are produced, your server forwards them, and the interface renders text as it arrives. The total time barely changes, but the wait feels far shorter.

A stream crosses several hops, from provider to your server, through your hosting platform and any proxies, to the browser, and each can buffer, time out, or drop the connection. Most streaming bugs are in that plumbing: a proxy that holds the whole response, a serverless timeout, or a client that doesn't parse partial events correctly.

Streaming also complicates the rest of the feature. You can't validate a full JSON object until it's complete, moderation has to handle partial text, errors can arrive mid-response, and if the user navigates away you should abort the upstream request so you aren't paying for output nobody will read.

Use streaming for anything a person waits on: chat, drafting, long summaries. Skip it for background jobs and short structured results, where a single complete response is simpler to handle and validate.