Speech-to-speech model

A speech-to-speech model, often called a realtime model, listens, reasons, and speaks inside one model over a persistent connection. OpenAI's Realtime API and Google's Gemini Live API are the best-known examples. Because there's no separate transcription and synthesis step, replies start quickly and the model can pick up tone and hesitation.

The alternative is a chained pipeline: speech-to-text, a text model, then text-to-speech. Pipelines are slower and depend on streaming every hop, but they let you use your strongest text model, choose any voice, and keep an exact text record of every step. Speech-to-speech fits when conversational feel matters most and tasks are fairly simple; a pipeline fits when reasoning, tool use, a custom voice, or compliance records matter more. A hybrid, where the realtime model calls a tool that hands hard questions to a text model, gets you some of both.

Realtime sessions bill for audio and cost money while they stay open, so cap session length and idle time. Never send your API key to the browser; have your server mint a short-lived client secret instead. And handle interruptions so the conversation history contains only what the user actually heard.