Unlock Exclusive Savings for Your Company’s Travel & More
Create a new Corporate Account between 9/1/25 - 3/31/27 and receive up to 10% off eligible Enterprise Profile rides taken in the U.S. (max $10 off per trip; 5 rides per user) for Authorized Users for 90 days or until 3/31/2027, whichever occurs first. Valid only for new Uber for Business clients in the U.S. that sign up here.
Promotion valid until 3/31/2027. Subject to change or cancellation at anytime. Other terms and conditions apply, see full terms.
For years, voice assistants have followed a familiar sequence: the user speaks, the system detects the end of speech, the speech is converted to text, a language model generates an answer, and another model converts the answer back to audio.
Each stage can be impressive individually, but the combined experience often feels mechanical. The assistant waits too long, interrupts at the wrong moment, ignores vocal cues — or makes awkward pauses before responding.
The latest generation of sound systems seeks to get rid of these basic limitations. Instead of treating a conversation as a series of audio requests, it treats speech as a continuous two-way stream. Models can receive audio while producing audio, determine whether to listen or respond, and delegate complex work to other models without interrupting the conversation.
This is a more profound change than faster speech generation. Voice AI is evolving from chatbots with microphones to real-time distributed systems.
Human conversation takes place over a very demanding period of time. Studies of everyday interactions show the distance between speakers is about 200 milliseconds. People often start planning their responses before the other person has finished speaking, (in practice) using vocabulary, syntax, intonation, pacing. And context to anticipate when their turn will end.
Traditional voice channels work against this natural rhythm.
The speech-to-text component must wait for enough audio to produce a reliable transcription. The turn detector must determine whether a pause means the user is finished or just thinking. The language model must process the completed prompt. Text-to-speech should then create and start playing the response.
Even though each component is fast, serial execution will pile up latency. Transcription can also remove information conveyed through tone, hesitation, emphasis, laughter, volume — or tempo.
The live speech-to-speech model improves this by understanding and producing audio natively. OpenAI’s Realtime documentation says that this avoids the intermediate steps of speech-to-text and text-to-speech, reducing latency while preserving information about tone and inflection.
But direct audio processing alone does not solve all interaction problems. Speech-to-speech models that are still waiting for turn detectors remain stuck in the same basic structure: listen, stop, decide, respond.
The more important transition is from turn-based inference to continuous inference.
Full-duplex sound systems can receive and output audio At the same time,. Instead of viewing conversations as alternating messages, it observes the ever-changing flow and repeatedly decides what conversational actions are appropriate.
The action may be to continue listening, start answering, pause, greet the speaker, interrupt, yield, call a tool, or wait for a delegated result. GPT-Live is designed based on this full duplex model, eliminating a separate loop detector from the main audio path.
This is important because silence is ambiguous.
User can pause because it has finished. They may also search for a word, emphasize a point, check information, or wait for a reaction. Small detectors that only examine silence duration or acoustic features have limited context to differentiate them.
Full voice models can theoretically make decisions using a wider range of conversations. This can consider what was said, how it was said, whether the sentence sounded complete, whether the user seemed unsure — and whether acknowledgment would be helpful without taking over the conversation.
The interaction becomes less like sending voice commands and more like maintaining a shared conversational state.
The main design decision was to divide the system into two paths.
The direct path handles work that cannot be postponed: receiving audio frames, performing continuous speech inference — and returning audio to the user.
Separate asynchronous paths handle tracing, tool calls, business logic, persistence, analytics, policy evaluation, and deeper reasoning. A slow external service may delay the results, but it doesn’t stop audio frames from moving through the direct path.
This resembles a pattern already used in high-performance transaction systems. The synchronous path contains only what must happen before the user can continue. Everything else is delegated to background processing.
The difference is that the deadline is now perceptual. Pending database writes may not be visible. Late audio frames can introduce pauses, (at least that’s the general idea) clicks, overlaps, or broken sentences that users hear immediately.
This separation creates an important product advantage: application behavior can change without destabilizing the media layer. Developers can add tools, modify policies, change databases, or introduce approval workflows while leaving the latency-sensitive audio transport relatively small and predictable.
OpenAI’s public Realtime architecture reflects the same principle through a sideband connection. The user’s client can communicate with the realtime session through WebRTC, while the application server connects separately to monitor the session, update instructions, handle tools, and keep sensitive business logic away from the client.
Separating voice from deeper reasoning creates another challenge: the fast model must decide when to delegate and what to do while it waits.
Suppose a user asks for nearby restaurants, a comparison of several products, or the status of an external account. The voice model cannot answer reliably without search or tool access. It must dispatch the task to another model or service.
However, simply saying “please wait” after every delegation would recreate the mechanical experience the architecture is supposed to eliminate.
The voice layer may instead acknowledge the request, ask a useful clarifying question, explain what it is checking, or temporarily remain quiet. When the delegated result arrives, it must incorporate that information into the conversation without sounding as though another system has taken control.
This makes the voice model a form of conversational scheduler. It manages several timelines simultaneously:
The user’s current speech.
Its own generated speech.
Pending reasoning requests.
External tool calls.
Application-level instructions.
The evolving conversation history.
That design also places a hard limit on how much latency can be concealed. Natural conversational behavior may cover a brief search or model call, but it cannot hide an arbitrarily slow backend. The entire delegation loop—including routing, prompt processing, inference, tools, and result integration—still belongs to the user’s responsiveness budget.
Most language model services are organized around limited demand. The request comes in, the token is created, the response is completed, and the resource can be released or reused.
Live voice sessions behave differently.
The connection may remain active for several minutes or hours. Audio frames arrive continuously. The context evolves over the course of the call. The same inference state must remain available even when the infrastructure is upgraded, upgraded, rebalanced, or recovered from failure.
This makes realtime speech rendering more like stateful stream processing than conventional stateless APIs.
OpenAI describes warming up a replacement model instance along with the active one, pre-populating OpenAI with existing session context, running both instances temporarily — and cutting only once the new instance is ready. The user continues to talk to the original instance Sure, a, but replacement is set up.
This mechanism can also support context compaction.
As a conversation grows, its history may eventually exceed the available context window. Summarizing or compacting earlier material reduces its size, but it changes the prompt and invalidates the model’s existing key-value cache. Rebuilding that cache synchronously would create a noticeable pause.
The alternative is to perform compaction away from the live instance, prefill a replacement with the compacted history, and switch only when it is ready. The user experiences a continuous conversation even though the underlying inference state has been replaced.
This is effectively a zero-downtime deployment pattern applied inside a single AI conversation.
The model may experience conversation as overlapping audio streams, but most surrounding software still expects messages.
Chat interfaces display user and assistant turns. Analytics pipelines process events. Safety systems review outputs. Memories and transcripts need durable ordering. Customer-support platforms expect individual utterances that can be searched and audited.
Continuous speech therefore has to be converted back into discrete records.
The system uses partial transcripts, timing information, speaker attribution, and floor ownership to build provisional messages. Recent content may be revised as more audio arrives. After enough evidence accumulates, the message becomes final.
This creates two versions of reality:
A speculative conversation view that is fresh but mutable.
An authoritative conversation record that is stable but delayed.
The distinction is essential. A live interface can tolerate a transcript changing as the speaker continues. A billing, compliance, evaluation, or audit system usually cannot.
Backchannels make the problem harder. A brief “yes,” “right,” or “mm-hmm” spoken while the user is talking may be conversationally meaningful without deserving a separate assistant message. By contrast, an interruption that changes the direction of the conversation should probably be represented as an independent event.
Research into natural conversation transcripts similarly finds that backchannels and overlapping speech must be distinguished from primary turns to preserve the structure of real interaction.
Voice responsiveness begins before inference. The system must establish a secure media connection before the first audio frame can reach the model.
WebRTC is well suited to this environment because it already supports low-latency audio, encrypted media, network address translation, jitter management, packet-loss handling, echo cancellation, and changing network conditions. OpenAI recommends WebRTC rather than WebSockets for browser and mobile realtime clients where consistent media performance matters.
However, conventional WebRTC setup contains multiple nested handshakes. Signaling, ICE connectivity checks, DTLS security negotiation, SCTP initialization, and data-channel establishment can require several network round trips.
The proposed WebRTC Abridged Roundtrip Protocol, or WARP, combines and pre-negotiates parts of those handshakes. The current Internet-Draft describes reducing setup from six round trips to one signaling round trip and one media round trip. It also reports that a production deployment saw substantially larger tail-latency improvements when fewer handshake packets reduced exposure to packet loss. The proposal remains an active Internet-Draft rather than an approved IETF standard.
The product architecture goes one step further with pre-negotiated session parameters. By moving the SDP signaling exchange away from the critical path, the server can begin materializing a session when the first valid media packet arrives. A normal signaling flow proceeds in parallel as a fallback.
The broader lesson is that model speed is only one component of perceived latency. Connection setup, geographic routing, packet loss, jitter buffers, tool calls, context preparation, and audio playback all contribute to whether a conversation feels responsive.
Traditional inference capacity planning tends to focus on tokens per second, requests per accelerator, queue depth, or batch utilization.
Realtime voice requires another metric: the number of concurrent sessions that can be sustained while every audio frame meets its deadline.
Voice sessions remain open and continuously consume CPU, networking, queues, memory, model capacity, and session-management resources. A supporting stream handler can saturate before the GPU does. Once frames begin queueing, latency compounds and the degradation becomes audible.
Long-running production traffic also exposes problems that short load tests miss. Context continues growing, clients reconnect, state must be restored, model instances are replaced, and disconnect races appear at service boundaries.
For platform teams, this means a successful benchmark cannot simply replay thousands of short conversations. It should reproduce realistic session duration, geographic distribution, background noise, interruptions, reconnects, tool latency, and context growth.
An effective voice SLO should therefore include more than average response time. Teams need to measure session-establishment latency, time to first audible response, frame-delivery deadlines, jitter, interruption recovery, transcript-finalization delay, tool-result latency, session-handoff success, and error rates by geography.
Tail latency is especially important. Users may tolerate a slightly slower average conversation more readily than an assistant that occasionally freezes, speaks over them, or cuts out mid-sentence.
Continuous speech-to-speech is an excellent fit for tutoring, accessibility, live translation, hands-free assistance, conversational search, interactive entertainment, and customer experiences where natural pacing is central.
It may be less appropriate for approval-heavy or tightly regulated workflows.
OpenAI’s developer guidance continues to distinguish direct speech-to-speech sessions from chained voice pipelines. A chained design—speech-to-text, controlled agent workflow, then text-to-speech—gives the application stronger access to intermediate text and may be preferable when the process requires deterministic logic, durable transcripts, explicit approvals, or extensive inspection between stages.
A bank confirming a transfer, a healthcare system recording consent, or an enterprise agent executing infrastructure changes may value explicit checkpoints more than conversational fluidity.
The architectural choice should therefore follow the workflow’s risk model:
Use continuous speech-to-speech when immediacy and natural interaction are central.
Use a chained pipeline when inspectability and deterministic control matter more.
Use a hybrid design when conversation can remain fluid but consequential actions require structured confirmation.
The arrival of full-duplex voice does not eliminate traditional pipelines. It makes the tradeoff clearer.
The most important implication is not that AI voices will sound more realistic. It is that voice can become a persistent interface for coordinating longer-running digital work.
A user may talk while an agent searches, opens applications, compares information, waits for another service, delegates to specialist agents, or performs tasks in the background. The conversation becomes the control layer, not necessarily the location where all computation happens.
That requires intelligence at two speeds.
The first speed is immediate and social. It maintains attention, timing, acknowledgements, interruptions, and conversational continuity.
The second speed is deliberate and computational. It performs search, reasoning, tool use, validation, and complex execution.
Separating those speeds is what allows the system to feel responsive without limiting it to simple answers. But it also introduces new distributed-systems problems: synchronizing state, reconciling transcripts, recovering delegated work, preventing stale results, enforcing permissions, and ensuring that a natural-sounding acknowledgement is not mistaken for confirmation that an external action succeeded.
The next generation of voice AI will therefore be judged by more than model intelligence or speech realism. It will be judged by whether the entire system can maintain conversational timing while remaining accurate, observable, recoverable, and safe.
That is the real engineering breakthrough behind continuous voice interaction: not simply teaching an AI to speak, but redesigning the stack so the conversation never has to stop.
Supported by Bright Data
Turn the Web Into A Data Pipeline. Generate Your API, in Minutes.
If your team is still patching selectors and rotating proxies, you’re maintaining infrastructure your competitors stopped owning.
Scraper Studio turns any public website into a hosted data API from a single prompt:
Self-healing scrapers that detect site changes and patch themselves
Proxies, unblocking, and parsing handled - not your problem
Build from Claude Code, Cursor, or Codex via CLI - or visually in the control panel
Scale production data pipelines without owning the stack
5,000 free credits every month
No sales call. No demo. Just spin up a scraper and watch it run.
No posts
Comments
Nothing yet. Say the first thing.
Sign in to join the conversation.