Say something to most "AI voice agents" today and here is what actually happens: your words get transcribed to text, the text gets sent to a language model, the model's text response gets synthesized back into audio, and the audio plays. Three separate systems, two format conversions, and a chain of network calls — all before you hear a reply. It works, but it has a ceiling, and that ceiling is becoming obvious as voice interfaces move from novelty demos into real customer-facing products.
Speech-to-speech models remove the middle conversions. Instead of transcribing, reasoning in text, and re-synthesizing, a single model processes audio in and produces audio out directly. No intermediate transcript. No separate text-to-speech step re-reading a script it didn't originally "hear." This is a structural change to how voice AI is built, not just a speed improvement, and it's worth understanding both what it fixes and what it doesn't. This guide covers why the three-stage stack became the default, where its latency and lost tone come from, how speech-to-speech voice AI works differently, what that means for teams building voice agents, and the limitations that still favour the older pipeline for some workloads.
The Three-Stage Stack and Why It Was the Default
For years, the practical way to build a voice agent was to stitch together three specialized systems, each doing one job well:
- Speech-to-text (STT) converts the caller's audio into a text transcript. Systems like Whisper-class models handle this — accurately, but not instantly, and with occasional errors on accents, crosstalk, or background noise.
- Large language model (LLM) reads the transcript and generates a text response, applying whatever reasoning, retrieval, or business logic the application needs.
- Text-to-speech (TTS) converts that generated text back into audio, using a voice model that has to guess at intonation from punctuation and word choice alone.
This pipeline made sense because each component was independently improvable. Better transcription models could be swapped in without touching the LLM. Better voices could be swapped in without retraining anything upstream. It's a classic separation-of-concerns architecture, and for a long time it was the only viable way to get a voice interface on top of a text-native language model.
But separation of concerns has a cost when the thing you're optimizing is a live conversation. Every stage boundary is a place where information gets lost and where latency accumulates.
What Gets Lost in Translation
The transcript is the chokepoint. Once speech becomes text, everything that isn't captured in words is gone — tone of voice, hesitation, a nervous laugh, emphasis on a particular word, whether "sure" was said enthusiastically or sarcastically. The LLM never hears any of that; it only sees "sure." Its response is generated blind to how the thing was actually said, and then that response is re-synthesized by a TTS model that also never heard the original — it only receives the LLM's flat text output.
The result is a pipeline that can produce grammatically correct, contextually reasonable responses that still feel emotionally tone-deaf, because the two points where emotional information exists — the caller's voice and the agent's voice — are separated by a text bottleneck that erases it in between.
Where the Latency Comes From
Each stage in the pipeline adds a discrete chunk of processing time, and those chunks are typically sequential rather than overlapping:
| Stage | What it does | Typical latency contribution |
|---|---|---|
| STT | Transcribe audio to text | Waits for speech end, then processes |
| LLM | Generate a text response | Token generation, often after full transcript arrives |
| TTS | Synthesize text to audio | Waits for full or partial text, then renders audio |
| Network hops | Data moves between 2-3 separate services | Added per hop, compounds with each stage |
Individually, each stage might only add a few hundred milliseconds. Stacked in sequence, with network round-trips between often-separate services, the cumulative delay is what makes STT→LLM→TTS voice agents feel like a very fast phone tree rather than a conversation. Humans notice gaps above roughly 200-300 milliseconds in natural dialogue; three-stage pipelines routinely land well past a second, especially once retrieval or tool calls are added to the LLM step — a gap our piece on the race to zero latency explores across real-time AI systems more broadly.
What Speech-to-Speech Actually Does Differently
A speech-to-speech model is trained to take audio as input and produce audio as output within a single model, without a mandatory text transcript as an intermediate representation. Architecturally, this usually means the model operates on audio tokens (or continuous audio representations) alongside — or instead of — text tokens, and it's trained end-to-end on the task of "hear this, respond with speech," rather than "transcribe, reason, then speak" as three separately optimized objectives.
This has a few direct consequences:
- Prosody and tone can be preserved. Because the model processes the actual acoustic signal rather than a stripped-down transcript, it has access to information about how something was said, not just what was said. It can, in principle, respond faster to an urgent tone or pick up on the difference between a genuine question and a rhetorical one.
- Latency drops because stages collapse. With one model handling the full path, there's no waiting for a transcript to finalize before language generation starts, and no waiting for a text response to finalize before audio synthesis starts. Many speech-to-speech systems can begin producing audio output while still processing the tail end of the input, enabling much tighter turn-taking.
- Interruptions and overlap become more natural to handle. Real conversation involves interruptions, "mm-hmm" backchannels, and talking over the end of someone's sentence. A unified audio model is better positioned to model this than a pipeline where the STT step is generally waiting for a clean end-of-utterance signal before it hands anything to the LLM.
- The agent's own voice can carry appropriate emotion. Rather than a TTS model reading a flat transcript with generic inflection, a speech-to-speech model can, in theory, generate a response whose delivery matches its content — because the "speaking" and "deciding what to say" happen inside the same process rather than as two independent guesses.
How the Two Approaches Actually Compare
| Dimension | STT → LLM → TTS pipeline | Speech-to-speech model |
|---|---|---|
| Components | 3+ separate systems | 1 unified model |
| Emotional/tonal signal | Lost at transcription | Potentially preserved end-to-end |
| Latency profile | Sequential, additive | Overlapping, generally lower |
| Modularity | Easy to swap individual components | Harder to swap parts independently |
| Debuggability | Transcript gives a clear audit trail | Audio-native reasoning is harder to inspect |
| Maturity | Well-established, widely deployed | Newer, fewer production battle scars |
| Cost predictability | Each stage billed/optimized separately | Bundled, less granular control |
| Language/accent robustness | Inherits STT's known strengths and gaps | Still maturing across languages |
Neither architecture is strictly better in every dimension — the table above is a set of trade-offs, not a scoreboard, and which side wins depends heavily on what a given application actually needs.
Why This Matters Right Now
Voice interfaces are having a moment precisely because latency and naturalness were the two biggest reasons they felt unusable for anything beyond simple IVR-style tasks. A support line that takes 1.5 seconds to respond to every utterance, and that responds to "I'm really frustrated" in the same flat tone it uses for "what are your hours," doesn't feel like talking to something competent — it feels like talking to a slow, tone-deaf script reader with a search engine attached.
The shift toward unified speech-to-speech models is happening because the pipeline approach hit a wall that better prompting or bigger LLMs couldn't fix. The latency floor of a three-stage architecture is set by physics and network topology, not by model quality — you can have the smartest LLM in the world and still lose the conversation to a two-second pause before it speaks. Collapsing the stages is the only way to meaningfully move that floor, which is why the industry's attention has shifted toward audio-native models rather than continuing to optimize each pipeline stage in isolation.
This matters especially for use cases where the interaction is inherently high-stakes or emotionally loaded — a caller reporting a problem, cancelling a service, or asking about something urgent, the kind of calls a healthcare voice AI receptionist has to handle correctly every time. In those situations, a half-second of extra silence or a mismatched tone reads as incompetence in a way it wouldn't in, say, a typed chat interface where pacing expectations are different.
Benefits of Speech-to-Speech Voice Agents
The architectural changes above translate into outcomes that callers and the businesses serving them can feel directly.
Conversations that keep their rhythm
When the gap between a caller finishing and the agent replying shrinks, the exchange stops feeling like a phone tree. Callers are less likely to repeat themselves, talk over a slow reply, or assume the line has dropped. That rhythm matters most in quick back-and-forth exchanges, such as confirming details or answering short follow-up questions, where pipeline pauses add up across a call.
Responses that fit the caller's mood
A model that hears tone can respond differently to a frustrated caller than to a relaxed one, slowing down, acknowledging the problem, or moving faster toward a human handoff. In a pipeline, that signal is gone before the language model sees the words. For support lines, cancellations, and complaints, matching tone is often the difference between a call that de-escalates and one that ends badly.
More natural interruptions
People interrupt, backchannel, and change their minds mid-sentence. A unified audio model is better placed to handle "actually, wait" or a quick "mm-hmm" without derailing the conversation or talking over the caller. Fewer awkward overlaps make the agent feel attentive rather than scripted, and callers are more willing to stay on the line long enough to get their problem solved.
Fewer moving parts in the voice path
Replacing three services and their network hops with one model simplifies the real-time path. There are fewer integration points to monitor, fewer version mismatches between components, and fewer places where latency can spike unexpectedly. Teams can spend less time tuning handoffs between stages and more time on the conversation design itself.
A better fit for brand voice
Because delivery and content come from the same process, the agent's voice can carry emphasis and warmth that match what it is saying. A pipeline's TTS stage has to infer intonation from punctuation. For brands where the voice experience is part of the product, that coherence is a meaningful advantage, especially across long calls where flat, uniform delivery quickly becomes tiring to listen to.
Speech-to-Speech Voice Agent Use Cases
Speech-to-speech is not the right choice for every voice workload. These are the places where its strengths matter most today.
Customer support front lines
Inbound support calls often start with a frustrated caller explaining a problem in their own words. A speech-to-speech front end can acknowledge the frustration, gather details conversationally, and route to the right place quickly. Complex account changes can still be handed to text-based reasoning or a human behind the scenes, so the caller gets a natural first exchange without losing access to deeper tools.
Reception and booking
Reception lines handle short, frequent exchanges: confirming a time, answering a question about hours, moving an appointment. Tight turn-taking makes these calls feel efficient, and a caller who only needed to move an appointment is done in moments rather than waiting through pauses after every sentence. Where a clean transcript is required for records, teams pair the model with a side transcription process so the audit trail survives.
In-app voice assistants
Apps that let users speak to an assistant, for navigation, hands-free control, or quick questions, benefit from low latency because users are often mid-task. A slow reply interrupts whatever they were doing. Speech-to-speech keeps the interaction brief and fluid, which is what makes voice worth using over tapping. It also handles the quick corrections people make when they misspeak, without forcing them to start the request again.
Conversational practice and coaching
Language practice, interview rehearsal, and sales role-play depend on realistic conversation, including interruptions and natural pacing. These are widely cited early uses because a delayed, flat-sounding partner breaks the exercise. A model that hears hesitation can also adjust its responses to the learner's confidence.
Hybrid agents for complex calls
For calls involving account lookups, policy checks, or multi-step problem solving, teams increasingly use speech-to-speech for the conversational surface and call a text-based LLM with tools when deeper reasoning is needed. The caller hears a natural conversation while the business keeps mature tool use, retrieval, and logging behind it. The handoff between the two layers needs careful latency budgeting so the deeper step does not reintroduce long pauses.
Speech-to-Speech Voice Agent Best Practices
If you're evaluating voice AI vendors or planning to build a voice agent, the STT→LLM→TTS-versus-speech-to-speech distinction has concrete effects on what you should expect and what to ask about.
What Changes for Buyers and Builders
- Latency claims need scrutiny. "Sub-second response time" means something very different depending on architecture. Ask specifically what's being measured — time to first audio byte, time to complete response, or something else — and whether that number holds under realistic network conditions, not just in a demo booth on the vendor's own infrastructure.
- Debugging gets harder without a transcript. Pipeline architectures give you a text transcript almost for free, which makes it easy to review what the agent "heard" and log conversations for QA. Speech-to-speech systems may not produce a clean intermediate transcript at all, which complicates auditing, compliance logging, and post-call analysis unless the vendor has built separate tooling for it.
- Component-level customization gets harder. If you wanted a specific TTS voice for brand consistency, or a specific STT model tuned for your industry's vocabulary, a pipeline lets you swap that one piece. A unified speech-to-speech model is generally a package deal — you get its voice, its behavior, and its accent handling as a bundle.
- Vendor lock-in risk changes shape. Pipelines can be re-assembled from different vendors' components. Speech-to-speech models are typically proprietary end-to-end systems, so switching providers later may mean a more complete rebuild rather than swapping one stage.
- Cost structure shifts. Pipeline costs are usually itemized per stage, which makes it easier to optimize spend on the specific bottleneck. Bundled speech-to-speech pricing is often simpler to reason about per minute but harder to optimize piece by piece.
A Practical Decision Checklist
- Does the use case need sub-second, emotionally attuned responses? High-stakes support, sales, or healthcare triage calls benefit most from speech-to-speech; a scheduling bot that mostly confirms dates and times may not need it.
- Do you need a clean transcript for compliance or QA? If regulatory logging requirements demand verbatim, auditable transcripts, confirm the vendor produces one — some speech-to-speech systems bolt on transcription as a side process specifically for this reason.
- How many languages and accents does your caller base span? Pipeline STT components generally have a longer track record across diverse languages — a challenge closely related to how real-time speech translation systems handle the same accent and language-coverage problems — so verify a speech-to-speech vendor's real coverage rather than assuming parity.
- What's your tolerance for vendor lock-in? If you expect to change voice AI vendors within a year or two, a modular pipeline built on platforms like Twilio or Amazon Lex is easier to migrate incrementally.
- Can you test under real network conditions, not a demo? Ask for a trial on your actual telephony or WebRTC setup, not a curated sandbox, since network jitter affects the two architectures differently.
Common Speech-to-Speech Voice Agent Mistakes
Teams adopting speech-to-speech models tend to trip over the same handful of issues, usually because they evaluate the voice and forget the system around it.
Judging on the demo voice
A natural-sounding demo is persuasive, but callers judge an agent on whether it solves their problem. Teams that pick a vendor because the voice sounds human, without measuring task completion, escalation rate, and error handling on real calls, often find the agent pleasant and ineffective. Evaluate outcomes first and voice quality second.
Forgetting the transcript until compliance asks
Without a mandatory text stage, some systems produce no clean record of what was said. Teams that discover this after launch scramble to bolt on transcription for QA, dispute handling, or regulatory logging. Confirm how transcripts are produced, and how accurate they are, before committing.
Putting business logic inside the voice model
Encoding refund rules, eligibility checks, or escalation policy in prompts to a bundled audio model ties those rules to one vendor and makes them hard to test. Keeping business logic in your own services, called as tools, makes it auditable and portable if you change providers.
Testing only on clean audio
Quiet rooms and good headsets hide the problems that appear on real phone lines: background noise, poor connections, speakerphones, and strong accents. Teams that skip testing on real telephony conditions overestimate both accuracy and latency gains. Record a sample of genuine calls, with consent, and use them as the benchmark.
Expecting deep reasoning from the conversational layer
Some speech-to-speech models trade reasoning depth for fluency. Routing a complicated billing dispute entirely through the conversational model, with no access to stronger text-based reasoning or tools, produces confident but shallow answers. Design a path for hard requests from the start, whether that is a tool call to a stronger model or a quick transfer to a person.
Real Limitations and Open Questions
Speech-to-speech is not a finished, universally superior replacement for the pipeline approach, and treating it as one leads to disappointment.
- Reasoning depth can suffer. Text is still an efficient medium for complex, multi-step reasoning, and some early speech-to-speech models trade off raw reasoning capability for conversational fluency. A pipeline that routes to a strong text-based LLM for the "thinking" step can sometimes out-perform a unified audio model on tasks that require careful multi-step logic, like working through a complicated billing dispute.
- Tool use and retrieval are less mature. Pipeline architectures have had years to develop patterns for function calling, retrieval-augmented generation, and structured tool use within the LLM stage. Unified speech-to-speech models are still catching up on cleanly integrating these capabilities without breaking the low-latency audio path.
- Evaluation is harder. There's no simple, universally agreed way to score "how good is this speech-to-speech response" the way transcript-based accuracy metrics exist for STT or standard benchmarks exist for text LLMs. Judging tone, appropriateness, and conversational quality at scale is still largely a manual or subjective exercise.
- Fine-tuning and customization tooling is younger. Businesses that want to steer an agent's exact phrasing, escalation behavior, or brand voice have more mature tooling for doing this at the LLM-text layer than at the audio-native layer, where the "prompt" and the "voice" are more entangled.
- Failure modes are less well understood. A pipeline that mis-transcribes a word fails in a legible, traceable way — you can see the wrong word in the transcript. A speech-to-speech model that misunderstands something audio-native may fail in ways that are harder to diagnose after the fact, since there's no intermediate artifact to inspect.
None of this means the pipeline approach is obsolete or that speech-to-speech is immature and to be avoided — it means the two architectures currently suit different priorities, and a lot of production systems will likely stay hybrid for some time, using speech-to-speech for the conversational front end while routing complex reasoning or tool-heavy tasks through more traditional text-based components behind the scenes.
What to Watch Next
The architecture question isn't settled, and a few signals will indicate which way things are moving:
- Whether major voice AI platforms start offering speech-to-speech as a default rather than an add-on option, which would signal confidence in its reliability at scale.
- Whether tool-calling and retrieval capabilities inside speech-to-speech models close the gap with what text-based LLMs already do well, since that's the biggest practical limitation for business use cases.
- Whether transcript-generation-as-a-side-effect becomes standard in speech-to-speech products, resolving the auditability gap without giving up the latency benefits.
- Whether hybrid architectures — speech-to-speech for the conversational surface, text-based LLMs for deep reasoning invoked as a "tool" mid-conversation — become the pragmatic middle ground rather than a stopgap.
- How pricing evolves as more vendors compete in this space, since bundled per-minute audio pricing versus itemized per-stage pricing will shape which architecture makes financial sense for high-volume deployments.
FAQ
What is speech-to-speech voice AI?
Speech-to-speech voice AI refers to models that take spoken audio as input and generate spoken audio as output directly, without converting speech to text and back in separate steps. This contrasts with the traditional STT, LLM, and TTS pipeline, where three distinct systems handle transcription, reasoning, and synthesis one after another. Because a single model hears the original audio, it can respond to tone and timing as well as words.
Is speech-to-speech faster than the STT-LLM-TTS pipeline?
Generally yes, because it removes sequential handoffs between separate systems and can begin generating audio before it has fully processed the caller's turn. The exact advantage depends on the implementation, model hosting location, and network conditions. A well-optimised streaming pipeline can still feel responsive, but collapsing three stages into one model removes the delay that accumulates at each stage boundary, which matters most in fast back-and-forth conversation.
Does speech-to-speech AI produce a transcript?
Not always automatically. Because these models operate natively on audio without a mandatory text step, some systems don't produce a clean transcript unless the vendor provides separate transcription alongside the core model. That matters for compliance, quality assurance, analytics, and dispute handling, where teams need a searchable record of what was said. Confirm how transcripts are generated, and how accurate they are, before committing to a platform for regulated or customer-facing use.
Why does the traditional pipeline lose emotional tone?
Because converting speech to text strips out everything that isn't captured in words — intonation, emphasis, hesitation, sarcasm. The language model only ever sees flat text, so its response is generated without knowledge of how something was actually said, and the resulting text is then read aloud by a separate speech synthesizer that also never heard the original audio.
Can businesses mix both architectures?
Yes, and many production systems likely will for the near term. A common pattern uses a speech-to-speech model for the natural, low-latency conversational front end while routing complex reasoning, tool calls, or retrieval-heavy tasks to a traditional text-based LLM behind the scenes. Another approach keeps the cascaded pipeline for regulated workflows that need exact transcripts and uses speech-to-speech where conversational feel matters most, such as reception or booking calls.
What are the biggest risks of adopting speech-to-speech today?
The main risks are reduced debuggability without a reliable transcript, less mature tool-calling and retrieval support than text-based pipelines, and a more bundled architecture that makes it harder to swap individual components or change vendors later. Pricing can also be less predictable, since audio tokens are often billed differently from text. Teams should pilot with real calls and measure task completion, not just how natural the voice sounds.
Do speech-to-speech models handle interruptions better?
They're generally better positioned to, since a unified model processing continuous audio can model natural turn-taking, backchannels, and overlapping speech more directly than a pipeline that typically waits for a clean end-of-utterance signal before handing text to the language model. In practice, interruption handling still depends on the surrounding telephony and voice activity detection, so test barge-in behaviour on real calls.
How should a business choose between speech-to-speech and a cascaded pipeline?
Start from the call types you need to handle. If conversations are short, emotional, or depend on quick turn-taking, speech-to-speech is worth testing. If calls involve heavy tool use, strict scripts, exact transcripts, or regulatory review, a cascaded pipeline is still the safer default. Run both against a sample of real recordings, compare task completion, latency, and error handling, and keep your business logic outside the voice model so you can switch later.
Conclusion
The STT, LLM, and TTS pipeline became the default because it was buildable from parts that each worked well on their own. Its costs are structural: delay accumulates at every handoff, and tone, hesitation, and emphasis vanish the moment speech becomes text. Speech-to-speech models address both problems by hearing and answering in audio directly.
That doesn't make the cascaded stack obsolete. Text-based pipelines still lead on transcripts, tool calling, retrieval, debuggability, and the freedom to swap components. For many teams the practical answer is a hybrid: a speech-to-speech front end for the conversation, with text-based reasoning and business logic behind it.
Treat vendor latency and naturalness claims as starting points. Measure with your own calls, your own noise conditions, and your own escalation rules, and check how transcripts and audit trails will work before anything goes live.
If you're deciding which architecture fits your voice agent, our voice AI team can help you prototype both approaches and compare them on real conversations.