Say something to most "AI voice agents" today and here is what actually happens: your words get transcribed to text, the text gets sent to a language model, the model's text response gets synthesized back into audio, and the audio plays. Three separate systems, two format conversions, and a chain of network calls — all before you hear a reply. It works, but it has a ceiling, and that ceiling is becoming obvious as voice interfaces move from novelty demos into real customer-facing products.
Speech-to-speech models remove the middle conversions. Instead of transcribing, reasoning in text, and re-synthesizing, a single model processes audio in and produces audio out directly. No intermediate transcript. No separate text-to-speech step re-reading a script it didn't originally "hear." This is a structural change to how voice AI is built, not just a speed improvement, and it's worth understanding both what it fixes and what it doesn't.
The Three-Stage Stack and Why It Was the Default
For years, the practical way to build a voice agent was to stitch together three specialized systems, each doing one job well:
- Speech-to-text (STT) converts the caller's audio into a text transcript. Systems like Whisper-class models handle this — accurately, but not instantly, and with occasional errors on accents, crosstalk, or background noise.
- Large language model (LLM) reads the transcript and generates a text response, applying whatever reasoning, retrieval, or business logic the application needs.
- Text-to-speech (TTS) converts that generated text back into audio, using a voice model that has to guess at intonation from punctuation and word choice alone.
This pipeline made sense because each component was independently improvable. Better transcription models could be swapped in without touching the LLM. Better voices could be swapped in without retraining anything upstream. It's a classic separation-of-concerns architecture, and for a long time it was the only viable way to get a voice interface on top of a text-native language model.
But separation of concerns has a cost when the thing you're optimizing is a live conversation. Every stage boundary is a place where information gets lost and where latency accumulates.
What Gets Lost in Translation
The transcript is the chokepoint. Once speech becomes text, everything that isn't captured in words is gone — tone of voice, hesitation, a nervous laugh, emphasis on a particular word, whether "sure" was said enthusiastically or sarcastically. The LLM never hears any of that; it only sees "sure." Its response is generated blind to how the thing was actually said, and then that response is re-synthesized by a TTS model that also never heard the original — it only receives the LLM's flat text output.
The result is a pipeline that can produce grammatically correct, contextually reasonable responses that still feel emotionally tone-deaf, because the two points where emotional information exists — the caller's voice and the agent's voice — are separated by a text bottleneck that erases it in between.
Where the Latency Comes From
Each stage in the pipeline adds a discrete chunk of processing time, and those chunks are typically sequential rather than overlapping:
| Stage | What it does | Typical latency contribution |
|---|---|---|
| STT | Transcribe audio to text | Waits for speech end, then processes |
| LLM | Generate a text response | Token generation, often after full transcript arrives |
| TTS | Synthesize text to audio | Waits for full or partial text, then renders audio |
| Network hops | Data moves between 2-3 separate services | Added per hop, compounds with each stage |
Individually, each stage might only add a few hundred milliseconds. Stacked in sequence, with network round-trips between often-separate services, the cumulative delay is what makes STT→LLM→TTS voice agents feel like a very fast phone tree rather than a conversation. Humans notice gaps above roughly 200-300 milliseconds in natural dialogue; three-stage pipelines routinely land well past a second, especially once retrieval or tool calls are added to the LLM step.
What Speech-to-Speech Actually Does Differently
A speech-to-speech model is trained to take audio as input and produce audio as output within a single model, without a mandatory text transcript as an intermediate representation. Architecturally, this usually means the model operates on audio tokens (or continuous audio representations) alongside — or instead of — text tokens, and it's trained end-to-end on the task of "hear this, respond with speech," rather than "transcribe, reason, then speak" as three separately optimized objectives.
This has a few direct consequences:
- Prosody and tone can be preserved. Because the model processes the actual acoustic signal rather than a stripped-down transcript, it has access to information about how something was said, not just what was said. It can, in principle, respond faster to an urgent tone or pick up on the difference between a genuine question and a rhetorical one.
- Latency drops because stages collapse. With one model handling the full path, there's no waiting for a transcript to finalize before language generation starts, and no waiting for a text response to finalize before audio synthesis starts. Many speech-to-speech systems can begin producing audio output while still processing the tail end of the input, enabling much tighter turn-taking.
- Interruptions and overlap become more natural to handle. Real conversation involves interruptions, "mm-hmm" backchannels, and talking over the end of someone's sentence. A unified audio model is better positioned to model this than a pipeline where the STT step is generally waiting for a clean end-of-utterance signal before it hands anything to the LLM.
- The agent's own voice can carry appropriate emotion. Rather than a TTS model reading a flat transcript with generic inflection, a speech-to-speech model can, in theory, generate a response whose delivery matches its content — because the "speaking" and "deciding what to say" happen inside the same process rather than as two independent guesses.
How the Two Approaches Actually Compare
| Dimension | STT → LLM → TTS pipeline | Speech-to-speech model |
|---|---|---|
| Components | 3+ separate systems | 1 unified model |
| Emotional/tonal signal | Lost at transcription | Potentially preserved end-to-end |
| Latency profile | Sequential, additive | Overlapping, generally lower |
| Modularity | Easy to swap individual components | Harder to swap parts independently |
| Debuggability | Transcript gives a clear audit trail | Audio-native reasoning is harder to inspect |
| Maturity | Well-established, widely deployed | Newer, fewer production battle scars |
| Cost predictability | Each stage billed/optimized separately | Bundled, less granular control |
| Language/accent robustness | Inherits STT's known strengths and gaps | Still maturing across languages |
Neither architecture is strictly better in every dimension — the table above is a set of trade-offs, not a scoreboard, and which side wins depends heavily on what a given application actually needs.
Why This Matters Right Now
Voice interfaces are having a moment precisely because latency and naturalness were the two biggest reasons they felt unusable for anything beyond simple IVR-style tasks. A support line that takes 1.5 seconds to respond to every utterance, and that responds to "I'm really frustrated" in the same flat tone it uses for "what are your hours," doesn't feel like talking to something competent — it feels like talking to a slow, tone-deaf script reader with a search engine attached.
The shift toward unified speech-to-speech models is happening because the pipeline approach hit a wall that better prompting or bigger LLMs couldn't fix. The latency floor of a three-stage architecture is set by physics and network topology, not by model quality — you can have the smartest LLM in the world and still lose the conversation to a two-second pause before it speaks. Collapsing the stages is the only way to meaningfully move that floor, which is why the industry's attention has shifted toward audio-native models rather than continuing to optimize each pipeline stage in isolation.
This matters especially for use cases where the interaction is inherently high-stakes or emotionally loaded — a caller reporting a problem, cancelling a service, or asking about something urgent. In those situations, a half-second of extra silence or a mismatched tone reads as incompetence in a way it wouldn't in, say, a typed chat interface where pacing expectations are different.
Practical Implications for Businesses Building Voice Agents
If you're evaluating voice AI vendors or planning to build a voice agent, the STT→LLM→TTS-versus-speech-to-speech distinction has concrete effects on what you should expect and what to ask about.
What Changes for Buyers and Builders
- Latency claims need scrutiny. "Sub-second response time" means something very different depending on architecture. Ask specifically what's being measured — time to first audio byte, time to complete response, or something else — and whether that number holds under realistic network conditions, not just in a demo booth on the vendor's own infrastructure.
- Debugging gets harder without a transcript. Pipeline architectures give you a text transcript almost for free, which makes it easy to review what the agent "heard" and log conversations for QA. Speech-to-speech systems may not produce a clean intermediate transcript at all, which complicates auditing, compliance logging, and post-call analysis unless the vendor has built separate tooling for it.
- Component-level customization gets harder. If you wanted a specific TTS voice for brand consistency, or a specific STT model tuned for your industry's vocabulary, a pipeline lets you swap that one piece. A unified speech-to-speech model is generally a package deal — you get its voice, its behavior, and its accent handling as a bundle.
- Vendor lock-in risk changes shape. Pipelines can be re-assembled from different vendors' components. Speech-to-speech models are typically proprietary end-to-end systems, so switching providers later may mean a more complete rebuild rather than swapping one stage.
- Cost structure shifts. Pipeline costs are usually itemized per stage, which makes it easier to optimize spend on the specific bottleneck. Bundled speech-to-speech pricing is often simpler to reason about per minute but harder to optimize piece by piece.
A Practical Decision Checklist
- Does the use case need sub-second, emotionally attuned responses? High-stakes support, sales, or healthcare triage calls benefit most from speech-to-speech; a scheduling bot that mostly confirms dates and times may not need it.
- Do you need a clean transcript for compliance or QA? If regulatory logging requirements demand verbatim, auditable transcripts, confirm the vendor produces one — some speech-to-speech systems bolt on transcription as a side process specifically for this reason.
- How many languages and accents does your caller base span? Pipeline STT components generally have a longer track record across diverse languages; verify a speech-to-speech vendor's real coverage rather than assuming parity.
- What's your tolerance for vendor lock-in? If you expect to change voice AI vendors within a year or two, a modular pipeline is easier to migrate incrementally.
- Can you test under real network conditions, not a demo? Ask for a trial on your actual telephony or WebRTC setup, not a curated sandbox, since network jitter affects the two architectures differently.
Real Limitations and Open Questions
Speech-to-speech is not a finished, universally superior replacement for the pipeline approach, and treating it as one leads to disappointment.
- Reasoning depth can suffer. Text is still an efficient medium for complex, multi-step reasoning, and some early speech-to-speech models trade off raw reasoning capability for conversational fluency. A pipeline that routes to a strong text-based LLM for the "thinking" step can sometimes out-perform a unified audio model on tasks that require careful multi-step logic, like working through a complicated billing dispute.
- Tool use and retrieval are less mature. Pipeline architectures have had years to develop patterns for function calling, retrieval-augmented generation, and structured tool use within the LLM stage. Unified speech-to-speech models are still catching up on cleanly integrating these capabilities without breaking the low-latency audio path.
- Evaluation is harder. There's no simple, universally agreed way to score "how good is this speech-to-speech response" the way transcript-based accuracy metrics exist for STT or standard benchmarks exist for text LLMs. Judging tone, appropriateness, and conversational quality at scale is still largely a manual or subjective exercise.
- Fine-tuning and customization tooling is younger. Businesses that want to steer an agent's exact phrasing, escalation behavior, or brand voice have more mature tooling for doing this at the LLM-text layer than at the audio-native layer, where the "prompt" and the "voice" are more entangled.
- Failure modes are less well understood. A pipeline that mis-transcribes a word fails in a legible, traceable way — you can see the wrong word in the transcript. A speech-to-speech model that misunderstands something audio-native may fail in ways that are harder to diagnose after the fact, since there's no intermediate artifact to inspect.
None of this means the pipeline approach is obsolete or that speech-to-speech is immature and to be avoided — it means the two architectures currently suit different priorities, and a lot of production systems will likely stay hybrid for some time, using speech-to-speech for the conversational front end while routing complex reasoning or tool-heavy tasks through more traditional text-based components behind the scenes.
What to Watch Next
The architecture question isn't settled, and a few signals will indicate which way things are moving:
- Whether major voice AI platforms start offering speech-to-speech as a default rather than an add-on option, which would signal confidence in its reliability at scale.
- Whether tool-calling and retrieval capabilities inside speech-to-speech models close the gap with what text-based LLMs already do well, since that's the biggest practical limitation for business use cases.
- Whether transcript-generation-as-a-side-effect becomes standard in speech-to-speech products, resolving the auditability gap without giving up the latency benefits.
- Whether hybrid architectures — speech-to-speech for the conversational surface, text-based LLMs for deep reasoning invoked as a "tool" mid-conversation — become the pragmatic middle ground rather than a stopgap.
- How pricing evolves as more vendors compete in this space, since bundled per-minute audio pricing versus itemized per-stage pricing will shape which architecture makes financial sense for high-volume deployments.
FAQ
What is speech-to-speech voice AI?
Speech-to-speech voice AI refers to models that take spoken audio as input and generate spoken audio as output directly, without converting speech to text and back in separate steps. This contrasts with the traditional STT→LLM→TTS pipeline, where three distinct systems handle transcription, reasoning, and synthesis separately.
Is speech-to-speech faster than the STT-LLM-TTS pipeline?
Generally yes, because it eliminates sequential handoffs between separate systems and can begin generating audio output before fully processing the input. The exact latency advantage depends on the specific implementation and network conditions, but collapsing three stages into one model removes accumulated delay at each stage boundary.
Does speech-to-speech AI produce a transcript?
Not always automatically. Because these models can operate natively on audio without a mandatory text intermediate, some speech-to-speech systems don't produce a clean transcript unless the vendor has built separate transcription tooling alongside the core model, which matters for compliance and QA use cases.
Why does the traditional pipeline lose emotional tone?
Because converting speech to text strips out everything that isn't captured in words — intonation, emphasis, hesitation, sarcasm. The language model only ever sees flat text, so its response is generated without knowledge of how something was actually said, and the resulting text is then read aloud by a separate speech synthesizer that also never heard the original audio.
Can businesses mix both architectures?
Yes, and many production systems likely will for the near term. A common pattern uses a speech-to-speech model for the natural, low-latency conversational front end while routing complex reasoning, tool calls, or retrieval-heavy tasks to a traditional text-based LLM behind the scenes.
What are the biggest risks of adopting speech-to-speech today?
The main risks are reduced debuggability without a transcript, less mature tool-calling and retrieval support compared to text-based LLM pipelines, and a more bundled architecture that makes it harder to swap individual components or switch vendors later.
Do speech-to-speech models handle interruptions better?
They're generally better positioned to, since a unified model processing continuous audio can model natural turn-taking, backchannels, and overlapping speech more directly than a pipeline that typically waits for a clean end-of-utterance signal before handing text to the language model.
Teams evaluating this shift for a real deployment can get hands-on architecture guidance from Woyce Technologies.