Building a voice agent usually means choosing between an easy, fully-hosted realtime API you don't control, or stitching together separate speech, language, and synthesis models yourself. Hugging Face's speech-to-speech is built to close that gap: an open-source pipeline that speaks the OpenAI Realtime API protocol so any client built for it works unmodified, while every component underneath — the transcription model, the language model, the voice synthesizer — is swappable, self-hostable, and can run fully offline.
Despite the Name, It's a Modular Pipeline, Not a Single-Model Architecture
This is worth clarifying directly, because the naming is easy to misread if you already know the term: "speech-to-speech" in the voice AI field increasingly refers to a specific architecture — a single model that processes audio in and produces audio out directly, with no intermediate text transcript. Hugging Face's project, despite the name, is the traditional four-stage cascade: Voice Activity Detection, then Speech-to-Text, then an LLM, then Text-to-Speech, with each stage running in its own thread connected by queues. That's not a criticism — the modular approach has real advantages the single-model approach doesn't (every component is independently swappable, debuggable, and upgradable) — but it's a meaningfully different architecture than what the term "speech-to-speech" usually implies elsewhere, and worth knowing before you assume this is a true unified audio model.
OpenAI Realtime Compatibility Is the Actual Product Decision
The pipeline exposes itself as an OpenAI Realtime-compatible WebSocket (and, as of the current release, WebRTC) API. That single design choice is what makes the project genuinely practical rather than just another voice pipeline demo: any client already built against OpenAI's Realtime protocol can point at a self-hosted speech-to-speech server instead, without rewriting the client. The project's own demo shows exactly that — an OpenAI Realtime client switching its endpoint from hosted OpenAI to a self-hosted server live, with no other changes. For a team that's already built a product against OpenAI's Realtime API and wants a self-hosted or fully local fallback path, that compatibility is the whole reason to look at this project first.
Every Stage Is Genuinely Swappable
The four-stage cascade — VAD, STT, LLM, TTS — has multiple interchangeable backends at every stage, selected via CLI flags, with a stated focus on models available through Transformers and the Hugging Face Hub. The default install alone covers a complete realtime path: Parakeet TDT for speech-to-text, an OpenAI-compatible API slot for the language model, and Qwen3-TTS for speech output. Because the LLM slot speaks OpenAI-compatible protocols specifically, it can point at a hosted provider, at Hugging Face's own Inference Providers, or at a local vLLM or llama.cpp server — meaning the exact same pipeline code supports everything from a fully hosted setup to a fully local, fully offline one, just by changing which endpoint the LLM slot points at.
Smart Turn-Taking Is the Detail That Actually Matters for Feel
A voice agent's most noticeable quality problem usually isn't transcription accuracy — it's turn-taking. Cutting a user off mid-thought, or leaving an awkward silence after they've clearly finished speaking, breaks the conversational feel faster than almost anything else. The current release enables a quantized Smart Turn model by default specifically to distinguish a completed turn from a mid-thought pause, running alongside speculative STT and LLM work rather than waiting for silence alone (the traditional Silero-only approach, still available as a fallback). This is exactly the kind of unglamorous latency-and-timing engineering that separates a voice agent that feels natural from one that technically works but feels robotic.
It's Not a Research Toy — It Runs in Production
The project states directly that this pipeline runs in production as the conversation backend for thousands of Reachy Mini robots — a real, shipped hardware product, not just a GitHub demo. That's a meaningful credibility signal for anyone evaluating whether an open-source voice pipeline is actually production-hardened: the recent release notes include exactly the kind of reliability work you'd expect from something running at real scale — fixing stuck pipeline units during session teardown, preventing stale teardown signals from releasing the wrong session, and locking around Metal command-buffer operations on Apple Silicon to prevent crashes when speech recognition overlaps other GPU work.
Getting a Server Running
The quickstart is genuinely three commands: pip install speech-to-speech, set an OPENAI_API_KEY, then speech-to-speech serve. That boots an OpenAI Realtime-compatible server at ws://localhost:8765/v1/realtime using Parakeet TDT for STT and Qwen3-TTS for speech output by default, with the LLM slot pointed at whatever OpenAI-compatible endpoint you configured. A second terminal running speech-to-speech talk --url ws://127.0.0.1:8765/v1/realtime connects the packaged microphone/speaker client, or speech-to-speech local composes both in one process over loopback for a single-command trial. Swapping in a fully local LLM is a matter of pointing the same server at a different endpoint — running llama-server with a GGUF model locally and passing its URL via --responses_api_base_url gets you an entirely offline pipeline with no code changes, just different CLI flags. Requires Python 3.10+, and Docker Compose support (with the NVIDIA Container Toolkit for GPU access) starts both a llama.cpp server and the Realtime server together for a two-port, fully self-hosted setup out of the box.
Backend Choices Go Deeper Than the Big Four
Beyond the default STT, LLM, and TTS choices, each stage has more backends available as pip extras than the core pitch suggests. STT options include Whisper through Transformers, Faster Whisper, Lightning Whisper MLX (Apple Silicon specifically), MLX Audio Whisper, and Paraformer through FunASR for Chinese-oriented transcription. TTS options extend to Kokoro-82M, Pocket TTS from Kyutai Labs (with streaming voice cloning across eight preset voices), ChatTTS for English and Chinese, and MMS TTS for broad multilingual coverage. That range matters for language coverage specifically: Parakeet TDT's default language set covers 25 European languages, so a team building for Chinese, Japanese, or another language outside that set needs to deliberately pick a different STT and TTS pairing rather than assume the defaults handle it — the pipeline supports both single-language operation and automatic per-utterance language detection via --language auto.
Practical Implications
- If you already have a client built for OpenAI's Realtime API, this is close to a drop-in self-hosted or local alternative — the protocol compatibility is specifically designed to make that switch require minimal client-side changes.
- The LLM proxy feature needs deliberate network placement. The project is explicit that its optional remote-LLM proxy doesn't authenticate or throttle requests by design, so any standalone deployment using it needs to sit behind a trusted network or an authenticated gateway — not exposed directly to the internet.
- For fully offline deployments, the combination of local STT, a local LLM server (llama.cpp or vLLM), and local TTS gives a genuinely complete, non-cloud-dependent voice stack — worth evaluating specifically against self-hosting cost versus a hosted realtime API.
- Don't assume "speech-to-speech" in the name means single-model latency characteristics. The modular cascade has different latency and quality trade-offs than a true unified audio model — evaluate it on its own architecture's merits, not assumptions carried over from the newer single-model approach.
Practical Takeaway
Hugging Face's speech-to-speech is a well-engineered, production-tested answer to a specific need: an open, self-hostable, protocol-compatible alternative to a hosted realtime voice API, built with the swappable-components philosophy that makes Hugging Face's own ecosystem useful in the first place. For teams building voice-first products that want a self-hosted or fully local option without abandoning OpenAI Realtime API compatibility, it's a strong starting point — just go in clear-eyed about the modular architecture versus the newer single-model speech-to-speech approach, since the name alone doesn't tell you which one you're getting.
Teams building voice AI products — hosted, self-hosted, or fully local — can get hands-on architecture help from Woyce Technologies.
FAQ
What is Hugging Face's speech-to-speech?
It's an open-source, modular voice agent pipeline — Voice Activity Detection, Speech-to-Text, an LLM, and Text-to-Speech — exposed through an OpenAI Realtime API-compatible WebSocket and WebRTC interface, with every component independently swappable.
Is this a true single-model speech-to-speech architecture?
No, despite the name. It's a four-stage cascade pipeline, not a unified model that processes audio directly in and out. That's a different architecture than what "speech-to-speech" typically refers to elsewhere in voice AI, though it offers different trade-offs — namely, independently swappable and debuggable components.
Can I run Hugging Face's speech-to-speech fully offline?
Yes — with local STT (like Parakeet TDT), a local LLM server (llama.cpp or vLLM), and local TTS (like Qwen3-TTS), the full pipeline can run without any cloud dependency.
Does it work with existing OpenAI Realtime API clients?
Yes — it implements the OpenAI Realtime API protocol over both WebSocket and WebRTC, so a client already built against that protocol can typically point at a self-hosted server with minimal changes.
Is this project just a demo, or does it run in production?
It runs in production as the conversation backend for thousands of Reachy Mini robots, a real shipped hardware product — not just a research demo.
Is Hugging Face's speech-to-speech free to use?
Yes, it's Apache 2.0 licensed and open source, installable via pip, with the underlying models it uses (Parakeet TDT, Qwen3-TTS, and others) available through the Hugging Face Hub.
What languages does it support?
It depends entirely on which STT and TTS backends you pick rather than the pipeline itself. The default Parakeet TDT covers 25 European languages; swapping to Whisper variants or Paraformer extends that to broader multilingual or Chinese-oriented coverage respectively, and Qwen3-TTS handles multiple output languages with automatic language selection by default.
Does it support WebRTC, or only WebSocket?
Both. WebSocket has been supported from early on; WebRTC support (installed via the webrtc pip extra) was added for SDP negotiation and RTP audio, and both transports share the same underlying pipeline pool, event dispatch, and interruption behavior.
Can it feed audio straight to an audio-capable LLM instead of transcribing first?
Yes, with --stt none and the Chat Completions backend pointed at a model that explicitly accepts audio input, such as OpenAI's audio-capable models. This bypass isn't available on the Responses API backend, and the CLI's default model doesn't accept audio, so this mode requires deliberately picking a compatible model and backend combination.
How is turn-taking handled without cutting the user off mid-sentence?
A quantized Smart Turn model, enabled by default, scores whether Silero VAD's detected pause is a completed turn or a mid-thought gap, running speculative STT and LLM work in the background while it decides — complete turns start processing immediately, incomplete ones wait briefly before starting and are held longer before their output is released.