2026-08-14 · 5 min · speech · voice-agents · open-source · realtime · latency · explainer
speech-to-speech is Hugging Face's voice-agent pipeline: VAD → STT → LLM → TTS, each stage in its own thread, connected by queues. Apache 2.0, on PyPI, first commit 2024-08-07 and still being merged the day I looked.
Two things make it worth more than a glance. The first is a compatibility decision. The second is a latency trick that I think is the actual contribution.
The compatibility decision
The server speaks the core OpenAI Realtime GA event set over WebSocket and WebRTC. Not a similar protocol — the same one, at /v1/realtime, such that the official OpenAI client connects to it by changing a URL:
client = OpenAI(
base_url="http://localhost:8765/v1",
websocket_base_url="ws://localhost:8765/v1",
api_key="not-needed",
)
with client.realtime.connect(model="local") as conn:
...
The repo is careful about the size of this claim, and I want to repeat its wording rather than improve on it:
This is a tested core subset, not a claim of full OpenAI Realtime API equivalence.
What is implemented inbound: input_audio_buffer.append, session.update, conversation.item.create, conversation.item.truncate, response.create, response.cancel. Outbound: speech start/stop, streaming transcription, audio deltas, tool calls, response.done. CI connects pinned @openai/agents RealtimeSession instances through the SDK's own WebSocket and WebRTC transports — so the compatibility claim is tested against the real client library, not against a hand-written mock.
That is the difference between "OpenAI-compatible" as a marketing word and as an engineering commitment.
Ninety pipelines
Ninety combinations, and only one slot can leave the machine. The LLM row is the interesting one: the “OpenAI-compatible API” option is remote only in the sense that it speaks HTTP — point it at a llama-server on localhost and the whole pipeline is local while still being an API client. That is a different design from embedding a model in-process, and it is why swapping the LLM does not mean rewriting the pipeline.
Six STT backends, three LLM backends, five TTS backends, one VAD. The defaults are Parakeet TDT for transcription and Qwen3-TTS for output — both local — with the LLM slot pointed at anything speaking OpenAI protocols.
That last choice is worth pulling apart, because "OpenAI-compatible API" sounds like a dependency and is not one. Point it at llama-server on localhost:
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full
speech-to-speech serve \
--model_name "ggml-org/gemma-4-E4B-it-GGUF" \
--responses_api_base_url "http://127.0.0.1:8080/v1" \
--responses_api_api_key ""Now the whole pipeline is local and the LLM is still reached over HTTP. Keeping the model behind a protocol boundary rather than in-process is what makes the slot genuinely swappable — the pipeline never learns which model it is talking to.
For fully disconnected operation, run the exact configuration once online to warm the caches, then set HF_HUB_OFFLINE=1. The repo is specific that this covers STT, LLM, TTS, Silero VAD, NLTK and Smart Turn assets — the kind of list you only write after being caught out by one of them.
There is also a --stt none mode that skips transcription entirely and hands each VAD-segmented audio chunk straight to an audio-input model over /v1/chat/completions. The README is blunt that this needs a model that actually accepts audio, and that the default gpt-5.4-mini does not.
Smart Turn is the interesting part
Silero decides that speech stopped; Smart Turn decides whether the person was finished, using content and prosody rather than silence alone. That distinction is most of what separates a voice agent that interrupts you from one that does not. The pipeline then does something slightly greedy: it starts STT and the LLM before it is sure, and relies on revision numbering to throw the work away if the guess was wrong. Latency you can reclaim, tokens you can waste — but only if a discarded revision can never be heard.
Silero VAD tells you that speech stopped. It cannot tell you whether the person was finished. That gap is why voice agents interrupt people who paused to think.
Smart Turn v3.2 classifies the turn using content and prosody, and speech-to-speech wires it in speculatively rather than as a gate:
- Complete turns start STT and the LLM immediately, with
--speculative_reopen_ms(800 ms) before output is committed. - Incomplete turns wait
--smart_turn_incomplete_delay_ms(600 ms) before spending anything, and their output stays gated by--smart_turn_max_wait_ms(2 s). - If speech resumes during either delay, the turn is reopened as a newer revision, the accumulated audio is re-emitted, and work from the previous revision is discarded before it reaches the user.
That third rule is what makes the first two safe. Speculation is only free if the wrong guesses are invisible, and revision numbering is the mechanism that makes them invisible. You spend tokens you might throw away in exchange for latency you cannot otherwise reclaim — a reasonable trade in a pipeline where every millisecond between "user stopped" and "audio starts" is audible.
It ships enabled by default, as a quantized CPU ONNX checkpoint, so the cost of running it is not a GPU.
The part that gives it weight
This pipeline runs in production as the conversation backend for thousands of Reachy Mini robots.
Voice-agent demos are cheap and voice agents that hold up in a room with background noise are not. A deployed fleet is the only evidence that separates the two, and it is the reason to read this repo rather than one of the dozens of similar cascades.
What to be aware of
The Realtime surface is a subset, and the repo says so. If your client depends on an event outside the tested set, it will not work, and "OpenAI-compatible" will have been true and useless simultaneously.
No latency numbers. For a project whose headline is "low-latency," there is no published end-to-end figure — no time-to-first-audio, no comparison against hosted OpenAI Realtime, on any hardware. The architecture is clearly built for latency; how much it achieves is unmeasured in public.
Installation has sharp edges. The Qwen3-TTS GGML backend's default PyPI wheel targets CUDA 12.8, and the README carries a table of alternate wheels for CUDA 13.x, 12.4, and CPU. That is honest documentation of a real problem, and also a sign of how much platform-specific machinery sits under pip install speech-to-speech.
Some components moved to archive/ — Moonshine STT, MeloTTS, Parler TTS. Worth knowing before you build on a backend that is on its way out.
The design I would steal is the one at the boundary: implement someone else's protocol exactly enough to be tested against their client, then make everything behind it yours. It converts a hosted API from a dependency into an interface.