# speech-to-speech: the OpenAI Realtime API, reimplemented as four swappable parts

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/speech-to-speech
> date: 2026-08-14
> tags: speech, voice-agents, open-source, realtime, latency, explainer
[speech-to-speech](https://github.com/huggingface/speech-to-speech) is Hugging Face's voice-agent pipeline: VAD → STT → LLM → TTS, each stage in its own thread, connected by queues. Apache 2.0, on PyPI, first commit 2024-08-07 and still being merged the day I looked.

Two things make it worth more than a glance. The first is a compatibility decision. The second is a latency trick that I think is the actual contribution.

## The compatibility decision

The server speaks the **core OpenAI Realtime GA event set** over WebSocket and WebRTC. Not a similar protocol — the same one, at `/v1/realtime`, such that the official OpenAI client connects to it by changing a URL:

```python
client = OpenAI(
    base_url="http://localhost:8765/v1",
    websocket_base_url="ws://localhost:8765/v1",
    api_key="not-needed",
)
with client.realtime.connect(model="local") as conn:
    ...
```

<Figure
  src="/articles/speech-to-speech/fig1.gif"
  alt="Animation showing an OpenAI Realtime client's endpoint being changed from the hosted OpenAI service to a self-hosted speech-to-speech server, with the conversation continuing to work after the swap."
  caption="The whole pitch in one edit: same client, same events, different endpoint. (huggingface/speech-to-speech, docs/assets.)"
/>

The repo is careful about the size of this claim, and I want to repeat its wording rather than improve on it:

> This is a tested core subset, not a claim of full OpenAI Realtime API equivalence.

What is implemented inbound: `input_audio_buffer.append`, `session.update`, `conversation.item.create`, `conversation.item.truncate`, `response.create`, `response.cancel`. Outbound: speech start/stop, streaming transcription, audio deltas, tool calls, `response.done`. CI connects pinned `@openai/agents` `RealtimeSession` instances through the SDK's own WebSocket *and* WebRTC transports — so the compatibility claim is tested against the real client library, not against a hand-written mock.

That is the difference between "OpenAI-compatible" as a marketing word and as an engineering commitment.

<Figure
  src="/articles/speech-to-speech/fig2.png"
  alt="Architecture flowchart: a packaged local audio client, an external WebSocket client and a WebRTC client all connect to realtime transports, which reach a RealtimeService and RuntimeConfig, which drive a pipeline of VAD, STT, TranscriptionNotifier, LLM, LMOutputProcessor and TTS threads."
  caption="A FastAPI server in front of a queue-backed pipeline; each session claims its own PipelineUnit. (huggingface/speech-to-speech, realtime engine README — rendered from the repo's mermaid source.)"
/>

## Ninety pipelines

<BackendMatrix />

Six STT backends, three LLM backends, five TTS backends, one VAD. The defaults are Parakeet TDT for transcription and Qwen3-TTS for output — both local — with the LLM slot pointed at anything speaking OpenAI protocols.

That last choice is worth pulling apart, because "OpenAI-compatible API" sounds like a dependency and is not one. Point it at `llama-server` on localhost:

```bash
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -np 2 -c 65536 -fa on --swa-full

speech-to-speech serve \
    --model_name "ggml-org/gemma-4-E4B-it-GGUF" \
    --responses_api_base_url "http://127.0.0.1:8080/v1" \
    --responses_api_api_key ""
```

Now the whole pipeline is local and the LLM is still reached over HTTP. Keeping the model behind a protocol boundary rather than in-process is what makes the slot genuinely swappable — the pipeline never learns which model it is talking to.

For fully disconnected operation, run the exact configuration once online to warm the caches, then set `HF_HUB_OFFLINE=1`. The repo is specific that this covers STT, LLM, TTS, Silero VAD, NLTK *and* Smart Turn assets — the kind of list you only write after being caught out by one of them.

There is also a `--stt none` mode that skips transcription entirely and hands each VAD-segmented audio chunk straight to an audio-input model over `/v1/chat/completions`. The README is blunt that this needs a model that actually accepts audio, and that the default `gpt-5.4-mini` does not.

## Smart Turn is the interesting part

<SmartTurn />

Silero VAD tells you *that* speech stopped. It cannot tell you whether the person was **finished**. That gap is why voice agents interrupt people who paused to think.

[Smart Turn v3.2](https://huggingface.co/pipecat-ai/smart-turn-v3) classifies the turn using content and prosody, and speech-to-speech wires it in speculatively rather than as a gate:

- **Complete turns** start STT and the LLM immediately, with `--speculative_reopen_ms` (800 ms) before output is committed.
- **Incomplete turns** wait `--smart_turn_incomplete_delay_ms` (600 ms) before spending anything, and their output stays gated by `--smart_turn_max_wait_ms` (2 s).
- **If speech resumes during either delay**, the turn is reopened as a newer revision, the accumulated audio is re-emitted, and work from the previous revision is discarded *before it reaches the user*.

That third rule is what makes the first two safe. Speculation is only free if the wrong guesses are invisible, and revision numbering is the mechanism that makes them invisible. You spend tokens you might throw away in exchange for latency you cannot otherwise reclaim — a reasonable trade in a pipeline where every millisecond between "user stopped" and "audio starts" is audible.

It ships enabled by default, as a quantized CPU ONNX checkpoint, so the cost of running it is not a GPU.

## The part that gives it weight

> This pipeline runs in production as the conversation backend for thousands of [Reachy Mini](https://huggingface.co/blog/reachy-mini) robots.

Voice-agent demos are cheap and voice agents that hold up in a room with background noise are not. A deployed fleet is the only evidence that separates the two, and it is the reason to read this repo rather than one of the dozens of similar cascades.

## What to be aware of

**The Realtime surface is a subset**, and the repo says so. If your client depends on an event outside the tested set, it will not work, and "OpenAI-compatible" will have been true and useless simultaneously.

**No latency numbers.** For a project whose headline is "low-latency," there is no published end-to-end figure — no time-to-first-audio, no comparison against hosted OpenAI Realtime, on any hardware. The architecture is clearly built for latency; how much it achieves is unmeasured in public.

**Installation has sharp edges.** The Qwen3-TTS GGML backend's default PyPI wheel targets CUDA 12.8, and the README carries a table of alternate wheels for CUDA 13.x, 12.4, and CPU. That is honest documentation of a real problem, and also a sign of how much platform-specific machinery sits under `pip install speech-to-speech`.

**Some components moved to `archive/`** — Moonshine STT, MeloTTS, Parler TTS. Worth knowing before you build on a backend that is on its way out.

The design I would steal is the one at the boundary: implement someone else's protocol exactly enough to be tested against their client, then make everything behind it yours. It converts a hosted API from a dependency into an interface.