2026-07-20 · 4 min · audio · tts · speech · generative · explainer
Modern text-to-speech has mostly converged on a shape: a language model predicts discrete speech tokens from text, and a generative decoder turns those tokens back into a waveform. Qwen-Audio-3.0-TTS — Alibaba's latest, in the CosyVoice lineage — runs that shape hard and adds the things that make a TTS model actually usable: instruction control, inline non-verbal events, 16 languages and 20 Chinese dialect regions, one-pass long-form synthesis, and 48 kHz output. It currently sits at #1 on the Artificial Analysis Text-to-Speech leaderboard. Here's how it's built.
The stack
Two models do the heavy lifting. An autoregressive LM predicts a sequence of discrete speech tokens from the text (and, for zero-shot cloning, a reference clip). A flow-matching decoder turns those tokens into a mel-spectrogram. A vocoder reconstructs and super-resolves the waveform to 48 kHz. Fronting all of it is a 12.5 Hz supervised speech tokenizer — the piece that quietly sets the model's latency. Click through the stages:
A supervised speech tokenizer running at 12.5 frames per second. Few tokens per second of audio means the autoregressive LM takes far fewer decode steps, cutting latency while still carrying content and speaker identity.
Two models, trained in five stages: the LM and the flow-matching decoder are pretrained independently, jointly annealed on high-quality data, then each gets its own reinforcement-learning pass (plus a robustness stage for the decoder). The split is why you can instruct what to say and how separately — content lives in the LM, voice and prosody in the decoder.
Splitting content (the LM) from voice-and-prosody (the flow-matching decoder) is what makes the model instructable: you can change what is said and how it's said through different parts of the system. It's also why the model is robust to a bad reference — a noisy or reverberant prompt still conditions the LM, and there's no explicit denoising step to break.
Why 12.5 Hz
The tokenizer's frame rate is the single most consequential number in an autoregressive TTS system, because the LM emits one token per step and the steps are serial. Fewer tokens per second of audio means fewer decode steps means less latency. Most neural speech codecs sit at 25–75 Hz; Qwen's supervised tokenizer runs at 12.5 Hz. Drag it and watch the step count move:
Every token is one autoregressive step, and steps are serial — so halving the frame rate roughly halves the time-to-audio. Most neural speech codecs sit at 25–75 Hz; Qwen's supervised tokenizer at 12.5 Hz keeps the token stream as short as it gets here while still preserving content and speaker — the whole point of making the tokenizer supervised rather than a raw reconstruction codec.
The word doing the work is supervised. A raw reconstruction codec at 12.5 Hz would throw away too much to sound good; a supervised tokenizer is trained to keep exactly the content and speaker information the LM needs, so it can afford the low frame rate. Short token stream, fast decode, intact voice — that's the trade the tokenizer is engineered to win.
Say what, and how
Controllability in most TTS models means "pick a preset voice." Qwen splits it into three independent knobs. A free-style natural-language instruction sets role, emotion, speaking style, rate, timbre, and accent for the whole utterance. 86 fine-grained inline tags drop non-verbal events — laughter, breathing, coughing, sighing — at the word level, inside the text. And the text itself is the content. Switch the instruction and the delivery re-colors without a word changing:
So — [breath] here's the thing. [laughter] It actually worked. [sigh] Finally.
The instruction re-colors the whole line — pitch, rate, affect — without touching a word of the text. The [tags] are separate: 86 of them, placed inline, fire non-verbal events exactly where they sit. Content, global style, and punctuated non-verbals are three independent knobs, which is what "controllable" actually has to mean.
On top of that: 16 languages and 20 Chinese dialect regions (seven languages new this version), one-pass long-form synthesis up to three minutes, a reproducible speaker fine-tuning protocol, and vocoder super-resolution to 48 kHz. It also handles hard text-normalization cases and degraded reference speech without a separate cleanup stage.
The receipts
The project page leads with two radar charts across the CV3-Eval multilingual set — one for content consistency (word-error rate) and one for speaker similarity — against MiniMax-Speech, ElevenLabs v3, VoxCPM2, DotsTTS, and the Qwen3-TTS base. The shape tells the story: Qwen-Audio-3.0- TTS holds a large, even envelope across all ~20 language axes, where the lighter baselines collapse on the long-tail languages.
The paper reports state-of-the-art results across SEED-TTS-Eval, CV3-Eval, instruction-following, long-form, and acoustic-robustness suites; the leaderboard #1 is the headline. (Exact WER/SIM figures live in those radar charts rather than a table on the page — the shape is the claim.)
The training, briefly
The two-model split has a matching two-track training recipe — five progressive stages: the LM and flow-matching decoder are pretrained independently, then jointly trained with a high-quality-data annealing phase, then the LM gets a reinforcement-learning pass, and the decoder gets its own robustness stage and then its own RL stage. The robustness stage is what lets the flow-matching decoder cope with degraded prompts; the separate RL passes are what sharpen intelligibility and speaker fidelity without the two objectives fighting.
The take
Qwen-Audio-3.0-TTS isn't a new paradigm — it's the LM-plus-flow-matching recipe executed with taste. The 12.5 Hz supervised tokenizer keeps it fast, the content/voice split keeps it controllable, the inline tags and instructions make it expressive, and the multilingual coverage is broad and even rather than English-plus-a-long-tail. The interesting lesson is how much of "good TTS" is now about the surfaces you expose — frame rate, instruction grammar, tag vocabulary — rather than the core generative trick, which the field has largely settled.
Source: the Qwen-Audio-3.0-TTS project page (Alibaba / FunAudioLLM). The radar figures are theirs; the interactive diagrams are mine.