Qwen-Audio-3.1: five audio APIs, and which of them the papers actually describe

mdjsonmcpcopy↗ai

2026-09-26 · 17 min · qwen · audio · speech · tts · asr · voice-agents · benchmarks · pricing

On 23 September Alibaba announced Qwen-Audio-3.1 as "five models, one complete audio stack": upgraded ASR, TTS and Realtime, plus new ASR-Next and TTS-Next, with "TTS ~70% off, Realtime ~85% off, and ASR up to 95% off". All five are API-only. With no weights, everything below comes from what Alibaba chose to publish.

I read what the post links, the price lists and docs in both regions, and every arXiv report with "Qwen-Audio" in the title. The evidence is uneven, and that is the story:

• Realtime has a real 3.1 technical report (arXiv 2609.25176, submitted 21 September).
• TTS links a blog that is the July Qwen-Audio-3.0-TTS report with "3.0" changed to "3.1".
• ASR has a model page, a price row and a 3.0 predecessor report.
• ASR-Next and TTS-Next get one sentence each in the post. TTS-Next also has a price row.

The lineup

Limits, from the docs (reported). qwen-audio-3.1-asr-flash takes up to 5 minutes or 2 GB (3.0 capped it at 10 MB) and adds speaker separation. -filetrans takes a URL of up to 12 hours. -streaming takes a mono stream with no duration limit. Both cover 30 languages plus ten named Chinese dialects. Realtime runs over WebSocket; the qwencloud sample sends 16 kHz mono PCM and plays 24 kHz back. Its context grew to 262K, with 245K max input and 16K max output, against 65K, 57K and 8K for 3.0-Realtime-Plus. Every qwencloud page I tried for TTS, ASR-Next and TTS-Next returned 404; the post says "More APIs: coming soon".

The TTS blog is the 3.0 report

Four things show it:

1. The blog's "Paper" button links arXiv 2607.23938, titled Qwen-Audio-3.0-TTS and submitted on 27 July 2026.
2. The blog's abstract and contribution list are the report's, with "3.0" replaced by "3.1", "report" by "paper", and "two-stage speaker-adaptation protocol" by "reproducible speaker fine-tuning protocol". Every number is the same: 12.5 Hz, five training stages, 86 inline tags, 16 languages, 20 dialect regions, 3 minutes.
3. The blog's two CV3-Eval radars are the report's Figure 2. Parsing both SVGs, every polygon's vertices agree to within 0.002 units, apart from a uniform 0.9994 scale on the blog copy (measured). Only the legend differs: "Qwen-Audio-3.1-TTS" against "Qwen-Audio-3.0-TTS".
4. The blog's URL path ends qwen_audio_3_tts_blog_review_260918/.

expand

So no evaluation of a 3.1 TTS model has been published. Either 3.1-TTS is the 3.0 model under a new name and price, or it is a new model described by an old paper, and nothing public tells those apart. The next two sections are the 3.0 report. My earlier piece on 3.0-TTS worked from the project page and said the exact numbers only existed in these radars; the report has full tables, and this piece uses them.

How the TTS stack works (3.0 report)

expand

There are three parts. A language model (the figure labels it "Qwen LM", with no size given) reads instruction, text and prompt tokens. It predicts semantic speech tokens at 12.5 Hz. A DiT flow-matching model turns that into a mel spectrogram. A causal BigVGAN vocoder turns the mel into a waveform. The flow model's input is a channel-wise concatenation of the LM's upsampled hidden states, a partial prompt mel, a broadcast speaker embedding and the noised mel $x_t$.

The tokenizer follows CosyVoice 3. A causal SenseVoice encoder (32 Transformer layers, 1,280 hidden, 20 heads) reads 16 kHz audio through a 128-bin Whisper-style frontend at 100 Hz. The first 12 layers bring it down to 25 Hz. A quantizer encoder then halves it to 12.5 Hz. The bottleneck is a 10-dimensional FSQ with three levels per dimension, so $3^{10} = 59{,}049$ codes. It is trained with supervision (ASR, language ID, emotion, audio events, speaker) through a MinMo-style LLM initialised from Qwen2.5-7B-Instruct, continuous first, with FSQ switched on later. All of that is reported.

Why halve the rate? An autoregressive LM pays one serial step per token, so a 3-minute clip, the one-pass ceiling, costs 2,250 steps at 12.5 Hz against 4,500 at 25 Hz (computed). The price is capacity. A CosyVoice-lineage retrospective by the same group bounds a tokenizer's nominal index budget at $B = r \log_2 K$ bits per second, for frame rate $r$ and codebook size $K$. That is about 317 bits/s for CosyVoice 3's 25 Hz, 6,561-code tokenizer and about 198 bits/s for the shipped one (reported). That is 62.5% of the budget in half the steps (computed). The ablation says content accuracy still improved.

The rows above are the report's Table 2. At 12.5 Hz with CosyVoice 3's 6,561 codes, test-zh CER rises from 1.45 to 2.59 and test-en speaker similarity falls from 73.60 to 61.64. With 59,049 codes, the numbers come back to 1.23 and 77.49. The 19,683-code variant keeps marginally higher similarity, and the authors picked the larger codebook for content (reported).

Training runs in five stages, each starting from the last checkpoint (reported):

1. Independent pretraining of LM and flow model, the CosyVoice 2/3 recipe.
2. Joint training. The flow model is conditioned on the LM's continuous hidden states instead of token embeddings, so the flow loss's gradient reaches the LM (credited to JoyVoice). Training then anneals onto curated high-quality data.
3. LM RL. Online GRPO with a KL penalty, rewarding content (token-domain ASR), duration, diversity and prosody, all scored before the flow model runs, so rollouts are token-only. A DiffRO branch and, later, a dialect-classification reward are added.
4. Flow-model robustness with the LM frozen: noise, reverb, phone and laptop mics, far field, codec artifacts, packet loss and echo in the prompt.
5. Flow-model RL with FlowTTS-GRPO: the ODE sampler becomes an SDE for exploration early in sampling, rewarded on speaker similarity, ASR and DNSMOS.

The 48 kHz output belongs to a separate speaker-adaptation path with its own super-resolution vocoder; the base model's output rate is not given. Also undisclosed: model sizes, training hours, streaming chunk size, first-packet latency and real-time factor. The report credits the tokenizer, chunked flow matching and the causal vocoder with lower latency, but prints no latency number. That is a real gap for a product whose Flash tier is sold for "low-latency interactive applications".

What the TTS numbers say (3.0 report)

All self-reported. Content is scored from Whisper-large-v3 and Paraformer transcripts, similarity by ERes2Net or WavLM, quality by DNSMOS.

• SEED-TTS-Eval. test-zh CER is 0.84, second to the open Qwen3-TTS-12Hz-1.7B-Base at 0.77. test-en WER is 1.54, behind Qwen3-TTS (1.24), Qwen3.5-Omni-Plus (1.26), Dots.TTS (1.30) and LongCat-AudioDiT (1.50). test-hard CER is 7.00, behind CosyVoice3-1.5B (5.83), LongCat-AudioDiT (6.04) and Dots.TTS (6.60). It has the best ERes2Net similarity on all three (0.847, 0.815, 0.824; the authors scored several baselines on ERes2Net themselves), but not the best WavLM: 0.792 on test-zh against LongCat's 0.818. The authors say they stopped short of the lowest CER because pushing it cost naturalness.
• CV3-Eval, 16 languages. Best on 6 languages (ja, ko, ru, ar, th, ms); MiniMax-Speech-2.8-HD is best on 9. Mean error is 3.96% against MiniMax's 4.00% (computed from Table 4): a tie on content. On the hard subsets it has the best similarity and DNSMOS in both languages (78.7 and 3.93 in Chinese, 76.6 and 4.04 in English), with error 7.44 against MiniMax's 7.42 and 6.71 against ElevenLabs-v3's 5.84.
• Cross-lingual cloning. Best in 8 of 12 directions; mean error 4.05% against CosyVoice3-1.5B's 10.09%. The open Qwen3-TTS-1.7B averages 4.42% on the same table (computed).
• Long-form (100 paragraphs per language, 1.5 to 3 minutes each). Chinese CER is 2.22, climbing from 0.30 on short inputs to 5.62 on long ones, while VoxCPM2 stays near 0.54. English WER is 5.00 against VoxCPM2's 3.20. The edge is voice consistency: prompt similarity 78.85 in Chinese against VoxCPM2's 61.73.
• Degraded prompts. With no denoise mode it reaches DNSMOS 3.962 and 3.925 on noisy and reverberant prompts at similarity 76.14 and 74.12; ElevenLabs-v3 in denoise mode gets 3.981 and 4.025 at 46.07 and 44.39. MiniMax keeps lower error on noisy prompts: 0.85 against 1.18.
• Instructions. 78.94 in Chinese and 80.45 in English, against CosyVoice3-1.5B's 75.91 and 64.09. The judge is Gemini-2.5-Pro, which agrees with humans 70.0% of the time on single-attribute cases (conservatively, McNemar $p = 0.007$) and 56.7% per criterion on complex ones. Speaker similarity under instruction trails CosyVoice3: 73.27 against 75.60 in Chinese.
• Human panels. Dialects: 20 dialects, 50 prompts each, 3 native annotators, mean 3.639 authenticity, 3.935 pronunciation and 3.680 prosody on a 1 to 4 scale, with no baseline. The arena gives win rates of 44.8% (instruction following) and 55.6% (prosody) against a "previous-gen baseline" at 30.9% and 42.9%, with no sample count, and the pairs do not sum to 100.

The one third-party number is the Artificial Analysis leaderboard on 16 July: Qwen-Audio-3.0-TTS-Plus first at Elo 1,237 from 1,427 samples, rank range 1–2, confidence interval overlapping Simba 3.2's (reported). First on point estimate, inside a statistical tie.

Realtime: the one model with a 3.1 report

expand

Two models listen to the same stream. A full-duplex decision model chooses, step by step, between keep listening, begin a response, stop playback and resume: $a_t = \pi(o_{\le t}, h_{\le t}, s_t)$, over audio so far, dialogue and event history, and system state (is speech or a tool call active?). A speech-to-text model writes the response as text, and a context-aware voice renderer turns text, history and acoustic context into streaming speech. The 3.0-Realtime page calls that model "end-to-end"; going by this figure, the 3.1 response path is speech in, text out, then rendered (reasoned).

Training has three layers, all reported:

• Think. Audio-LM pretraining follows the 3.0-ASR report, then Core-Cocktail SFT on million-hour-scale paired data. On-policy distillation follows: the student's own trajectories are scored token by token by a text teacher that sees the transcript and a frozen audio reference that hears the audio. GRPO-trained specialists (empathy, pragmatic intent, acoustic scenes) are then distilled back into one model.
• Act. Executable environments are seeded from open-source tool and MCP server definitions; a code agent expands each into tools, a policy document and a JSON database. Each task has a script and an expected end: a write, a justified refusal, or "unsupported". Its reference solution is replayed first to prove it is solvable; in rollouts a simulated user follows the script, and scoring checks the final database state, the permitted writes and rules such as confirming before a write. GRPO rewards whole dialogues, continuations from saved milestones, and single turns. Search is rewarded as $r = q \cdot \min(1, n_{\text{ref}}/n_{\text{pred}})$: judged query quality $q$, cut when the model issues more queries $n_{\text{pred}}$ than the judge thinks the request needs, $n_{\text{ref}}$.
• Speak and Coordinate. Whether, when and how to speak or act.

Results, from the report's own runs (reported):

• τ-Voice, their half-duplex speech-to-text adaptation: task success 78.4% to 82.0%, not comparable with the official full-duplex protocol.
• Full-Duplex-Bench v1.5: response rate to background speech 0.73 to 0.13; to the user talking to someone else, 0.13 to 0.03. The in-house multi-party session pass rate goes from 0.08 to 0.96.
• Recognition and reasoning: FLEURS macro WER over 14 varieties 9.01 to 3.98. Multilingual Big Bench Audio 81.7 to 88.1, with Arabic 52.5 to 79.1 and Thai 51.5 to 79.0.
• Safety: multi-turn attack success 80.5% to 26.0% in Chinese, 63.5% to 23.5% in English.
• Worse, or still behind: turn-taking latency 1.539 s to 1.921 s; time to stop on interruption 1.041 s to 1.116 s, against GPT-Realtime-2's 0.383 s; EVA-A Mean 70.50 to 66.26 (on 200 of 213 sessions); web-search trigger F1 60.87 to 58.61 as mean queries fall from 4.37 to 1.05; a human red-team pass rate of 92% against GPT-Realtime-2's 96%.

expand

Not disclosed: model sizes, the LLM backbone, the audio encoder, the renderer, and any end-to-end latency, which the limitations section lists as future work. For a model called Realtime, the only latency figures are Full-Duplex-Bench's turn-taking seconds.

ASR, ASR-Next and TTS-Next

ASR. The closest disclosed design is the Qwen-Audio-3.0-ASR report (reported). It keeps Fun-ASR's topology (SenseVoice encoder, two-layer adapter, a CTC head that seeds hotwords, an LLM decoder) but swaps the previous 7B dense decoder for a Qwen MoE of undisclosed size, pretrained on about 20 million hours. "Native polishing" is a binary instruction flag inside the same decoding pass, so the feature the 3.1 post highlights is already in the 3.0 report. What 3.1 changed is not documented.

ASR-Next. I found no page, no model ID and no paper.

TTS-Next. The post says "unified LM + diffusion", and the Beijing list prices qwen-audio-3.1-tts-next. The nearest paper, Qwen-Audio-3.0-Gen-Preview, generates whole mixed scenes with a non-autoregressive DiT over a shared VAE that compresses 48 kHz stereo to 25 Hz latents. That is not "LM + diffusion", so I would not assume the two are the same model.

What the prices show

ASR and TTS move from per-second and per-character billing to tokens, and no ASR or TTS page says how many tokens a second of audio is. The only published rule is for the realtime models: seconds × 12.5.

• ASR. At 12.5 tokens/s an hour is 45,000 input tokens: 0.036 CNY against 0.792 CNY in Beijing (95.5% cheaper) and $0.00675 against $0.126 on qwencloud (94.6%). "Up to 95%" matches the audio side if ASR bills like Realtime (computed, on a reasoned assumption). Transcript tokens pull Beijing to about 91% at 12,000 tokens an hour.
• TTS. This depends on the output rate. The blog's own long-form demos read 449 Chinese characters in 98 s and 1,122 English characters in 72 s (measured from the page). At that pace Mandarin is 84% cheaper at 12.5 tokens/s and 68% at 25 (computed). "~70%" fits some rate; which one is not published.
• Realtime. List prices are identical for 3.0 and 3.1 in both regions. The Beijing page says it shows list prices only, with promotions in the console. I cannot find the 85% anywhere public.
• Context re-billing. User audio and text stay in context and are billed again as input on every later turn; the model's output audio is billed once (reported). At 12.5 tokens/s, 3.1's 245K input cap is about 5.4 hours of audio, against 76 minutes for 3.0's 57K (computed).

Lineage: two families, one name

Alibaba has two audio lines, and the naming blurs them. The Qwen-team line is open; I checked each on Hugging Face today: Qwen-Audio (2023), Qwen2-Audio-7B, Qwen2.5-Omni 3B and 7B, Qwen3-Omni-30B-A3B, Qwen3-TTS-12Hz 0.6B and 1.7B, and Qwen3-ASR 0.6B and 1.7B. Qwen3.5-Omni, Qwen3.8-Omni-Flash and Qwen3.8-LiveTranslate have no listing; they are API-only like this release. The open Qwen3-TTS is the default voice of the speech-to-speech pipeline.

Qwen-Audio-3.x comes from the CosyVoice, SenseVoice and Fun-ASR group, credited as Alibaba Token Foundry. The TTS report calls CosyVoice 2 and 3 "our previous work", the ASR report calls its model "the successor to Fun-ASR", and Realtime builds on Fun-Audio-Chat. The retrospective traces the TTS side:

• CosyVoice: a 4,096-code VQ inside an ASR encoder.
• CosyVoice 2: FSQ with 6,561 codes at full utilisation against 23% for the VQ, an LM initialised from Qwen2.5-0.5B, and chunk-aware streaming.
• CosyVoice 3: 25 Hz tokens trained on 530,000 hours, a 1.5B LM and a 300M DiT.
• Qwen-Audio-3.0-TTS: 12.5 Hz, 59,049 codes and hidden-state conditioning.

That line was partly open (CosyVoice2-0.5B, Fun-CosyVoice3-0.5B-2512, Fun-ASR-Nano-2512). The 3.x models are not. The one open 3.x artifact is qwen-audio-agent, a JavaScript voice-agent runtime that informs the Realtime report's persistent Voice Harness, which was evaluated with a 3.0 foreground, not 3.1.

For builders, one row matters. In the 3.0-TTS report's own tables the open Qwen3-TTS-12Hz-1.7B-Base wins test-zh CER, beats the closed model on test-en WER and Chinese CV3 error, and trails it only slightly on cross-lingual error. Its open tokenizer also runs at 12.5 Hz; Breeze TTS 2 is built on it. The closed model's measured lead over it is speaker similarity and robustness to bad prompts. That is a real lead, but not a lead on getting the words right.

What I would want before building on it

• A 3.1 TTS evaluation, or a plain statement that 3.1-TTS is the 3.0 model.
• The tokens-per-second rule for the ASR and TTS APIs, so the price cuts can be checked.
• First-packet latency and real-time factor for TTS, and end-to-end latency for Realtime.
• A page, a model ID and a document for ASR-Next.

The Realtime report is the real substance of this release. It spells out a two-model duplex design, an executable-environment RL setup, and a benchmark table that shows the regressions next to the wins. Everything else is a product launch that borrows a July paper for its evidence.