# Qwen-Audio-3.1: five audio APIs, and which of them the papers actually describe

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/qwen-audio-3-1
> date: 2026-09-26
> tags: qwen, audio, speech, tts, asr, voice-agents, benchmarks, pricing

On 23 September Alibaba announced [Qwen-Audio-3.1](https://x.com/Alibaba_Qwen/status/2102687258990026993) as "five models, one complete audio stack": upgraded ASR, TTS and Realtime, plus new ASR-Next and TTS-Next, with "TTS ~70% off, Realtime ~85% off, and ASR up to 95% off". All five are API-only. With no weights, everything below comes from what Alibaba chose to publish.

I read what the post links, the price lists and docs in both regions, and every arXiv report with "Qwen-Audio" in the title. The evidence is uneven, and that is the story:

- **Realtime** has a real 3.1 technical report ([arXiv 2609.25176](https://arxiv.org/abs/2609.25176), submitted 21 September).
- **TTS** links a blog that is the July [Qwen-Audio-3.0-TTS report](https://arxiv.org/abs/2607.23938) with "3.0" changed to "3.1".
- **ASR** has a model page, a price row and a 3.0 predecessor report.
- **ASR-Next** and **TTS-Next** get one sentence each in the post. TTS-Next also has a price row.

<Callout type="note">
Labels: **reported** means Alibaba's own report, page or price list (every benchmark here is self-reported unless marked); **measured** means I checked it directly; **computed** is my arithmetic on their numbers; **reasoned** is an inference.
</Callout>

## The lineup

| Model | What it does (vendor's words, condensed) | In → out | API IDs I found | Technical document |
|---|---|---|---|---|
| ASR | multilingual and dialect recognition; "native polishing" drops fillers and repetitions | audio → text | `qwen-audio-3.1-asr-flash`, `-flash-filetrans`, `-flash-streaming` | none for 3.1; the [3.0-ASR report](https://arxiv.org/abs/2609.07549) |
| ASR-Next | multi-speaker ASR with speaker labels and timestamps; emotion, ambient and machine sounds; captioning, event localisation, audio QA | audio → text (reasoned) | none | none |
| TTS | multilingual and dialect synthesis, cross-lingual voice transfer, instruction control | text → audio | `qwen-audio-3.1-tts-flash` (Beijing list only) | the 3.0-TTS report, relabelled |
| TTS-Next | "unified LM + diffusion framework" for voice, sound effects and background audio in one pass | text → audio (reasoned) | `qwen-audio-3.1-tts-next` (Beijing list only) | none |
| Realtime | full-duplex speech dialogue with interruption and tool use | text + audio → text + audio | `qwen-audio-3.1-realtime-plus` | the 3.1-Realtime report |

Limits, from the docs (reported). `qwen-audio-3.1-asr-flash` takes up to 5 minutes or 2 GB (3.0 capped it at 10 MB) and adds speaker separation. `-filetrans` takes a URL of up to 12 hours. `-streaming` takes a mono stream with no duration limit. Both cover 30 languages plus ten named Chinese dialects. Realtime runs over WebSocket; the qwencloud sample sends 16 kHz mono PCM and plays 24 kHz back. Its context grew to 262K, with 245K max input and 16K max output, against 65K, 57K and 8K for 3.0-Realtime-Plus. Every qwencloud page I tried for TTS, ASR-Next and TTS-Next returned 404; the post says "More APIs: coming soon".

## The TTS blog is the 3.0 report

Four things show it:

1. The blog's "Paper" button links arXiv 2607.23938, titled *Qwen-Audio-3.0-TTS* and submitted on 27 July 2026.
2. The blog's abstract and contribution list are the report's, with "3.0" replaced by "3.1", "report" by "paper", and "two-stage speaker-adaptation protocol" by "reproducible speaker fine-tuning protocol". Every number is the same: 12.5 Hz, five training stages, 86 inline tags, 16 languages, 20 dialect regions, 3 minutes.
3. The blog's two CV3-Eval radars are the report's Figure 2. Parsing both SVGs, every polygon's vertices agree to within 0.002 units, apart from a uniform 0.9994 scale on the blog copy (measured). Only the legend differs: "Qwen-Audio-3.1-TTS" against "Qwen-Audio-3.0-TTS".
4. The blog's URL path ends `qwen_audio_3_tts_blog_review_260918/`.

<Figure
  src="/articles/qwen-audio-3-1/fig1.png"
  alt="Radar chart of per-language word or character error rate on CV3-Eval across sixteen languages (zh, en, ja, ko, de, es, fr, it, ru, ar, id, pt, th, vi, ms, tl) for MiniMax-Speech-2.8-HD, ElevenLabs-v3, Dots.TTS-2B, VoxCPM2, Qwen3-TTS-12Hz-1.7B-Base and a red polygon labelled Qwen-Audio-3.1-TTS; outer is better."
  caption="The launch blog's content-consistency radar, legend reading Qwen-Audio-3.1-TTS. Its vertices match Figure 2a of the Qwen-Audio-3.0-TTS arXiv report. The numbers behind it are that report's Table 4 (Qwen blog, Figure 1)."
/>

So no evaluation of a 3.1 TTS model has been published. Either 3.1-TTS is the 3.0 model under a new name and price, or it is a new model described by an old paper, and nothing public tells those apart. The next two sections are the 3.0 report. My [earlier piece on 3.0-TTS](/articles/qwen-audio-3-tts) worked from the project page and said the exact numbers only existed in these radars; the report has full tables, and this piece uses them.

## How the TTS stack works (3.0 report)

<Figure
  src="/articles/qwen-audio-3-1/fig2.png"
  alt="Architecture diagram: instruction and text tokens followed by audio-semantic codec tokens feed a Qwen LM that autoregressively predicts codec tokens and hidden states; the hidden states are upsampled by a factor r and channel-concatenated with noised mel, a partial prompt mel and a broadcast speaker embedding into a DiT flow-matching model, whose mel spectrogram a streaming vocoder turns into a waveform."
  caption="Language model, flow-matching model and vocoder. The flow-matching model is conditioned on the LM's upsampled hidden states rather than on token embeddings (Qwen-Audio-3.0-TTS report, Figure 3)."
/>

There are three parts. A language model (the figure labels it "Qwen LM", with no size given) reads instruction, text and prompt tokens. It predicts semantic speech tokens at 12.5 Hz. A DiT flow-matching model turns that into a mel spectrogram. A causal BigVGAN vocoder turns the mel into a waveform. The flow model's input is a channel-wise concatenation of the LM's upsampled hidden states, a partial prompt mel, a broadcast speaker embedding and the noised mel $x_t$.

The tokenizer follows CosyVoice 3. A causal SenseVoice encoder (32 Transformer layers, 1,280 hidden, 20 heads) reads 16 kHz audio through a 128-bin Whisper-style frontend at 100 Hz. The first 12 layers bring it down to 25 Hz. A quantizer encoder then halves it to 12.5 Hz. The bottleneck is a 10-dimensional FSQ with three levels per dimension, so $3^{10} = 59{,}049$ codes. It is trained with supervision (ASR, language ID, emotion, audio events, speaker) through a MinMo-style LLM initialised from Qwen2.5-7B-Instruct, continuous first, with FSQ switched on later. All of that is reported.

Why halve the rate? An autoregressive LM pays one serial step per token, so a 3-minute clip, the one-pass ceiling, costs 2,250 steps at 12.5 Hz against 4,500 at 25 Hz (computed). The price is capacity. A [CosyVoice-lineage retrospective](https://arxiv.org/abs/2609.16514) by the same group bounds a tokenizer's nominal index budget at $B = r \log_2 K$ bits per second, for frame rate $r$ and codebook size $K$. That is about 317 bits/s for CosyVoice 3's 25 Hz, 6,561-code tokenizer and about 198 bits/s for the shipped one (reported). That is 62.5% of the budget in half the steps (computed). The ablation says content accuracy still improved.

<TokenizerBudget />

The rows above are the report's Table 2. At 12.5 Hz with CosyVoice 3's 6,561 codes, test-zh CER rises from 1.45 to 2.59 and test-en speaker similarity falls from 73.60 to 61.64. With 59,049 codes, the numbers come back to 1.23 and 77.49. The 19,683-code variant keeps marginally higher similarity, and the authors picked the larger codebook for content (reported).

Training runs in five stages, each starting from the last checkpoint (reported):

1. **Independent pretraining** of LM and flow model, the CosyVoice 2/3 recipe.
2. **Joint training.** The flow model is conditioned on the LM's continuous hidden states instead of token embeddings, so the flow loss's gradient reaches the LM (credited to JoyVoice). Training then anneals onto curated high-quality data.
3. **LM RL.** Online GRPO with a KL penalty, rewarding content (token-domain ASR), duration, diversity and prosody, all scored before the flow model runs, so rollouts are token-only. A DiffRO branch and, later, a dialect-classification reward are added.
4. **Flow-model robustness** with the LM frozen: noise, reverb, phone and laptop mics, far field, codec artifacts, packet loss and echo in the prompt.
5. **Flow-model RL** with FlowTTS-GRPO: the ODE sampler becomes an SDE for exploration early in sampling, rewarded on speaker similarity, ASR and DNSMOS.

The 48 kHz output belongs to a separate speaker-adaptation path with its own super-resolution vocoder; the base model's output rate is not given. Also undisclosed: model sizes, training hours, streaming chunk size, first-packet latency and real-time factor. The report credits the tokenizer, chunked flow matching and the causal vocoder with lower latency, but prints no latency number. That is a real gap for a product whose Flash tier is sold for "low-latency interactive applications".

## What the TTS numbers say (3.0 report)

All self-reported. Content is scored from Whisper-large-v3 and Paraformer transcripts, similarity by ERes2Net or WavLM, quality by DNSMOS.

- **SEED-TTS-Eval.** test-zh CER is 0.84, second to the open Qwen3-TTS-12Hz-1.7B-Base at 0.77. test-en WER is 1.54, behind Qwen3-TTS (1.24), Qwen3.5-Omni-Plus (1.26), Dots.TTS (1.30) and LongCat-AudioDiT (1.50). test-hard CER is 7.00, behind CosyVoice3-1.5B (5.83), LongCat-AudioDiT (6.04) and Dots.TTS (6.60). It has the best ERes2Net similarity on all three (0.847, 0.815, 0.824; the authors scored several baselines on ERes2Net themselves), but not the best WavLM: 0.792 on test-zh against LongCat's 0.818. The authors say they stopped short of the lowest CER because pushing it cost naturalness.
- **CV3-Eval, 16 languages.** Best on 6 languages (ja, ko, ru, ar, th, ms); MiniMax-Speech-2.8-HD is best on 9. Mean error is 3.96% against MiniMax's 4.00% (computed from Table 4): a tie on content. On the hard subsets it has the best similarity and DNSMOS in both languages (78.7 and 3.93 in Chinese, 76.6 and 4.04 in English), with error 7.44 against MiniMax's 7.42 and 6.71 against ElevenLabs-v3's 5.84.
- **Cross-lingual cloning.** Best in 8 of 12 directions; mean error 4.05% against CosyVoice3-1.5B's 10.09%. The open Qwen3-TTS-1.7B averages 4.42% on the same table (computed).
- **Long-form** (100 paragraphs per language, 1.5 to 3 minutes each). Chinese CER is 2.22, climbing from 0.30 on short inputs to 5.62 on long ones, while VoxCPM2 stays near 0.54. English WER is 5.00 against VoxCPM2's 3.20. The edge is voice consistency: prompt similarity 78.85 in Chinese against VoxCPM2's 61.73.
- **Degraded prompts.** With no denoise mode it reaches DNSMOS 3.962 and 3.925 on noisy and reverberant prompts at similarity 76.14 and 74.12; ElevenLabs-v3 in denoise mode gets 3.981 and 4.025 at 46.07 and 44.39. MiniMax keeps lower error on noisy prompts: 0.85 against 1.18.
- **Instructions.** 78.94 in Chinese and 80.45 in English, against CosyVoice3-1.5B's 75.91 and 64.09. The judge is Gemini-2.5-Pro, which agrees with humans 70.0% of the time on single-attribute cases (conservatively, McNemar $p = 0.007$) and 56.7% per criterion on complex ones. Speaker similarity under instruction trails CosyVoice3: 73.27 against 75.60 in Chinese.
- **Human panels.** Dialects: 20 dialects, 50 prompts each, 3 native annotators, mean 3.639 authenticity, 3.935 pronunciation and 3.680 prosody on a 1 to 4 scale, with no baseline. The arena gives win rates of 44.8% (instruction following) and 55.6% (prosody) against a "previous-gen baseline" at 30.9% and 42.9%, with no sample count, and the pairs do not sum to 100.

The one third-party number is the Artificial Analysis leaderboard on 16 July: Qwen-Audio-3.0-TTS-Plus first at Elo 1,237 from 1,427 samples, rank range 1–2, confidence interval overlapping Simba 3.2's (reported). First on point estimate, inside a statistical tie.

## Realtime: the one model with a 3.1 report

<Figure
  src="/articles/qwen-audio-3-1/fig3.png"
  alt="Framework diagram: streaming audio feeds two parallel Audio Encoder plus LLM models; the full-duplex decision model outputs listen, speak, interrupt or resume, the speech-to-text model outputs a text response; a context-aware voice renderer, also fed conversation history, voice and acoustic state, produces the streaming speech output."
  caption="Two models with the same Audio Encoder + LLM shape: one decides when to speak, one decides what to say. A renderer voices the text (Qwen-Audio-3.1-Realtime report, Figure 3)."
/>

Two models listen to the same stream. A full-duplex decision model chooses, step by step, between keep listening, begin a response, stop playback and resume: $a_t = \pi(o_{\le t}, h_{\le t}, s_t)$, over audio so far, dialogue and event history, and system state (is speech or a tool call active?). A speech-to-text model writes the response as text, and a context-aware voice renderer turns text, history and acoustic context into streaming speech. The 3.0-Realtime page calls that model "end-to-end"; going by this figure, the 3.1 response path is speech in, text out, then rendered (reasoned).

Training has three layers, all reported:

- **Think.** Audio-LM pretraining follows the 3.0-ASR report, then Core-Cocktail SFT on million-hour-scale paired data. On-policy distillation follows: the student's own trajectories are scored token by token by a text teacher that sees the transcript and a frozen audio reference that hears the audio. GRPO-trained specialists (empathy, pragmatic intent, acoustic scenes) are then distilled back into one model.
- **Act.** Executable environments are seeded from open-source tool and MCP server definitions; a code agent expands each into tools, a policy document and a JSON database. Each task has a script and an expected end: a write, a justified refusal, or "unsupported". Its reference solution is replayed first to prove it is solvable; in rollouts a simulated user follows the script, and scoring checks the final database state, the permitted writes and rules such as confirming before a write. GRPO rewards whole dialogues, continuations from saved milestones, and single turns. Search is rewarded as $r = q \cdot \min(1, n_{\text{ref}}/n_{\text{pred}})$: judged query quality $q$, cut when the model issues more queries $n_{\text{pred}}$ than the judge thinks the request needs, $n_{\text{ref}}$.
- **Speak and Coordinate.** Whether, when and how to speak or act.

<DuplexEvents />

Results, from the report's own runs (reported):

- **τ-Voice**, their half-duplex speech-to-text adaptation: task success 78.4% to 82.0%, not comparable with the official full-duplex protocol.
- **Full-Duplex-Bench v1.5:** response rate to background speech 0.73 to 0.13; to the user talking to someone else, 0.13 to 0.03. The in-house multi-party session pass rate goes from 0.08 to 0.96.
- **Recognition and reasoning:** FLEURS macro WER over 14 varieties 9.01 to 3.98. Multilingual Big Bench Audio 81.7 to 88.1, with Arabic 52.5 to 79.1 and Thai 51.5 to 79.0.
- **Safety:** multi-turn attack success 80.5% to 26.0% in Chinese, 63.5% to 23.5% in English.
- **Worse, or still behind:** turn-taking latency 1.539 s to 1.921 s; time to stop on interruption 1.041 s to 1.116 s, against GPT-Realtime-2's 0.383 s; EVA-A Mean 70.50 to 66.26 (on 200 of 213 sessions); web-search trigger F1 60.87 to 58.61 as mean queries fall from 4.37 to 1.05; a human red-team pass rate of 92% against GPT-Realtime-2's 96%.

<Figure
  src="/articles/qwen-audio-3-1/fig4.png"
  alt="Grid of eight horizontal bar panels comparing GPT-Realtime-2, SeedDuplex 1.2.6.1, Gemini 3.7 Flash, Qwen-Audio-3.0-Realtime and Qwen-Audio-3.1-Realtime on AMC, multilingual BBA, FLEURS WER, tau-Voice overall, SpeechFCEval, EchoMind, HalluQA and DNA."
  caption="The report's own selection of speech-to-text results. Baselines and judges are chosen by the authors, and the τ-Voice bar is their half-duplex adaptation (Qwen-Audio-3.1-Realtime report, Figure 1)."
/>

Not disclosed: model sizes, the LLM backbone, the audio encoder, the renderer, and any end-to-end latency, which the limitations section lists as future work. For a model called Realtime, the only latency figures are Full-Duplex-Bench's turn-taking seconds.

## ASR, ASR-Next and TTS-Next

**ASR.** The closest disclosed design is the [Qwen-Audio-3.0-ASR report](https://arxiv.org/abs/2609.07549) (reported). It keeps Fun-ASR's topology (SenseVoice encoder, two-layer adapter, a CTC head that seeds hotwords, an LLM decoder) but swaps the previous 7B dense decoder for a Qwen MoE of undisclosed size, pretrained on about 20 million hours. "Native polishing" is a binary instruction flag inside the same decoding pass, so the feature the 3.1 post highlights is already in the 3.0 report. What 3.1 changed is not documented.

**ASR-Next.** I found no page, no model ID and no paper.

**TTS-Next.** The post says "unified LM + diffusion", and the Beijing list prices `qwen-audio-3.1-tts-next`. The nearest paper, [Qwen-Audio-3.0-Gen-Preview](https://arxiv.org/abs/2607.27011), generates whole mixed scenes with a non-autoregressive DiT over a shared VAE that compresses 48 kHz stereo to 25 Hz latents. That is not "LM + diffusion", so I would not assume the two are the same model.

## What the prices show

| Model | 3.0 list price | 3.1 list price |
|---|---|---|
| ASR-Flash, Beijing | 0.00022 CNY/s | 0.8 CNY/M in, 2.7 CNY/M out |
| ASR-Flash, qwencloud | \$0.000035/s | \$0.15/M in, \$0.47/M out |
| TTS-Flash, Beijing | 1 CNY per 10,000 characters | 1.5 CNY/M in, 12 CNY/M out |
| Realtime-Plus, Beijing | 5 / 40 CNY/M in (text/audio), 40 / 150 out | identical |
| Realtime-Plus, qwencloud | \$0.8 / \$6.4 /M in, \$6.4 / \$24 out | identical |
| TTS-Next, Beijing | new | 6 CNY/M in, 12 CNY/M out |

ASR and TTS move from per-second and per-character billing to tokens, and no ASR or TTS page says how many tokens a second of audio is. The only published rule is for the realtime models: seconds × 12.5.

<PriceMath />

- **ASR.** At 12.5 tokens/s an hour is 45,000 input tokens: 0.036 CNY against 0.792 CNY in Beijing (95.5% cheaper) and \$0.00675 against \$0.126 on qwencloud (94.6%). "Up to 95%" matches the audio side if ASR bills like Realtime (computed, on a reasoned assumption). Transcript tokens pull Beijing to about 91% at 12,000 tokens an hour.
- **TTS.** This depends on the output rate. The blog's own long-form demos read 449 Chinese characters in 98 s and 1,122 English characters in 72 s (measured from the page). At that pace Mandarin is 84% cheaper at 12.5 tokens/s and 68% at 25 (computed). "~70%" fits some rate; which one is not published.
- **Realtime.** List prices are identical for 3.0 and 3.1 in both regions. The Beijing page says it shows list prices only, with promotions in the console. I cannot find the 85% anywhere public.
- **Context re-billing.** User audio and text stay in context and are billed again as input on every later turn; the model's output audio is billed once (reported). At 12.5 tokens/s, 3.1's 245K input cap is about 5.4 hours of audio, against 76 minutes for 3.0's 57K (computed).

## Lineage: two families, one name

Alibaba has two audio lines, and the naming blurs them. The **Qwen-team line** is open; I checked each on Hugging Face today: [Qwen-Audio](https://huggingface.co/Qwen/Qwen-Audio) (2023), [Qwen2-Audio-7B](https://huggingface.co/Qwen/Qwen2-Audio-7B-Instruct), [Qwen2.5-Omni](https://huggingface.co/Qwen/Qwen2.5-Omni-7B) 3B and 7B, [Qwen3-Omni-30B-A3B](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct), [Qwen3-TTS-12Hz](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-Base) 0.6B and 1.7B, and [Qwen3-ASR](https://huggingface.co/Qwen/Qwen3-ASR-1.7B) 0.6B and 1.7B. Qwen3.5-Omni, [Qwen3.8-Omni-Flash](/articles/qwen3-8-omni-flash) and [Qwen3.8-LiveTranslate](/articles/qwen-livetranslate) have no listing; they are API-only like this release. The open Qwen3-TTS is the default voice of the [speech-to-speech](/articles/speech-to-speech) pipeline.

**Qwen-Audio-3.x** comes from the CosyVoice, SenseVoice and Fun-ASR group, credited as Alibaba Token Foundry. The TTS report calls CosyVoice 2 and 3 "our previous work", the ASR report calls its model "the successor to Fun-ASR", and Realtime builds on Fun-Audio-Chat. The retrospective traces the TTS side:

- CosyVoice: a 4,096-code VQ inside an ASR encoder.
- CosyVoice 2: FSQ with 6,561 codes at full utilisation against 23% for the VQ, an LM initialised from Qwen2.5-0.5B, and chunk-aware streaming.
- CosyVoice 3: 25 Hz tokens trained on 530,000 hours, a 1.5B LM and a 300M DiT.
- Qwen-Audio-3.0-TTS: 12.5 Hz, 59,049 codes and hidden-state conditioning.

That line was partly open ([CosyVoice2-0.5B](https://huggingface.co/FunAudioLLM/CosyVoice2-0.5B), [Fun-CosyVoice3-0.5B-2512](https://huggingface.co/FunAudioLLM/Fun-CosyVoice3-0.5B-2512), [Fun-ASR-Nano-2512](https://huggingface.co/FunAudioLLM/Fun-ASR-Nano-2512)). The 3.x models are not. The one open 3.x artifact is [qwen-audio-agent](https://github.com/QwenAudio/qwen-audio-agent), a JavaScript voice-agent runtime that informs the Realtime report's persistent Voice Harness, which was evaluated with a 3.0 foreground, not 3.1.

For builders, one row matters. In the 3.0-TTS report's own tables the open Qwen3-TTS-12Hz-1.7B-Base wins test-zh CER, beats the closed model on test-en WER and Chinese CV3 error, and trails it only slightly on cross-lingual error. Its open tokenizer also runs at 12.5 Hz; [Breeze TTS 2](/articles/breeze-tts-2) is built on it. The closed model's measured lead over it is speaker similarity and robustness to bad prompts. That is a real lead, but not a lead on getting the words right.

## What I would want before building on it

- A 3.1 TTS evaluation, or a plain statement that 3.1-TTS is the 3.0 model.
- The tokens-per-second rule for the ASR and TTS APIs, so the price cuts can be checked.
- First-packet latency and real-time factor for TTS, and end-to-end latency for Realtime.
- A page, a model ID and a document for ASR-Next.

The Realtime report is the real substance of this release. It spells out a two-model duplex design, an executable-environment RL setup, and a benchmark table that shows the regressions next to the wins. Everything else is a product launch that borrows a July paper for its evidence.