Fish Audio Drama 3: a Qwen3.5 MoE that reads stage directions as plain text

mdjsonmcpcopy↗ai

2026-10-14 · 19 min · tts · speech · audio · mixture-of-experts · linear-attention · inference · benchmarks · licensing

Why read this

High· top 30%

Drama 3 counted from its headers and tokenizer: Qwen3.5 backbone, S2's codec, 5.95 GB per frame, tags as BPE you can count, claims against the report's tables.

• Checked against the source
• Widely used
• A lasting reference

·Speech & audio·Needs a workstation GPU·Custom licence·Practitioner model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

3 of 3: Reproduces the headline result, or shows from primary files it is wrong

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

2 of 3: A reference for a year or more

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 72 of 100, ranked 80 of 517 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Two weeks ago I wrote up Fish Audio S2 because its successor, Drama 3, was a gated preview with nothing to read. That piece ended with a list of what would have to be published to evaluate Drama 3. On 13 October most of it was: a technical report, weights at fishaudio/drama-3 (repo created 1 October, commit e0a9218), the model code in fishaudio/fish-speech and a day-0 serving path in SGLang-Omni.

The launch post calls it "the most emotive and controllable TTS model in the world". This piece reads what was shipped: the safetensors headers by HTTP range request (eight shards, 62,245 tensors, no weights downloaded), config.json, tokenizer.json, the licence, the report's LaTeX source, and the prompt builder in SGLang-Omni at commit 5e9b664.

The backbone is Qwen3.5-35B-A3B, as the name says

A community conversion on the Hub is named Qwen3.5-35B-A3B-FishAudio-Drama3. I did not take the name on trust. Diffing text_config against Qwen/Qwen3.5-35B-A3B's config.json, every field that sets a shape matches: 40 layers, hidden_size 2048, 16 query heads over 2 KV heads with head_dim 256, 256 experts of width 512, 8 per token, a 512-wide shared expert, and the Gated DeltaNet dimensions (16 key heads, 32 value heads, 128 each, a 4-tap convolution) (measured). What differs is bookkeeping: the multimodal rope sections are gone, Qwen's native MTP layer is gone, and the vocabulary grows from 248,320 rows to 252,288.

The report agrees: the backbone "is initialized from Qwen3.5-35B-A3B", language model only. So the name is fair, for half the model.

Summing the headers by role (measured):

Routed experts are 89.7% of the file (reasoned). Per token the backbone reads its non-expert layers (1.43 B), 8 of 256 experts in each of 40 layers (1.01 B) and the LM head (0.52 B): about 2.95 B, which is the "A3B" (reasoned from measured shapes). The README's "35B parameters, 3B active per token" holds.

The tokenizer shows what Fish added. Qwen3.5 ships 33 added tokens; Drama 3's tokenizer.json has 4,172, so Fish added 4,139: 4,096 <|semantic:i|> codes, sixteen speaker tokens, <|voice|>, <|audio_separator|>, <|endofturn|> and a handful of others (measured). Some are not in the report at all: sixteen <|speaker:u0|> to <|speaker:u15|>, <|speaker_change|> and a <|back_channel_start|> / <|back_channel_end|> pair. Nothing in the released code writes them. They read like a full-duplex dialogue format in progress (reasoned; unverified).

The codec did not change

codec.pth in fishaudio/drama-3 and in fishaudio/s2-pro is 1,871,099,728 bytes with SHA-256 74fc41c5…07052bb2f in both (measured from the Hub's LFS metadata). The report says it reuses the S2 tokenizer "unchanged", and the licence covers "the copy of that codec distributed with Fish Audio S2-Pro". Everything the S2 piece measured about it carries over: 44.1 kHz in, a hop of 2,048 samples, so 21.53 frames per second, each frame one semantic code and nine residual codes.

So Drama 3 is a new language model over the same audio vocabulary, which makes the S2 comparison clean.

expand

What changed: three moves away from S2's Dual-AR

S2 was a dense Qwen3-4B stepping once per frame (the "slow AR") plus a four-layer, full-width "fast AR" that looped ten times inside each frame to fill the ten codebooks. Drama 3 keeps one backbone step per frame and changes the other three things.

1. Partial codebook fusion. S2 summed the embeddings of all ten codes of a frame into the backbone's input. Drama 3 feeds back two:

where $q^{(k)}_t$ is codebook $k$ of frame $t$, $E^{\text{LM}}$ is the backbone's token embedding (the semantic code is one of the 4,096 added vocabulary entries), $E^{(1)}$ is the slice for codebook 1 of a 40,960 x 2048 codebook table (measured), and $1/\sqrt{2}$ keeps the sum at the scale of one embedding. The stated reason is exposure bias: a backbone that conditions on all ten of its own sampled codes conditions on ten sources of sampling noise.

2. A codebook decoder that sees the whole sequence. Codes 2 to 9 do not vanish. They go to a 13-layer decoder that runs over every position of the sequence, text included, on top of the backbone's hidden state. Each layer merges an injected code embedding with the hidden state from below, W[RMSNorm(e); RMSNorm(h)], then runs one transformer block with causal attention over the whole sequence. The headers have exactly that: per layer an enorm, an hnorm and a [2048, 4096] eh_proj (measured). Four shared layers take the semantic code just predicted plus, in the first, the plain sum of the input frame's codes 2 to 9. Nine per-codebook layers follow, each with its own 4,096-row code table and 4,096-wide head, each reading the code the layer below just sampled and emitting the next (measured: nine heads.N, nine codebook_embedding, four semantic_embedding).

The difference from S2's fast AR is what each acoustic prediction can see. S2's fast AR saw one backbone state per frame. Drama 3's codebook-9 layer attends over every earlier position, including all ten codebooks of the reference clip, so fine acoustic detail is predicted from the reference's fine acoustic detail, not from a summary vector.

3. Linear attention and MoE in the backbone. Thirty of 40 backbone layers are Gated DeltaNet, which carries history in a fixed-size state. Only the ten full-attention layers and the 13 decoder layers keep a KV cache that grows with the sequence: 23 layers, as the report's Table 1 says (measured from layer_types and the decoder config). At BF16 that is 23 layers x (K and V) x 2 KV heads x 256 dimensions x 2 bytes = 47,104 bytes per frame; S2's 36 slow layers with 8 KV heads of 128 needed 147,456 (reasoned from configs). A ten-minute session is about 12,900 frames, so about 0.61 GB of KV for Drama 3 against 1.9 GB for S2's backbone, plus a constant 63 MB of DeltaNet state per stream at FP32 (reasoned). For a model built for long directed sessions, that is the point of the hybrid. The linear-attention roundup covers the state-size trade in general.

Weights per frame: the report's Table 1 checks out

At batch size 1, decoding is bound by reading weights. The S2 piece found that S2's "lightweight" fast AR read as much per frame as its 4B backbone, because four full-width layers ran ten times. Drama 3's report now says the same, and claims 6.0 GB per frame against S2's 15.6.

I recounted from the headers. A Gated DeltaNet layer reads 74.79 MB of BF16 plus eight FP8 experts at 25.17 MB; a full-attention layer 61.88 MB plus the same experts; a shared decoder layer 146.82 MB; a per-codebook layer 163.60 MB with its head; the semantic slice of the LM head 16.78 MB. Forty backbone layers plus 13 decoder layers come to 5.95 GB (reasoned from measured shapes). S2's 36 slow layers plus 40 fast layer passes and their heads come to 15.55 GB (reasoned from the S2 piece's measured shapes). The report's 2.9 + 1.0 + 2.1 split matches mine to rounding.

expand

The bytes ignore activations and KV reads, which grow with context. The FP8 experts are not a serving afterthought: the RL rollouts ran in the same block-scaled FP8, so serving inherits the numerics the policy was optimised under.

Direction is text, except the speaker

The launch post's "control how even a single word should sound" is Appendix C of the report, the control syntax. It has four kinds of direction, all free natural language:

• a point tag at a word position: [sighs], [pause], [clears throat]
• a scoped tag around a span: <whisper>you won't tell her</whisper>, <emphasis>…</emphasis>, <slow>…</slow>
• a style prefix for the whole line: [icy calm, as if you already accepted it] Fine.
• a director's note or, in later turns, a relative note: [softer and warmer versus previous]

The appendix says it plainly: "Structural tokens, written <|name|>, are single vocabulary entries; every other control is ordinary text." The tokenizer confirms it: none of the 4,172 added tokens contains a square bracket or the word whisper (measured). <whisper> is four BPE pieces, [sighs] is four, [laughs] three (measured). Scoped tags are the new thing over S2. They are how one word gets its own delivery: the tag says where the delivery starts and where it stops.

The speaker tag is the one that changed. In S2, <|speaker:1|> was a string BPE split into pieces. In Drama 3, <|speaker:0|> to <|speaker:15|> are vocabulary entries 248,083 to 248,098 (measured). "Shift your voice mid-sentence" is a speaker token in the middle of a line: the appendix's own example is <|speaker:0|> Ready? <|speaker:1|> Always.

The shipped chat_template.jinja is Qwen3.5's, with vision, tool-call and <think> handling, and never emits <|voice|> (measured). The real prompt is built in code: SGLang-Omni's prompt.py writes a system turn with the instruction and each reference as a numbered [Reference N] block holding its speaker-tagged transcript and its audio frames, a user turn with the text (prefixed <|speaker:0|> when untagged), and then <|im_start|>assistant\n<|voice|>; generation runs until the model emits <|im_end|>. It tokenizes each literal separately, "because tokenizing the joined string changes ids at the seams".

The explorer builds that prompt for one line, with the user turn shown as the BPE pieces the model receives, counted with the real vocabulary.

Three things fall out of it (measured with the shipped tokenizer):

• The plain line, <|speaker:0|> Promise me you won't tell her about the party tonight., is 13 tokens. Whispering two words and putting a sigh before a third makes it 26. Direction is paid in context, like any prompt.
• A scoped tag changes how its own word is cut. tell is one token; <whisper>tell ends in >t and ell. The model was trained on such strings, so this is not a bug, but a tagged word is a different token sequence, not the same word with a flag on it.
• With one 3.9 s reference, about 84 of the prompt's positions are audio frames (reasoned: 3.9 x 21.53), against 46 text tokens for the rest of the system turn (measured). The reference is the bulk of the prompt and, as in S2, a cacheable prefix.

New tasks, written as conversations

Multi-turn revision, scene dubbing, editing and streaming input are new in Drama 3, and none of them needs new architecture. Each is a conversation layout over the same token stream (report, Appendices A and B):

• Revision: earlier takes stay in context as audio, and the next user turn is a note, [softer and warmer versus previous], read against what was generated.
• Scene dubbing: characters described in the system turn, every line up front, separated by <|audio_separator|>, which the model also emits.
• Streaming input: each user turn holds the chunk to voice, <|endofturn|>, then the next chunk as lookahead.
• Editing by infilling: the user turn holds the full new transcript; the audio after the edit comes first, behind <|fim_suffix|>, then the audio before it, behind <|fim_prefix|>, and the model generates the middle.

That last one answers the S2 piece, which argued that a left-to-right decoder cannot condition on the audio after a gap. Drama 3 does it by ordering: suffix first, prefix last, so the model continues from the prefix having already read what follows. It is code models' fill-in-the-middle trick, applied to codec frames.

expand

The open code does not implement most of these yet. In the SGLang-Omni and fish-speech clones I read, fim_, <|audio_separator|>, <|endofturn|> and previous_takes appear nowhere (measured by grep); the request parser reads text and references, though the model card's editing example sends previous_takes. The day-0 server builds cloning prompts; the rest you build by hand from the appendix.

The reward is one model, and it was tested

S2's RL reward was ASR confidence plus a quality score plus speaker similarity. Drama 3 drops all three for a single learned reward model trained on human preference pairs, and the report's reason is the most useful table in it. On Expresso, where the same speakers read the same sentences in different styles, speaker similarity to a neutral recording prefers the neutral reading in 99.7% of whisper pairs and 93.9% of laughing pairs, and laughing raises WER by 0.79 points (reported, Table 3). A similarity reward punishes exactly the performances the model is for. The reward model, given the direction in the script, prefers the whispered reading in 62.2% of pairs and the laughing one in 64.4%, against 10.6% and 27.1% without it (reported).

Two details matter for the results. The codebook decoder is frozen during RL, so preference training moves only the semantic stream. And every benchmark regenerates outputs that fail "simple signal checks" (silence, runaway length, noise) before scoring. On Seed-TTS-Eval English, 1.84% of Drama 3's first takes are flagged, and the first-take WER is 2.20% against 1.54% after regeneration (reported, Table 4).

The claims against the tables

"The most emotive and controllable TTS model in the world." No table measures this. There is no human listening test against another system and no word-level control benchmark; S2's Tag Activation Rate is not repeated. The one direct test of emotional direction is the Step-Audio-Edit benchmark, judged by Gemini 2.5 Pro. After three revision rounds Drama 3 reaches 58.2% emotion accuracy against Step-Audio-EditX's 69.3%, and 81.8% style accuracy against 63.8% (reported, Table 9). Adding a laugh or breath by point tag scores 2.32 out of 3 against Step-Audio-EditX's 2.89. It wins one of three, loses two. The superlative is marketing.

Multilingual cloning, "a new level of SOTA" (fish-speech README). Recounting the report's Table 6, 24 languages and seven systems: Drama 3 has the strictly lowest error rate in 5 (Chinese, Dutch, English, German, Turkish) and ties for lowest in 2 more; ElevenLabs v4, run by Fish through its API, is lowest or tied in 9 (measured from the table). Against S2 it is better in 18 languages and worse in 6, including Japanese (5.02 against 2.76) and Korean (2.36 against 1.18). Strong, not a sweep.

Voice similarity. In the one table that compares it (LibriSpeech-PC, Table 5), Drama 3's is the lowest: 0.605 with one reference and 0.633 with three, against 0.694 to 0.729 for CosyVoice 3, Qwen3-TTS and OmniVoice (reported). That fits an RL stage that stopped rewarding similarity; the report does not draw the connection.

Against S2's own numbers. S2's report gave Seed-TTS-Eval WER of 0.99% English and 0.54% Chinese; Drama 3's gives 1.54% and 0.83% (reported). Different runs, but on the benchmark S2 led with, the successor reports worse intelligibility (reasoned).

Editing is a real new capability: in Chinese, the highest similarity on insertion and substitution among the systems CosyEdit2 reports, and lower WER than Cartesia Sonic-3's infill API on both splits in both languages (reported, Tables 7 and 8). In English, SSR-Speech and CosyEdit2 beat it on WER and similarity in every column.

Serving it

The report's engine decodes one stream at 271 frames per second on an H200, against 177 for S2 on the same engine (reported). The model card's SGLang-Omni table measures 166.7 frames per second at concurrency 1, with first audio at 88.9 ms and RTF 0.131, and RTF 0.801 at 64 streams (reported). Playback consumes 21.53 frames per second, so one stream runs about 7.7 times faster than real time on SGLang-Omni (reasoned). The report does not say its 271 came from the SGLang-Omni build.

The weights are 39.60 GB plus a 1.87 GB codec (measured), so an 80 GB-class GPU at the shipped precision before KV cache (reasoned); Fish's production runs prefill and decode on separate RTX PRO 6000 GPUs (reported). Within a day the Hub had int4, GGUF and MLX conversions (measured by search). One is labelled BF16; the experts were released in FP8, so that is an upcast with no added precision (reasoned).

The report's engine section is worth reading on its own: one B300 frame ran 1,171 kernels in 4.0 ms, about 830 of which moved almost no data yet took 2.7 ms, and the fix was deleting framework kernels, not tuning maths (reported).

Licence

"Open-weight" is accurate. The report's abstract says "open-source", which is not. LICENSE is the Fish Audio Community License 1.0, dated 13 October 2026 (measured):

• Research and non-commercial use: free.
• Commercial use: free only while you and your affiliates have under USD 500,000 of gross annual revenue from all sources; cross it and you notify Fish Audio within 30 days and get 90 days of transition.
• A hosted service (an API, SDK or app whose primary purpose is TTS, voice generation or cloning for third parties) needs a separate licence at any revenue. Voicing your own product, or delivering finished audio as a studio, is allowed.
• No cloning an identifiable person without consent, no removing watermarks, and no using the weights or their outputs to train or improve another speech or foundational generative model. Distribution needs a Notice file and "Built with Fish Audio".

That is more permissive than S2's research-only licence, and the revenue line is generous for individuals and small studios. One loose end: the README's front matter still says license_name: fish-audio-research-license and links LICENSE.md, a file the repo does not contain (measured). The LICENSE file is the one that ships.

The ledger

What is genuinely new. The codebook-MTP decoder: full-width per-codebook layers that attend along time, so acoustic codes are predicted from the reference's acoustic codes. It is also cheaper per frame than the design it replaces, which is unusual for a model seven times larger. And editing, revision and streaming as prompt formats over one token stream, with fill-in-the-middle ordering solving the right-context problem.

What is carried over. The codec, byte for byte. Direction as plain text, now with scoped spans, and speaker tags promoted to real tokens.

What is overstated. "Most emotive and controllable" has no measurement behind it, and the one emotion benchmark shows a rival ahead. "SOTA" multilingual cloning is lowest in 5 of 24 languages. "Open-source" is a revenue-capped licence that excludes hosted TTS.

What I would watch. Whether SGLang-Omni grows the revision and infilling formats, and speaker similarity, the visible cost of dropping the similarity reward in the use most people try first.

Related: mixture-of-experts for total versus active parameters, AuK for a dedicated speech editor, vLLM-Omni for the other omni serving stack, and Nar TTS, which built emotion rewards and declined to switch them on.