Nemotron 3.5 Transcribe: 33.7 B stored, 3.6 B read per token, and what it costs to listen

mdjsonmcpcopy↗ai

2026-10-30 · 18 min · asr · mixture-of-experts · mamba · nvidia · quantization · speculative-decoding

Why read this

High· top 30%

Sums 1,395 header shapes into a parameter ledger, checks the 4.3 B claim, prices an hour of audio, times the loop guardrail. Model not run.

• Interactive explanations
• Original analysis
• Runs on a consumer GPU

·Speech & audio·OpenMDW-1.1·Practitioner model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

3 of 3: Mechanism carried by interactives built from real code or data

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 71 of 100, ranked 111 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

@NVIDIAAI introduced Nemotron 3.5 Transcribe as "an open-weight, promptable 30B MoE model with ~4B active parameters" that does transcription in 40 languages, speaker attribution, translation, contextual prompting and audio understanding in one checkpoint. The weights went up on 28 October.

I did not run it. It wants one H100 80 GB in BF16, or a 32 GB card in NVFP4, and I have four shared CPU cores. What I did instead is read everything a reader can read. That is the model card, the 67-page technical report, the configs, the licence, the tokenizer, the chat template and the safetensors header of the checkpoint (by HTTP range request, so no weights were downloaded). Then I ran the pieces that do not need a GPU: the log-mel front end on synthetic audio, the real tokenizer on sentences in twelve languages, and the loop guardrail the card tells you to apply.

Everything below is labelled measured (I computed it from a file or ran it), reported (NVIDIA's number, not re-run) or reasoned (my arithmetic on the other two). The repository is nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B.

What is in the checkpoint

The card gives the headline as "34B (4.3B active), including the audio encoder". The pieces, from the report's overview:

expand

Four things in that diagram matter for the rest of this piece.

The audio goes in as tokens. The ASR branch is a 32-layer Transformer at width 1,280 that eats 128-bin log-mel features. It stacks 8 frames into one, so each output frame covers 80 ms. A linear layer maps 1,280 to the language model's 2,688 and the result is spliced into the prompt where the <|audio|> token sits (reported, report §2.1; the 32 layers, 1,280, 128 mels and the factor 8 are also in config.json, measured).

Speaker identity is added to the audio features. A frozen diarizer, a Sortformer model of about 100 M parameters, estimates per-frame activity for up to eight speakers. The report binarizes it at 0.5, normalizes it, projects it through a fixed sinusoidal kernel of size 8 x 1,280 and adds the result to the ASR features. The language model therefore receives audio that is already tagged with who is speaking. It does not have to discover speakers from raw sound (reported).

Timestamps come after the text. A CTC adapter of about 50 M parameters reuses the encoder's 1,280-dimensional output, upsamples 80 ms to 40 ms frames and aligns the finished transcript to the audio. The text is never changed by it. Its vocabulary is 23,862 entries (measured, config.json), and I counted what is in it: 10 event tags such as <ev:laughter>, <ev:breath> and <ev:throat_cough>, 16 speaker tags, and paired <sty:stt:…> and <sty:end:…> style tags.

One draft module. Multi-token prediction (MTP) is one attention block and one MoE block, trained at two depths and reusable for up to eight draft tokens at inference (reported). There is also a separate draft model, -DFlash2, which I read from its config: 570,642,688 parameters, six Qwen3-architecture layers at width 2,688, blocks of 16 tokens, with target_layer_ids 1, 5, 19, 29, 41 and 51 in its config, which I read as the target layers whose hidden states it conditions on (measured config, reasoned reading).

Counting the parameters

I summed every tensor shape in the header. The file holds 33,697,781,031 elements; the Hub's own count is 33,697,781,029, so I am two scalars off. Switch the view below between what is stored, what one decoded token multiplies by, and the card's 4.3 B.

Routed experts are 29.375 B of the 33.698 B stored, 87%. Inside the 52-layer backbone they are 95%. An expert is two matrices, 2,688 x 1,856 in and 1,856 x 2,688 out, 9,977,856 parameters. The activation is squared ReLU, so there is no gate matrix.

The card's active count checks out, with one reading. A token at batch size 1 multiplies by 3.228 B parameters: six of 128 experts in each of 23 MoE layers (1.377 B), every Mamba mixer, shared expert, router and attention layer (1.498 B), and the 0.352 B output head (measured and reasoned). Adding the 0.352 B embedding table, which is a lookup and not a multiply, gives 3.580 B. The report says the backbone has "approximately 3.5B active parameters per token out of approximately 31.6B total". My stored backbone, including both tables, is 31.578 B. So the report counts the embedding and the output head, and "3.5 B" is 3.58 B rounded down. The audio side adds 0.781 B (ASR encoder 0.631, diarizer 0.099, CTC head 0.051). That makes 4.36 B, which the card calls 4.3 B. The numbers are consistent; the convention is not the one a reader would pick first.

Almost all of the file's 67 GB is experts that any one token ignores.

The backbone is mostly not attention

The language model is Nemotron 3.5 Lightning 30B-A3B: 52 layers, of which 23 are Mamba-2, 23 are MoE feed-forward and 6 are grouped-query attention.

expand

That pattern is in llm_backbone/config.json as layers_block_type, and I read it from there too (measured). The reason to build it this way is memory. Attention keeps a key and a value for every past token in every attention layer. Mamba layers keep a fixed state however long the input is. Here the attention layers have 32 query heads, only 2 key-value heads and a head size of 128, so each token adds 6 layers x 2 (K and V) x 2 heads x 128 x 2 bytes = 6,144 bytes of cache (measured from the config, arithmetic). The 23 Mamba layers add one fixed state of 64 heads x 64 x 128 x 4 bytes (fp32) per layer, 48.2 MB in all (reasoned, from the same config).

An hour of audio is the case this model is built for. With English speech at 2.5 words a second it is about 56,000 tokens, and the cache for it is 343 MB. The same heads in all 52 layers would need 8.7 times as much (reasoned). Use the controls below to see what it costs.

The front end sets the token count. I implemented the preprocessing from config.json in NumPy (25 ms Hann windows, a 10 ms hop, a 512-point FFT, 128 Slaney mel bins, log) and ran it on synthetic audio. A 3 s clip gives 301 frames, a 30 s chunk gives 3,001 frames, and stacking by 8 gives 376 tokens per chunk (measured). That is 12.5 tokens a second of audio. An hour is about 45,000 tokens, or 45,120 if each 30 s chunk is stacked on its own. The model was trained with a 65,536-token context and recordings up to an hour, so the trained limit has room for about 87 minutes of audio and no transcript at all (reasoned).

What a transcript adds is small by comparison. I ran the real tokenizer (131,072 entries plus the <|audio|> token at id 131,072) on one sentence per language. These are sentences I wrote, one per language, so read the ratios as rough:

At 2.5 words a second, English speech produces about 3 text tokens a second against 12.5 audio tokens a second. Hindi produces about 5.7. The tokenizer also gives the special tags single tokens: <spk:0> is id 100 and <ev:laughter> is id 114 (measured).

What a second of audio costs, next to Whisper

I wanted to know whether a 34 B model is a lot more work per recording than the usual baseline. I fetched the Whisper-large-v3 config and counted: 1.540 B parameters (encoder 629 M, decoder 839 M, embeddings 66 M). Then I priced one second of audio in multiply-adds for both, 2 FLOPs a parameter.

This is reasoned: it ignores cross-attention, convolutions, the Mamba scan and routing overhead, and assumes 3 text tokens a second. It says that a model with twice Whisper's active parameters and 22 times its stored ones costs 1.4 times as many operations per second of audio. Whisper encodes 50 frames a second and Nemotron's encoder emits 12.5, so the larger language model works on a quarter as many positions. The operation count is not the cost, since a Mamba-and-MoE decoder is bound by memory traffic, not arithmetic. It does tell you the size of the number NVIDIA's throughput claim has to carry: 2,238 seconds of audio per second is about 247 TFLOP/s of useful work by this count.

The prompt is the interface

I rendered the model's chat template with a sandboxed Jinja environment. With reasoning off, an audio request becomes:

<|im_start|>system
<|im_end|>
<|im_start|>user
<|audio|>Transcribe the following audio.<|im_end|>
<|im_start|>assistant
<think></think>

With reasoning on, the last line is <think> and a newline instead (measured). The system turn is empty for transcription. The card says to leave it out of the request and put the instruction in the user turn, and every transcription style is a different instruction. Verbatim keeps fillers and repetitions. A written form drops them. Context terms: <names and terms> biases vocabulary. Adding speaker tags <spk:0> <spk:1> and turning on word timestamps gives who-said-what. Please translate the given speech to <target language>. translates. These are in the card's prompt guide (reported).

It also states the limit plainly: this is an offline model. The 30-second chunking inside the encoder is a processing choice, not a streaming mode.

The most striking number in the card is about steering by prompt. On LibriSpeech test-clean speech mixed with far-field meeting audio, the near-field-only prompt gives a 4.78% word error rate and the plain single-speaker prompt gives 27.32% (reported, marked experimental). The same weights, two instructions.

Decoding: why batching helps and where it stops

A mixture-of-experts layer reads only the experts a token picked. A batch reads the union. At batch size 1 a token reads 6 of 128 experts in each layer. At batch size 64, if routing were spread evenly, 95% of experts are read every step. The calculator below does that arithmetic with the measured sizes: 9,977,856 parameters an expert, 23 MoE layers, 1.851 B parameters that are always read.

The bytes per parameter in the calculator are the repositories' file sizes divided by the 33.698 B parameters (measured, Hub blob sizes): BF16 67.50 GB, FP8 34.67 GB, NVFP4 21.56 GB. That is 2.003, 1.029 and 0.640 bytes per original parameter. The NVFP4 header shows why it is not 0.5: the routed experts are stored packed, 14.99 GB of U8 holding about 30 B 4-bit values, with 2.85 GB of FP8 tensors, which I take to include the block scales (I did not break that down), and 0.82 GB left in BF16. The MTP module ships as its own 1.34 GB FP8 file.

NVIDIA's throughput plot has the shape the calculator predicts, with the cause left for you to infer.

expand

Reported figures from the report and card: up to 2,238 seconds of audio per second with NVFP4 and DFlash2, and a 30 second clip transcribed in 0.13 s. The curves bend in the 64 to 256 range, which is where my calculator says nearly every expert is already being read. That is my explanation, not theirs. Real routers are not uniform, which would push the bend later.

Two caveats from NVIDIA itself, and both are easy to read past. The serving optimizations behind these numbers are in the Nemotron Transcribe NIM and "are not yet part of the open-source release", and the vLLM integration that shipped on release day is "expected to be roughly two times slower". The headline is a product number.

Speculation is where the gain comes from, and the report lists mean accepted length (MAL) for the shipped drafters, pooled over nine short-form English sets and three long-form corpora (reported, Table 18):

MAL counts the token the main model adds in each round, so at K = 7 a round yields 6.72 tokens on short audio: about 5.7 of the 7 drafts accepted, 82% (reasoned from the definition in the report). At K = 15 it is 9.0 of 15, 60%. The audio fixes most of the text, so there is little for the drafter to get wrong, and long-form audio loses part of that.

Quantization costs a little, and not evenly

The report evaluates all three checkpoints in one run on one GB300 (reported, Table 16). English held up. Other things did not:

NVFP4 is 32% of the BF16 file and gives up 0.03 points on English, 0.27 on the multilingual average and 0.86 on Hindi. The Mandarin meeting number moves 6.8 points, which the report itself says is too noisy to trust: 20 sessions, and little Mandarin multi-speaker training data. Its advice is FP8 where non-English accuracy matters and NVFP4 where memory or throughput is the constraint.

The loop guardrail has a price

The card admits the model can fail to stop. In rare cases it loops. The report puts a number on it: on the long-form track, loops hit 7 of 451 recordings, and without the guardrail the mean WER would be 12.87% instead of 10.98% (reported). The guardrail is two regular expressions, adapted from Qwen3-ASR (Apache-2.0, with the notice in the repository): collapse any character repeated 21 or more times, then collapse any block of 1 to 4,096 characters repeated 20 or more times.

It runs on every transcript, so I timed it. I generated a pseudo-transcript of Zipf-distributed words and ran both patterns at four lengths on this machine, which has four shared cores and was running at low priority (measured):

The cost grows with the length times the block window, because the block pattern tries every block size at every position. At the reported aggregate throughput an hour of audio is 1.6 s of GPU time (3,600 s at 2,238 audio seconds a second); that is throughput across many requests, not the latency of one. My box is slow and busy, and a fast core would narrow the gap, but the post-processing is not free and it does not speed up with the GPU. A pipeline that runs it on every hour-long transcript should expect seconds, not microseconds.

It also has collateral damage, which I tested on synthetic text (measured). It removes a 40-fold block loop and leaves a 10-fold one alone, as intended. It also turns a legitimate ha ha ha ... repeated 25 times into a single ha, and no. repeated 21 times into one no. . A transcript of a long laugh or a count is edited by the very step meant to fix the model. The model has <ev:laughter> for exactly that case, which is a reason to prompt for events.

What the benchmarks say

NVIDIA's table compares against the Hugging Face Open ASR leaderboard as of 21 October 2026 (reported, I did not re-run any of it). Mean WER over nine English public sets:

That makes it the most accurate open-weights model in the table by 0.23 points. The report says seven commercial systems are ahead of it, the best at 3.88. On the long-form track it is 10.98% against 13.37% for the next open model. On multi-speaker full sessions the mean cpWER is 15.80 against 16.83. These are vendor results on a vendor's evaluation setup, with the loop guardrail applied, and the card's own words are "results captured on October 12, 2026". The training mixture is described as 1.72 million unique hours of audio (6.53 million counting repeats), so contamination of public test sets is a question I cannot answer from the card.

The licence

The weights are under the OpenMDW License Agreement 1.1, and I read the LICENSE file. It grants permission to deal in the model materials "without restriction". If you redistribute, you keep a copy of the agreement and the notices. Rights end if you sue someone alleging the materials infringe a patent or copyright, unless it was a counterclaim. It puts no restriction on outputs. And you are "solely responsible" for clearing anyone else's rights in the materials. The repository is not gated. Two third-party components carry their own Apache-2.0 notices (the guardrail code and a vLLM patch snippet). I am not a lawyer; this is what the text says.

What I did not run, and what would settle it

I did not run the model, so none of the accuracy, speed or MAL numbers is mine. Three open questions:

• Is the throughput reachable? The report's 2,238x is for NVIDIA's NIM. The open release is said to be about half that speed. A measurement on the open vLLM path, on one named GPU, would turn "up to" into a number.
• Do the experts behave evenly? My batch calculator assumes uniform routing. Logging real expert counts on a speech workload would say where the curve really bends.
• How often does it loop on your audio? 7 in 451 is a rate on earnings calls and interviews. Noisy phone audio in another language might differ, and so might the cost of the fix.

If I had a GPU, the first thing I would do is run the card's ask function on a clip with and without the near-field prompt and print the 4.78% against 27.32% gap on my own audio. The prompt as interface is the part of this model most likely to change what people build with it.