# Nemotron 3.5 Transcribe: 33.7 B stored, 3.6 B read per token, and what it costs to listen

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/nemotron-3-5-transcribe
> date: 2026-10-30
> tags: speech-recognition, mixture-of-experts, mamba, nvidia, quantization, speculative-decoding

[@NVIDIAAI](https://x.com/NVIDIAAI/status/2115462300241141882) introduced Nemotron 3.5 Transcribe as "an open-weight,
promptable 30B MoE model with ~4B active parameters" that does transcription in 40 languages, speaker attribution,
translation, contextual prompting and audio understanding in one checkpoint. The weights went up on 28 October.

I did not run it. It wants one H100 80 GB in BF16, or a 32 GB card in NVFP4, and I have four shared CPU cores. What I
did instead is read everything a reader can read. That is the model card, the 67-page
[technical report](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Transcribe-Technical-Report.pdf),
the configs, the licence, the tokenizer, the chat template and the safetensors header of the checkpoint (by HTTP range
request, so no weights were downloaded). Then I ran the pieces that do not need a GPU: the log-mel front end on synthetic
audio, the real tokenizer on sentences in twelve languages, and the loop guardrail the card tells you to apply.

Everything below is labelled **measured** (I computed it from a file or ran it), **reported** (NVIDIA's number, not
re-run) or **reasoned** (my arithmetic on the other two). The repository is
[`nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B`](https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B).

<ModelCard repo="nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B" />

## What is in the checkpoint

The card gives the headline as "34B (4.3B active), including the audio encoder". The pieces, from the report's overview:

<Figure
  src="https://media.thesatyajit.com/ai/articles/nemotron-3-5-transcribe/fig1.png?v=3dc2ad7d7bf6d9d0"
  alt="Block diagram. Audio at 16 kHz becomes log-mel features in 30-second chunks. A trainable 32-layer Transformer ASR branch and a frozen Sortformer diarization branch feed a speaker kernel and a linear projection from 1,280 to 2,688 dimensions. The result is spliced into the prompt of a hybrid language model with 23 Mamba-2, 23 MoE and 6 attention layers, which has a language-model head and a multi-token-prediction module. A CTC timestamp adapter works on the ASR output after generation."
  caption="Nemotron 3.5 Transcribe: two audio branches, a projection into the prompt, a hybrid MoE language model, a draft module and a timestamp adapter that runs after decoding (Technical report, Figure 2)."
/>

Four things in that diagram matter for the rest of this piece.

**The audio goes in as tokens.** The ASR branch is a 32-layer Transformer at width 1,280 that eats 128-bin log-mel
features. It stacks 8 frames into one, so each output frame covers 80 ms. A linear layer maps 1,280 to the language
model's 2,688 and the result is spliced into the prompt where the `<|audio|>` token sits (**reported**, report §2.1;
the 32 layers, 1,280, 128 mels and the factor 8 are also in `config.json`, **measured**).

**Speaker identity is added to the audio features.** A frozen diarizer, a Sortformer model of about 100 M parameters,
estimates per-frame activity for up to eight speakers. The report binarizes it at 0.5, normalizes it, projects it through a fixed
sinusoidal kernel of size 8 x 1,280 and adds the result to the ASR features. The language model therefore receives
audio that is already tagged with who is speaking. It does not have to discover speakers from raw sound (**reported**).

**Timestamps come after the text.** A CTC adapter of about 50 M parameters reuses the encoder's 1,280-dimensional
output, upsamples 80 ms to 40 ms frames and aligns the finished transcript to the audio. The text is never changed by
it. Its vocabulary is 23,862 entries (**measured**, `config.json`), and I counted what is in it: 10 event tags such as
`<ev:laughter>`, `<ev:breath>` and `<ev:throat_cough>`, 16 speaker tags, and paired `<sty:stt:…>` and `<sty:end:…>`
style tags.

**One draft module.** Multi-token prediction (MTP) is one attention block and one MoE block, trained at two depths and
reusable for up to eight draft tokens at inference (**reported**). There is also a separate draft model,
`-DFlash2`, which I read from its config: 570,642,688 parameters, six Qwen3-architecture layers at width 2,688, blocks
of 16 tokens, with `target_layer_ids` 1, 5, 19, 29, 41 and 51 in its config, which I read as the target layers whose hidden states it conditions on (**measured** config, **reasoned** reading).

### Counting the parameters

I summed every tensor shape in the header. The file holds 33,697,781,031 elements; the Hub's own count is
33,697,781,029, so I am two scalars off. Switch the view below between what is stored, what one decoded token multiplies
by, and the card's 4.3 B.

<ParamLedger />

Routed experts are 29.375 B of the 33.698 B stored, 87%. Inside the 52-layer backbone they are 95%. An expert is two
matrices, 2,688 x 1,856 in and 1,856 x 2,688 out, 9,977,856 parameters. The activation is squared ReLU, so there is no
gate matrix.

The card's active count checks out, with one reading. A token at batch size 1 multiplies by 3.228 B parameters: six of
128 experts in each of 23 MoE layers (1.377 B), every Mamba mixer, shared expert, router and attention layer
(1.498 B), and the 0.352 B output head (**measured** and **reasoned**). Adding the 0.352 B embedding table, which is a
lookup and not a multiply, gives 3.580 B. The report says the backbone has "approximately 3.5B active parameters per
token out of approximately 31.6B total". My stored backbone, including both tables, is 31.578 B. So the report counts the
embedding and the output head, and "3.5 B" is 3.58 B rounded down. The audio side adds 0.781 B (ASR encoder 0.631,
diarizer 0.099, CTC head 0.051). That makes 4.36 B, which the card calls 4.3 B. The numbers are consistent; the
convention is not the one a reader would pick first.

Almost all of the file's 67 GB is experts that any one token ignores.

## The backbone is mostly not attention

The language model is Nemotron 3.5 Lightning 30B-A3B: 52 layers, of which 23 are Mamba-2, 23 are MoE feed-forward and 6
are grouped-query attention.

<Figure
  src="https://media.thesatyajit.com/ai/articles/nemotron-3-5-transcribe/fig2.png?v=41537546e3dd61a8"
  alt="A row of 52 narrow bars in blue, green and orange for Mamba-2, MoE feed-forward and attention layers, followed by a dashed box with two bars for the multi-token-prediction module."
  caption="The 52-layer pattern: Mamba-2 (blue), MoE (green), attention (orange), and the MTP module in the dashed box (Technical report, Figure 3)."
/>

That pattern is in `llm_backbone/config.json` as `layers_block_type`, and I read it from there too (**measured**). The
reason to build it this way is memory. Attention keeps a key and a value for every past token in every attention
layer. Mamba layers keep a fixed state however long the input is. Here the attention layers have 32 query heads, only
2 key-value heads and a head size of 128, so each token adds 6 layers x 2 (K and V) x 2 heads x 128 x 2 bytes =
6,144 bytes of cache (**measured** from the config, arithmetic). The 23 Mamba layers add one fixed state of 64 heads x
64 x 128 x 4 bytes (fp32) per layer, 48.2 MB in all (**reasoned**, from the same config).

An hour of audio is the case this model is built for. With English speech at 2.5 words a second it is about 56,000 tokens,
and the cache for it is 343 MB. The same heads in all 52 layers would need 8.7 times as much (**reasoned**). Use the controls
below to see what it costs.

<AudioBudget />

The front end sets the token count. I implemented the preprocessing from `config.json` in NumPy (25 ms Hann windows, a
10 ms hop, a 512-point FFT, 128 Slaney mel bins, log) and ran it on synthetic audio. A 3 s clip gives 301 frames, a
30 s chunk gives 3,001 frames, and stacking by 8 gives 376 tokens per chunk (**measured**). That is 12.5 tokens a second
of audio. An hour is about 45,000 tokens, or 45,120 if each 30 s chunk is stacked on its own. The model was trained with
a 65,536-token context and recordings up to an hour, so the trained limit has room for about 87 minutes of audio and no
transcript at all (**reasoned**).

What a transcript adds is small by comparison. I ran the real tokenizer (131,072 entries plus the `<|audio|>` token at
id 131,072) on one sentence per language. These are sentences I wrote, one per language, so read the ratios as rough:

| Language | Words | Tokens | Tokens per word |
|---|---:|---:|---:|
| English | 14 | 17 | 1.21 |
| German | 13 | 17 | 1.31 |
| Spanish | 17 | 21 | 1.24 |
| French | 17 | 23 | 1.35 |
| Vietnamese | 20 | 29 | 1.45 |
| Arabic | 12 | 18 | 1.50 |
| Russian | 12 | 28 | 2.33 |
| Hindi | 14 | 32 | 2.29 |

At 2.5 words a second, English speech produces about 3 text tokens a second against 12.5 audio tokens a second.
Hindi produces about 5.7. The tokenizer also gives the special tags single tokens: `<spk:0>` is id 100 and
`<ev:laughter>` is id 114 (**measured**).

## What a second of audio costs, next to Whisper

I wanted to know whether a 34 B model is a lot more work per recording than the usual baseline. I fetched the
Whisper-large-v3 config and counted: 1.540 B parameters (encoder 629 M, decoder 839 M, embeddings 66 M). Then I priced one
second of audio in multiply-adds for both, 2 FLOPs a parameter.

| Per second of audio | Whisper-large-v3 | Nemotron 3.5 Transcribe |
|---|---:|---:|
| Frames into the encoder | 50 | 12.5 |
| Encoder (incl. attention over its window) | 75.2 GFLOP | 16.5 GFLOP (ASR) + 2.8 (diarizer) |
| Language model reading the audio | none | 71.9 GFLOP (12.5 tokens x 2 x 2.875 B) |
| Decoding the text at 3 tokens a second | 5.4 GFLOP | 19.4 GFLOP |
| **Total** | **80.6 GFLOP** | **110.6 GFLOP** |

This is **reasoned**: it ignores cross-attention, convolutions, the Mamba scan and routing overhead, and assumes 3
text tokens a second. It says that a model with twice Whisper's active parameters and 22 times its stored ones costs
1.4 times as many operations per second of audio. Whisper encodes 50 frames a second and Nemotron's encoder emits 12.5,
so the larger language model works on a quarter as many positions. The operation count is not the cost, since
a Mamba-and-MoE decoder is bound by memory traffic, not arithmetic. It does tell you the size of the number NVIDIA's
throughput claim has to carry: 2,238 seconds of audio per second is about 247 TFLOP/s of useful work by this count.

## The prompt is the interface

I rendered the model's chat template with a sandboxed Jinja environment. With reasoning off, an audio request becomes:

```text
<|im_start|>system
<|im_end|>
<|im_start|>user
<|audio|>Transcribe the following audio.<|im_end|>
<|im_start|>assistant
<think></think>
```

With reasoning on, the last line is `<think>` and a newline instead (**measured**). The system turn is empty for
transcription. The card says to leave it out of the request and put the instruction in the user turn, and every
transcription style is a different instruction. Verbatim keeps fillers and repetitions. A written form drops them.
`Context terms: <names and terms>` biases vocabulary. Adding speaker tags `<spk:0> <spk:1>` and turning on word
timestamps gives who-said-what. `Please translate the given speech to <target language>.` translates. These are in the
card's prompt guide (**reported**).

It also states the limit plainly: this is an offline model. The 30-second chunking inside the encoder is a processing
choice, not a streaming mode.

The most striking number in the card is about steering by prompt. On LibriSpeech test-clean speech mixed with far-field
meeting audio, the near-field-only prompt gives a 4.78% word error rate and the plain single-speaker prompt gives 27.32%
(**reported**, marked experimental). The same weights, two instructions.

## Decoding: why batching helps and where it stops

A mixture-of-experts layer reads only the experts a token picked. A batch reads the union. At batch size 1 a token reads
6 of 128 experts in each layer. At batch size 64, if routing were spread evenly, 95% of experts are read every step.
The calculator below does that arithmetic with the measured sizes: 9,977,856 parameters an expert, 23 MoE layers,
1.851 B parameters that are always read.

<ExpertCoverage />

The bytes per parameter in the calculator are the repositories' file sizes divided by the 33.698 B parameters
(**measured**, Hub blob sizes): BF16 67.50 GB, FP8 34.67 GB, NVFP4 21.56 GB. That is 2.003, 1.029 and 0.640 bytes per
original parameter. The NVFP4 header shows why it is not 0.5: the routed experts are stored packed, 14.99 GB of `U8`
holding about 30 B 4-bit values, with 2.85 GB of FP8 tensors, which I take to include the block scales (I did not break that down), and 0.82 GB left in
BF16. The MTP module ships as its own 1.34 GB FP8 file.

NVIDIA's throughput plot has the shape the calculator predicts, with the cause left for you to infer.

<Figure
  src="https://media.thesatyajit.com/ai/articles/nemotron-3-5-transcribe/fig3.png?v=23d1d14bbb0c452c"
  alt="Two panels of throughput in audio seconds per second against median request latency on a log axis, for 30-second and 10-minute audio. Lines for BF16, FP8 and NVFP4, each with no speculation, MTP with 8 draft tokens, and DFlash2 with 15. Throughput climbs steeply with concurrent requests up to 64, then flattens. The best 30-second line, NVFP4 with DFlash2, reaches about 2,240 at 256 requests."
  caption="Throughput (RTFx) against median request latency on one RTX PRO 6000 Blackwell, for 30-second and 10-minute audio, by precision and speculative mode. Points are 1, 4, 16, 64, 128 and 256 concurrent requests (Technical report, Figure 8)."
/>

Reported figures from the report and card: up to 2,238 seconds of audio per second with NVFP4 and DFlash2, and a 30
second clip transcribed in 0.13 s. The curves bend in the 64 to 256 range, which is where my calculator says nearly
every expert is already being read. That is my explanation, not theirs. Real routers are not uniform, which would push
the bend later.

Two caveats from NVIDIA itself, and both are easy to read past. The serving optimizations behind these numbers are
in the Nemotron Transcribe NIM and "are not yet part of the open-source release", and the vLLM integration that shipped
on release day is "expected to be roughly two times slower". The headline is a product number.

Speculation is where the gain comes from, and the report lists mean accepted length (MAL) for the shipped drafters, pooled over nine
short-form English sets and three long-form corpora (**reported**, Table 18):

| Draft model | Draft tokens K | Short-form MAL | Long-form MAL |
|---|---:|---:|---:|
| MTP | 2 | 2.93 | 2.82 |
| MTP | 7 | 6.72 | 5.98 |
| MTP | 15 | 10.02 | 7.84 |
| DFlash2 | 7 | 6.39 | 4.05 |
| DFlash2 | 15 | 8.54 | 4.80 |

MAL counts the token the main model adds in each round, so at K = 7 a round yields 6.72 tokens on short audio: about 5.7
of the 7 drafts accepted, 82% (**reasoned** from the definition in the report). At K = 15 it is 9.0 of 15, 60%. The audio fixes
most of the text, so there is little for the drafter to get wrong, and long-form audio loses part of that.

## Quantization costs a little, and not evenly

The report evaluates all three checkpoints in one run on one GB300 (**reported**, Table 16). English held up. Other things
did not:

| Benchmark | BF16 | FP8 | NVFP4 |
|---|---:|---:|---:|
| Open ASR Leaderboard, 9 English sets (WER %) | 4.70 | 4.74 | 4.73 |
| Multilingual Open ASR Leaderboard (WER %) | 4.63 | 4.75 | 4.90 |
| FLEURS, 37 languages (WER %) | 9.31 | 9.38 | 9.67 |
| Monsoon, Hindi (OIWER %) | 9.27 | 9.67 | 10.13 |
| AMI SDM, far-field meetings (cpWER %) | 17.64 | 17.85 | 18.75 |
| AISHELL-4 Mandarin meetings (cpCER %) | 27.64 | 26.96 | 34.44 |

NVFP4 is 32% of the BF16 file and gives up 0.03 points on English, 0.27 on the multilingual average and 0.86 on Hindi.
The Mandarin meeting number moves 6.8 points, which the report itself says is too noisy to trust: 20 sessions, and
little Mandarin multi-speaker training data. Its advice is FP8 where non-English accuracy matters and NVFP4 where memory
or throughput is the constraint.

## The loop guardrail has a price

The card admits the model can fail to stop. In rare cases it loops. The report puts a number on it: on the long-form
track, loops hit 7 of 451 recordings, and without the guardrail the mean WER would be 12.87% instead of 10.98%
(**reported**). The guardrail is two regular expressions, adapted from Qwen3-ASR (Apache-2.0, with the notice in the
repository): collapse any character repeated 21 or more times, then collapse any block of 1 to 4,096 characters
repeated 20 or more times.

It runs on every transcript, so I timed it. I generated a pseudo-transcript of Zipf-distributed words and ran both
patterns at four lengths on this machine, which has four shared cores and was running at low priority (**measured**):

| Transcript length | Time |
|---|---:|
| 10,000 characters (~1,700 words) | 1.2 s |
| 50,000 characters (~8,300 words, about an hour of speech) | 9.3 s |
| 200,000 characters | 48.4 s |
| 400,000 characters | 79.2 s |

The cost grows with the length times the block window, because the block pattern tries every block size at every
position. At the reported aggregate throughput an hour of audio is 1.6 s of GPU time (3,600 s at 2,238 audio seconds a
second); that is throughput across many requests, not the latency of one. My box is slow and busy, and a fast core would narrow the gap, but the post-processing is not free and it does not
speed up with the GPU. A pipeline that runs it on every hour-long transcript should expect seconds, not microseconds.

It also has collateral damage, which I tested on synthetic text (**measured**). It removes a 40-fold block loop and leaves a
10-fold one alone, as intended. It also turns a legitimate `ha ha ha ...` repeated 25 times into a single `ha`, and
`no. ` repeated 21 times into one `no. `. A transcript of a long laugh or a count is edited by the very step meant to fix
the model. The model has `<ev:laughter>` for exactly that case, which is a reason to prompt for events.

## What the benchmarks say

NVIDIA's table compares against the Hugging Face Open ASR leaderboard as of 21 October 2026 (**reported**, I did not
re-run any of it). Mean WER over nine English public sets:

| Model | Mean WER % |
|---|---:|
| Nemotron 3.5 Transcribe | 4.71 |
| Qwen3-ASR-1.7B | 4.94 |
| Granite-Speech-5.0-2B | 5.06 |
| Canary-Qwen-2.5B | 5.38 |
| Parakeet-TDT-0.6B-v3 | 5.58 |
| Whisper-large-v3 | 6.66 |
| Best commercial system in the table (OtterAI speech-v3.1) | 3.88 |

That makes it the most accurate open-weights model in the table by 0.23 points. The report says seven commercial systems are
ahead of it, the best at 3.88. On
the long-form track it is 10.98% against 13.37% for the next open model. On multi-speaker full sessions the mean
cpWER is 15.80 against 16.83. These are vendor results on a vendor's evaluation setup, with the loop guardrail applied, and
the card's own words are "results captured on October 12, 2026". The training mixture is described as 1.72 million
unique hours of audio (6.53 million counting repeats), so contamination of public test sets is a question I cannot
answer from the card.

## The licence

The weights are under the [OpenMDW License Agreement 1.1](https://openmdw.ai/license/1-1/), and I read the `LICENSE` file. It
grants permission to deal in the model materials "without restriction". If you redistribute, you keep a copy of the
agreement and the notices. Rights end if you sue someone alleging the materials infringe a patent or copyright, unless
it was a counterclaim. It puts no restriction on outputs. And you are "solely responsible" for clearing anyone else's
rights in the materials. The repository is not gated. Two third-party components carry their own Apache-2.0 notices (the
guardrail code and a vLLM patch snippet). I am not a lawyer; this is what the text says.

## What I did not run, and what would settle it

I did not run the model, so none of the accuracy, speed or MAL numbers is mine. Three open questions:

- **Is the throughput reachable?** The report's 2,238x is for NVIDIA's NIM. The open release is said to be about half that
  speed. A measurement on the open vLLM path, on one named GPU, would turn "up to" into a number.
- **Do the experts behave evenly?** My batch calculator assumes uniform routing. Logging real expert counts on a speech
  workload would say where the curve really bends.
- **How often does it loop on your audio?** 7 in 451 is a rate on earnings calls and interviews. Noisy phone audio in another
  language might differ, and so might the cost of the fix.

If I had a GPU, the first thing I would do is run the card's `ask` function on a clip with and without the near-field
prompt and print the 4.78% against 27.32% gap on my own audio. The prompt as interface is the part of this model most
likely to change what people build with it.