# Nemotron 3 Diarization: who spoke when, sorted by who spoke first
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/nemotron-3-diarization
> date: 2026-09-26
> tags: audio, speech, small-models, open-weights, benchmarks, explainer
A transcript that is word-perfect and speaker-blind is half a transcript. You can read that someone
promised the report by Friday; you cannot tell who. NVIDIA's answer, released on September 23 as
[`nvidia/Nemotron-3-Diarization`](https://huggingface.co/nvidia/Nemotron-3-Diarization), is a
speaker diarizer: audio in, something like "speaker 2 talked from 12.4 s to 15.1 s" out, for up to
eight speakers, streaming or offline, from one checkpoint. The announcement leads with a leaderboard:
"ranked #1 in VoiceArena's Diarization leaderboard with a 14.72% Diarization Error Rate (DER)". A
Korean repost that did the rounds compressed it further, to three claims: 100M (0.1B) parameters,
real-time, and overwhelmingly #1 on speaker-diarization benchmarks.
Each of those can be checked, and most of them without running anything. I read the parameter count
out of the `model.safetensors` header over an HTTP range request, the training recipe out of the
`model_config.yaml` inside the `.nemo` tarball (its first 4 KB, same trick), the streaming logic
out of NeMo's source, and the benchmark claims out of NVIDIA's model card, VoiceArena's launch page
and Argmax's OpenBench tables. The short version: the parameter count is right to the rounding,
"real-time" is right on a data-centre GPU and unmeasured anywhere else, and "#1" is true on the
leaderboard NVIDIA cites and a one-point edge on the other public one.
## Who spoke when, and how it is scored
A diarizer's output is a matrix: time frames down one axis, speakers across the other, and in each
cell the probability that this speaker is talking now. Nemotron 3 Diarization emits exactly that,
a `[T, 8]` tensor of sigmoid probabilities, one row every 10 ms. Thresholded and merged into
segments, it becomes the familiar `start end speaker_k` list. Two things about that matrix matter
for everything that follows. Nothing stops two cells in the same row from both being on, so
overlapping speech is representable. And the column labels are anonymous: "speaker 2" is not a
person, it is a slot.
The standard score is the diarization error rate. For every scoring frame, count the speakers
the reference says are talking, $N_\text{ref}$, the speakers the system says are talking,
$N_\text{hyp}$, and how many of the system's speakers are the right ones, $N_\text{ok}$ (after the
scorer has found the best one-to-one mapping between system slots and real speakers). Then
$$
\text{DER} = \frac{\sum_t \Big[\underbrace{\max(0, N_\text{ref}-N_\text{hyp})}_{\text{missed speech}} + \underbrace{\max(0, N_\text{hyp}-N_\text{ref})}_{\text{false alarm}} + \underbrace{\min(N_\text{ref}, N_\text{hyp}) - N_\text{ok}}_{\text{speaker confusion}}\Big]}{\sum_t N_\text{ref}}
$$
The denominator is speaker-time, not wall-clock time: a second where two people talk counts as two.
Two protocol knobs move the number a lot. A **collar** excuses a window around every reference
boundary (0.25 s either side is traditional); the card scores DIHARD III, AMI, AliMeeting and
NOTSOFAR1 with no collar and CALLHOME with 0.25 s. And **overlap** can be scored or skipped; every
number in the card scores it. NVIDIA's evaluation subcard says it plainly: "DER is not fully
interpretable without these settings."
The widget is a made-up 60-second conversation between three people, A, B and C, with three
overlaps: A and B for 3 seconds, A and B again for 2, B and C for 4. That puts 61 seconds of
speaker time into 60 seconds of audio. Each slider injects one error type, and the readout does the
arithmetic above. The button is the point. A system that can only say one speaker per frame loses
one speaker in every overlapped frame, 9 seconds here, and C's second turn, which sits entirely
inside B's, disappears. That is 14.8% DER before the system has made a single mistake it could have
avoided. Overlap is not an edge case in meetings, and a system that cannot represent it starts
every meeting in the hole.
## Two ways to build a diarizer
The classic pipeline is a cascade. A voice-activity detector finds speech. The speech is cut into
short segments. A speaker-embedding network turns each segment into a vector. A clustering step
groups the vectors, and each cluster becomes a speaker. It has real strengths: it handles any
number of speakers, and every stage can be swapped. Its weakness is baked into the output format:
one segment, one embedding, one cluster, one speaker. Overlapped speech gets one label at best,
which is the button in the widget above. Production pipelines such as pyannote's now run a small
neural segmentation model over short windows first and cluster across windows afterwards, which is
a hybrid built to patch exactly this.
End-to-end neural diarization (EEND) throws the cascade away and treats diarization as frame-wise
multi-label classification: one network, one sigmoid per speaker slot per frame, trained with
binary cross-entropy. Overlap is free. The catch is the labels. If the reference says A spoke first
and B second, is the model wrong for putting A in slot 1? It shouldn't be, so training uses a
permutation-invariant loss (PIL): score every assignment of reference speakers to output slots and
keep the cheapest,
$$
\mathcal{L}_\text{PIL}(\mathbf{Y}, \mathbf{P}) = \min_{\pi \in \Pi} \mathcal{L}_\text{BCE}(\mathbf{Y}_\pi, \mathbf{P}),
$$
where $\mathbf{Y}$ is the $K \times T$ reference activity, $\mathbf{P}$ the predicted one, and
$\Pi$ the $K!$ permutations of the speaker rows. With eight slots that is 40,320 permutations per
training example, which is why implementations solve it as an assignment problem instead of
enumerating. The larger problem is at inference in a stream: chunk 7 has no idea which slot chunk 6 gave to whom, so a
streaming EEND needs some way to re-align slots across chunks, every chunk.
## Sorting speakers by arrival
Sortformer ([Park et al., 2024](https://arxiv.org/abs/2409.06656)), the design Nemotron 3
Diarization inherits, removes the ambiguity instead of searching over it. Its rule: output slot 1
is whoever spoke first in the audio, slot 2 whoever spoke second, and so on. With the reference
rows sorted by each speaker's first segment, $\eta$, the target is fixed and the loss is plain
BCE, the paper's Sort Loss:
$$
\mathcal{L}_\text{Sort} = \frac{1}{K} \sum_{k=1}^{K} \mathcal{L}_\text{BCE}\big(\mathbf{y}_{\eta(k)}, \mathbf{q}_k\big).
$$
The paper is candid that "arrival time estimation is not always correct", more so as speaker counts
rise, so it trains on a mix, $\alpha \cdot \mathcal{L}_\text{Sort} + (1-\alpha) \cdot
\mathcal{L}_\text{PIL}$. The model also needs positional information to sort at all: attention
without positions is permutation-equivariant, so it could not tell first from second.
The `.nemo` config shows how NVIDIA weighted it this time, and what it added:
| config key (`model_config.yaml`) | Nemotron 3 Diarization | `diar_streaming_sortformer_4spk-v2.1` |
|---|---:|---:|
| `pil_weight` / `ats_weight` | 0.75 / 0.25 | 0.5 / 0.5 |
| `activity_weight` | 0.5 | absent |
| `phantom_weight` | 0.1 | absent |
| `max_num_of_spks` | 8 | 4 |
| `high_resolution` | `true` (10 ms output) | absent (80 ms output) |
The two new losses live in NeMo's `aux_diarization_loss.py` on `main`. `ActivityLoss` is a
three-class cross-entropy on a small auxiliary head: is this frame silence, one speaker, or
overlap? It needs no speaker alignment at all, so it teaches overlap detection without touching
the permutation problem. `PhantomLoss` penalises confident activity (sigmoid above 0.25) in any
output slot that has no speaker in the reference for the whole segment: a direct penalty on
inventing a fifth speaker in a four-person meeting. The Transformers-format safetensors carries no
activity head, so at inference the eight speaker sigmoids are the whole output.
Arrival order pays off at inference. If slot $k$ always means "the $k$-th voice to appear", then a
model that sees the earlier audio again can put the same voice back into the same slot without any
matching step. That is the whole streaming design.
## What is in the checkpoint
The safetensors header is 48,568 bytes of JSON describing 417 float32 tensors. Summed:
| component (tensor prefix) | shape | parameters |
|---|---|---:|
| `model.audio_tower.embedder.projection` | 512 × 1024 | 524,288 |
| `model.audio_tower.layers.{0..30}` | 31 × 3,150,848 | 97,676,288 |
| layer norms, input and final | | 2,048 |
| `model.proj` (512 → 192) | | 98,496 |
| `model.upsampler.conv` (192 → 1536, kernel 3) | | 886,272 |
| `classifier.dense` + `classifier.out_proj` (192 → 192 → 8) | | 38,600 |
| `silence_embeds` | 512 | 512 |
| **total** | | **99,226,504** |
So "100M" is 99.2M, float32, 396,954,592 bytes on disk. The shapes explain the architecture
better than the diagram does. The input is 16 kHz mono audio, turned into 128 mel bins every 10 ms.
The input projection is 1024 wide because 128 mel bins times 8 stacked 10 ms frames is 1024: the
model downsamples by concatenation, not convolution, producing one 512-wide vector per 80 ms. Each of the 31 layers is a plain pre-norm Transformer block (8
heads, 2,048-wide MLP, no bias on Q, K and V) with rotary position embeddings. After the stack, a
projection drops to 192 dimensions, and the upsampler turns 192 channels into 1,536, which is 8 ×
192: a sub-pixel convolution that unfolds each 80 ms frame into eight 10 ms frames. NeMo initialises
it to an identity, so every 10 ms frame starts as a copy of its 80 ms parent and training learns the
refinement. A two-layer head then gives eight sigmoids per 10 ms. The 512-wide
`silence_embeds` is a learned stand-in used to pad the speaker cache (more on that below).
The predecessor it is benchmarked against is a different shape. `diar_streaming_sortformer_4spk-v2.1`
stacks a 17-layer FastConformer (convolution modules, a depthwise-striding downsampler) under a
separate 18-layer, 192-wide Transformer, for 117M parameters per its card. Nemotron 3 Diarization
replaces both with one homogeneous 31-layer Transformer, initialised from a Transformer-based NEST
self-supervised checkpoint ([Huang et al., 2024](https://arxiv.org/abs/2408.13106)). It is smaller
by about 18M parameters, has twice the speaker slots, and, as the next section shows, is cheaper
per step.
The GA `.nemo` is 198,676,480 bytes, about 2 bytes per parameter, where the gated preview's was
397,199,360; a `q8_0` GGUF of 107,012,128 bytes ships alongside, presumably for the C++ runtime.
Training, per the card: 8 nodes of 8 A100-80GB, offline on simulated mixtures first, then
streaming fine-tuning on real and simulated audio together. The data is about 10,000 hours of real
conversations plus 82,611 hours of mixtures with one to eight speakers, simulated with the FastMSS
toolkit from [Polok et al.](https://arxiv.org/abs/2605.15442), much of the source audio licensed
from David AI. NVIDIA's blog attributes a 0.77-point DER drop (11.19% to 10.42%) to adding the David
AI data, without saying which test set that compound figure pools.
## Streaming: a cache sorted by speaker
Streaming Sortformer ([Medennikov et al., 2025](https://arxiv.org/abs/2507.18446)) turns the
arrival-order property into memory. Each step, the model sees four things concatenated along time,
all as 512-wide post-stacking embeddings:
1. the **speaker cache** (Arrival-Order Speaker Cache, AOSC): up to 264 frames kept from the whole
past, grouped by speaker slot in arrival order;
2. the **FIFO**: the most recent past frames, in order;
3. the **chunk**: the new frames to be scored;
4. the **right context**: a few frames of look-ahead, attended to but scored on the next step.
All of it goes through all 31 layers with full attention. Because the cache is laid out speaker by
speaker in arrival order, the first voices the model reads every step are slot 1's, then slot 2's,
and a returning speaker lands in their old slot without a matching step. That is what the
Sortformer training bought.
When the FIFO overflows, its oldest frames (222 of them in the streaming configs) are merged into
the cache and the cache is compressed back to 264. Reading `_compress_spkcache` in NeMo's
`sortformer_modules.py`, each frame gets a per-speaker score
$$
s_k = \log p_k - \log(1-p_k) + \sum_{j} \log(1-p_j) - \log 0.5,
$$
with every probability clamped at 0.25 before the logs. It is positive when speaker $k$ is
confidently on and everyone else is off: clean, single-speaker evidence. Non-speech is dropped,
overlapped frames are dropped once a speaker has 16 clean ones, the newest frames get a 0.05 nudge,
and two boosting passes guarantee each of the eight speakers up to 24 of the 264 slots (1.92 s
of their voice) before the rest go to the highest scores. With eight speakers, 264 slots is 33
per speaker: 32 for voice, 2.56 s of it, and one filled with the learned silence embedding. The
cache is a curated voice sample of everyone heard so far, not a transcript of the past.
Latency is set by how much new audio a step waits for. The card's formula is
$(\text{chunk} + \text{right context}) \times 80\,\text{ms}$, which gives the four recommended
settings: 380 frames for 30.4 s ("offline-style"), 13 for 1.04 s, 8 for 0.64 s, and 4 for 0.32 s.
The model can technically run with an 80 ms buffer; 0.32 s is the lowest NVIDIA recommends. None of
these include compute.
The table in that widget is where "real-time" can be checked against NVIDIA's own throughput
numbers. At batch size 1 on an RTX PRO 5000 in BF16, the card reports an RTFx (audio seconds per
processing second) of 12.5 eager and 54 compiled at 0.32 s. Each step there scores 3 new frames,
240 ms of audio, while reading 532 frames. Dividing, a step costs about 19 ms eager or 4.4 ms
compiled, and the same division at 1.04 s and 0.64 s lands on the same 19 ms. Per-step cost is
flat because the cache and FIFO dominate the input. Lower latency does not make steps cheaper, it
makes more of them. That reading is mine, derived from the card's numbers, not a measurement. It
also explains some of the speedup over the predecessor at 1.04 s: the old model reached that
latency with a 6-frame chunk and 7 frames of look-ahead, the new one with 9 and 4, so it takes a
third fewer steps per minute of audio on top of each step being cheaper.
So: real-time, yes, with 12.5× headroom at the most demanding setting on that GPU, before
`torch.compile`. On a CPU or a phone, the card says nothing. The official hardware list is NVIDIA
GPUs from Ampere to Blackwell; the GGUF and NVIDIA's `NeMo-Speech.cpp` runtime suggest CPU use, and
Argmax ships the model in its on-device SDK, but no CPU real-time factor is published and I did not
measure one.
## The numbers NVIDIA reports
The card evaluates on 901 recordings across eight conditions, with forced-alignment reference labels
for AMI, AliMeeting and NOTSOFAR1 ([Horiguchi et al., 2025](https://arxiv.org/abs/2507.09226)
explains why the original ASR-style labels over-count speech). The only comparison model is NVIDIA's
own predecessor. Full-set DER, lower is better, reported:
| dataset | 4spk-v2.1, 30.4 s | Nemotron 3, 30.4 s | 4spk-v2.1, 1.04 s | Nemotron 3, 1.04 s | Nemotron 3, 0.32 s |
|---|---:|---:|---:|---:|---:|
| DIHARD III eval | 19.09 | 12.73 | 19.60 | 13.18 | 13.55 |
| CALLHOME Part 2 | 10.32 | 9.10 | 11.31 | 10.29 | 11.32 |
| AliMeeting test, near | 11.57 | 6.40 | 12.47 | 6.59 | 7.19 |
| AliMeeting test, far | 13.69 | 10.47 | 15.58 | 10.80 | 11.60 |
| AMI test, headset mix | 15.81 | 9.25 | 16.36 | 9.48 | 10.05 |
| AMI test, single distant mic | 21.42 | 11.14 | 21.73 | 12.80 | 12.95 |
| NOTSOFAR1 eval, headset mix | 21.77 | 6.77 | 22.12 | 7.70 | 8.65 |
| NOTSOFAR1 eval, single channel | 30.49 | 11.00 | 31.81 | 12.77 | 14.53 |
I recomputed the blog's headline from these rows: the relative reductions at 1.04 s run from 9.0%
(CALLHOME) to 65.2% (NOTSOFAR1 headset mix), and their unweighted mean is 41.0%, as stated. The
blog is careful to call it an average of per-dataset ratios, not a pooled DER. The biggest gains are
on NOTSOFAR1, where 90 of the 160 recordings have five to seven speakers and a four-slot model is
simply too small.
Going from offline to the 0.32 s setting costs Nemotron 3 between 0.8 and 3.5 points; the
predecessor at 0.32 s is worse than the new model at every setting.
Three things in the same tables are less flattering, and the first two the blog does not
highlight.
- **Two speakers is not better.** On two-speaker CALLHOME, 5.98 against the predecessor's 5.68 at
30.4 s, and 6.98 against 6.83 at 1.04 s. The blog does mention the offline number. For phone
calls, the old four-speaker model is still marginally ahead.
- **Speaker counting got worse on AMI.** The card also reports speaker-counting accuracy (SCA). On
both AMI conditions the predecessor gets the count right on 93.75% of meetings at every latency;
Nemotron 3 gets 87.50% offline and 81.25% on the headset mix at 1.04 s. My reading, labelled as
one: AMI meetings have three or four speakers, and a model with four slots cannot over-count a
four-person meeting. A model with eight can. That is the failure `PhantomLoss` is aimed at, and it
has not closed it. At low latency the counting also slips on NOTSOFAR1 single-channel, from 78.12%
offline to 55.00% at 0.64 s and 0.32 s.
- **The 5-9 speaker DIHARD bucket includes nine-speaker audio**, one more than the model can
represent. The blog says so; it means 27.58 is partly a measurement of a hard cap.
Throughput, reported, batch 32 compiled: 15,113× against 2,619× offline and 865× against 136× at
1.04 s. Those are batched-server numbers, and the blog itself warns against reading them as
single-stream latency.
## "#1": one leaderboard, and a closer second one
The #1 claim comes from VoiceArena's Diarization-Bench v1: 139 English conversations, about 22
hours, 115 in person and 24 over VoIP, sessions up to 26 minutes, overlap scored, no collar, and the
speaker count never given to the system. The operator calls these "early results".
On that board the claim holds with room to spare: 14.72% against 19.3% for BUT's open-source
DiariZen and 20.6% for pyannoteAI's commercial Precision-3, about 24% relative. It also leads with
100 ms and 250 ms collars. Two caveats. The corpus is English conversation only, and VoiceArena's
page does not say where the 139 recordings came from. And the transcription APIs ranked 7th to
12th are general speech-to-text products with diarization attached, not dedicated diarizers.
The second public comparison is Argmax's [OpenBench](https://github.com/argmaxinc/OpenBench), which
NVIDIA's blog also quotes. It runs the model through Argmax's own SDK (`SpeakerKit`, a `684_74MB`
variant) on an M2 Ultra Mac Studio, not through NeMo.
Argmax ships the model in its commercial SDK, so it is a partner, not a neutral party. Three of the
six columns of its DER table, rounded to two decimals as published, lowest of the three in bold:
| dataset | pyannoteAI Precision-3 | Sortformer v2 (Argmax) | Nemotron 3 (Argmax) |
|---|---:|---:|---:|
| AISHELL-4 | **0.10** | 0.32 | 0.11 |
| AMI-IHM | 0.30 | 0.18 | **0.09** |
| AMI-SDM | 0.32 | 0.23 | **0.11** |
| AVA-AVD | **0.34** | 0.60 | 0.45 |
| AliMeeting | **0.11** | 0.21 | 0.18 |
| CallHome | **0.15** | 0.19 | **0.15** |
| DIHARD-III | 0.14 | 0.21 | **0.13** |
| EGO4D | **0.37** | 0.50 | 0.41 |
| Earnings-21 | **0.10** | 0.45 | 0.20 |
| MSDWILD | **0.22** | 0.33 | 0.23 |
| VoxConverse | 0.09 | 0.22 | **0.08** |
| **macro average** | 0.20 | 0.31 | **0.19** |
Lowest average, yes, by 0.01. Lowest on 5 of 11 datasets, as Argmax says, but only by counting the
CallHome tie; pyannoteAI's API is lowest or tied on 7. The losses cluster where speaker counts and
recording conditions are wildest: in-the-wild video (AVA-AVD, EGO4D) and long earnings calls.
And one of the five wins is not evidence. OpenBench's `voxconverse.yaml` scores
`diarizers-community/voxconverse` on its `test` split, while the model card lists VoxConverse
v0.3 "Development and test" among its training data. That is presumably why VoxConverse, the benchmark
you might expect in the card, is absent from NVIDIA's own evaluation. Held-out evidence should not
include it either.
So the Korean repost's third claim needs a qualifier. First on VoiceArena by a wide margin, yes.
Overwhelmingly first on speaker-diarization benchmarks, no: on the broadest public comparison it
is level with a commercial API, and ahead of NVIDIA's own previous model by a lot.
## Licence, the preview, and running it
The GA weights are under OpenMDW-1.1: use, modify and redistribute, commercially, with notices
retained, a patent-retaliation clause, and no restriction on outputs. That is looser than the
predecessor's NVIDIA Open Model License. The earlier
[`Nemotron-3-Diarization-preview`](https://huggingface.co/nvidia/Nemotron-3-Diarization-preview)
repo (created August 24) is different: gated, under NVIDIA's Software and Model Evaluation License,
internal testing only, no production use, and no disclosure of evaluation results without NVIDIA's
consent. I did not request access, and nothing here is about the preview's weights.
I did not run the model, and the reason is itself a finding. The card's NeMo quick start installs
`nemo-toolkit[asr]` from PyPI, which today is 3.0.0 (released August 7). Reading that wheel's source:
its `TransformerEncoder` accepts `abs_pos`, `rel_pos` and `no_pos` and raises on anything else, and
this checkpoint's config asks for `rope`. The wheel also has no `aux_diarization_loss` module and no
10 ms upsampler. On the Transformers side, 5.17.0 (September 9) has no `nemotron3_diarization`
model; the card says to install from source. Both paths need unreleased code from GitHub, so there is
no CPU real-time factor or DER of my own in this piece. Every benchmark number above is reported by
NVIDIA, VoiceArena or Argmax.
For a speaker-attributed transcript, NVIDIA pairs the diarizer with a streaming ASR model: either a
multitalker Parakeet that consumes the speaker activity as conditioning, or Nemotron 3.5 ASR run
once per active speaker with the diarizer's activity as a mask.
[Qwen3.8-LiveTranslate](/articles/qwen-livetranslate#diarization-and-the-benchmark-that-was-not-released)
took the other route and built diarization into the translation model itself, and that piece made the
point that offline diarization is easier because it sees the whole recording. This model's own rows
put a size on it: 0.8 to 3.5 DER points between the 30.4 s and 0.32 s settings. The VAD, STT, LLM,
TTS cascade in [speech-to-speech](/articles/speech-to-speech) is the kind of pipeline a separate
diarizer plugs into, between the audio and the transcript. Separation, the harder problem of pulling
overlapped voices apart rather than labelling them, is what
[FLASepformer](/articles/flasepformer-jaec) attempts. Diarization only tells you both people
were talking.
## What holds
| claim | verdict |
|---|---|
| 100M (0.1B) parameters | **Holds.** 99,226,504, read from the safetensors header (measured). |
| Up to eight speakers | **Holds, as a hard cap.** Eight output channels; nine-speaker audio is scored anyway in DIHARD's 5-9 bucket. |
| Real-time | **Holds on NVIDIA's GPU.** RTFx 12.5 eager at 0.32 s latency (reported); about 19 ms per step (derived). No CPU figure is published. |
| #1 on VoiceArena, 14.72% DER | **Holds** as VoiceArena reports it, on 22 hours of English, marked early. |
| Overwhelmingly #1 | **Does not hold.** OpenBench: 0.19 against 0.20, 5 of 11 datasets counting a tie, one of them trained on. |
| 41.0% average DER reduction at 1.04 s | **Holds** against its own predecessor; worse on two-speaker CALLHOME and on AMI speaker counts. |
| Install with `nemo-toolkit[asr]` | **Not yet.** The released package cannot build this config. |
The architecture is the part worth keeping. One plain Transformer, a loss that makes output slots
mean "order of arrival", and a cache that stores each arrival's cleanest seconds: that combination
lets a 99.2M-parameter model diarize a stream at a third of a second of latency with no clustering
and no per-chunk permutation search. The benchmark story is good and narrower than the headline.