Nemotron 3 Diarization: who spoke when, sorted by who spoke first

mdjsonmcpcopy↗ai

2026-09-26 · 22 min · audio · speech · small-models · open-weights · benchmarks · explainer

A transcript that is word-perfect and speaker-blind is half a transcript. You can read that someone promised the report by Friday; you cannot tell who. NVIDIA's answer, released on September 23 as nvidia/Nemotron-3-Diarization, is a speaker diarizer: audio in, something like "speaker 2 talked from 12.4 s to 15.1 s" out, for up to eight speakers, streaming or offline, from one checkpoint. The announcement leads with a leaderboard: "ranked #1 in VoiceArena's Diarization leaderboard with a 14.72% Diarization Error Rate (DER)". A Korean repost that did the rounds compressed it further, to three claims: 100M (0.1B) parameters, real-time, and overwhelmingly #1 on speaker-diarization benchmarks.

Each of those can be checked, and most of them without running anything. I read the parameter count out of the model.safetensors header over an HTTP range request, the training recipe out of the model_config.yaml inside the .nemo tarball (its first 4 KB, same trick), the streaming logic out of NeMo's source, and the benchmark claims out of NVIDIA's model card, VoiceArena's launch page and Argmax's OpenBench tables. The short version: the parameter count is right to the rounding, "real-time" is right on a data-centre GPU and unmeasured anywhere else, and "#1" is true on the leaderboard NVIDIA cites and a one-point edge on the other public one.

Who spoke when, and how it is scored

A diarizer's output is a matrix: time frames down one axis, speakers across the other, and in each cell the probability that this speaker is talking now. Nemotron 3 Diarization emits exactly that, a [T, 8] tensor of sigmoid probabilities, one row every 10 ms. Thresholded and merged into segments, it becomes the familiar start end speaker_k list. Two things about that matrix matter for everything that follows. Nothing stops two cells in the same row from both being on, so overlapping speech is representable. And the column labels are anonymous: "speaker 2" is not a person, it is a slot.

The standard score is the diarization error rate. For every scoring frame, count the speakers the reference says are talking, $N_\text{ref}$, the speakers the system says are talking, $N_\text{hyp}$, and how many of the system's speakers are the right ones, $N_\text{ok}$ (after the scorer has found the best one-to-one mapping between system slots and real speakers). Then

The denominator is speaker-time, not wall-clock time: a second where two people talk counts as two. Two protocol knobs move the number a lot. A collar excuses a window around every reference boundary (0.25 s either side is traditional); the card scores DIHARD III, AMI, AliMeeting and NOTSOFAR1 with no collar and CALLHOME with 0.25 s. And overlap can be scored or skipped; every number in the card scores it. NVIDIA's evaluation subcard says it plainly: "DER is not fully interpretable without these settings."

The widget is a made-up 60-second conversation between three people, A, B and C, with three overlaps: A and B for 3 seconds, A and B again for 2, B and C for 4. That puts 61 seconds of speaker time into 60 seconds of audio. Each slider injects one error type, and the readout does the arithmetic above. The button is the point. A system that can only say one speaker per frame loses one speaker in every overlapped frame, 9 seconds here, and C's second turn, which sits entirely inside B's, disappears. That is 14.8% DER before the system has made a single mistake it could have avoided. Overlap is not an edge case in meetings, and a system that cannot represent it starts every meeting in the hole.

Two ways to build a diarizer

The classic pipeline is a cascade. A voice-activity detector finds speech. The speech is cut into short segments. A speaker-embedding network turns each segment into a vector. A clustering step groups the vectors, and each cluster becomes a speaker. It has real strengths: it handles any number of speakers, and every stage can be swapped. Its weakness is baked into the output format: one segment, one embedding, one cluster, one speaker. Overlapped speech gets one label at best, which is the button in the widget above. Production pipelines such as pyannote's now run a small neural segmentation model over short windows first and cluster across windows afterwards, which is a hybrid built to patch exactly this.

End-to-end neural diarization (EEND) throws the cascade away and treats diarization as frame-wise multi-label classification: one network, one sigmoid per speaker slot per frame, trained with binary cross-entropy. Overlap is free. The catch is the labels. If the reference says A spoke first and B second, is the model wrong for putting A in slot 1? It shouldn't be, so training uses a permutation-invariant loss (PIL): score every assignment of reference speakers to output slots and keep the cheapest,

where $\mathbf{Y}$ is the $K \times T$ reference activity, $\mathbf{P}$ the predicted one, and $\Pi$ the $K!$ permutations of the speaker rows. With eight slots that is 40,320 permutations per training example, which is why implementations solve it as an assignment problem instead of enumerating. The larger problem is at inference in a stream: chunk 7 has no idea which slot chunk 6 gave to whom, so a streaming EEND needs some way to re-align slots across chunks, every chunk.

Sorting speakers by arrival

Sortformer (Park et al., 2024), the design Nemotron 3 Diarization inherits, removes the ambiguity instead of searching over it. Its rule: output slot 1 is whoever spoke first in the audio, slot 2 whoever spoke second, and so on. With the reference rows sorted by each speaker's first segment, $\eta$, the target is fixed and the loss is plain BCE, the paper's Sort Loss:

The paper is candid that "arrival time estimation is not always correct", more so as speaker counts rise, so it trains on a mix, $\alpha \cdot \mathcal{L}_\text{Sort} + (1-\alpha) \cdot \mathcal{L}_\text{PIL}$. The model also needs positional information to sort at all: attention without positions is permutation-equivariant, so it could not tell first from second.

The .nemo config shows how NVIDIA weighted it this time, and what it added:

The two new losses live in NeMo's aux_diarization_loss.py on main. ActivityLoss is a three-class cross-entropy on a small auxiliary head: is this frame silence, one speaker, or overlap? It needs no speaker alignment at all, so it teaches overlap detection without touching the permutation problem. PhantomLoss penalises confident activity (sigmoid above 0.25) in any output slot that has no speaker in the reference for the whole segment: a direct penalty on inventing a fifth speaker in a four-person meeting. The Transformers-format safetensors carries no activity head, so at inference the eight speaker sigmoids are the whole output.

Arrival order pays off at inference. If slot $k$ always means "the $k$-th voice to appear", then a model that sees the earlier audio again can put the same voice back into the same slot without any matching step. That is the whole streaming design.

What is in the checkpoint

The safetensors header is 48,568 bytes of JSON describing 417 float32 tensors. Summed:

So "100M" is 99.2M, float32, 396,954,592 bytes on disk. The shapes explain the architecture better than the diagram does. The input is 16 kHz mono audio, turned into 128 mel bins every 10 ms. The input projection is 1024 wide because 128 mel bins times 8 stacked 10 ms frames is 1024: the model downsamples by concatenation, not convolution, producing one 512-wide vector per 80 ms. Each of the 31 layers is a plain pre-norm Transformer block (8 heads, 2,048-wide MLP, no bias on Q, K and V) with rotary position embeddings. After the stack, a projection drops to 192 dimensions, and the upsampler turns 192 channels into 1,536, which is 8 × 192: a sub-pixel convolution that unfolds each 80 ms frame into eight 10 ms frames. NeMo initialises it to an identity, so every 10 ms frame starts as a copy of its 80 ms parent and training learns the refinement. A two-layer head then gives eight sigmoids per 10 ms. The 512-wide silence_embeds is a learned stand-in used to pad the speaker cache (more on that below).

expand

The predecessor it is benchmarked against is a different shape. diar_streaming_sortformer_4spk-v2.1 stacks a 17-layer FastConformer (convolution modules, a depthwise-striding downsampler) under a separate 18-layer, 192-wide Transformer, for 117M parameters per its card. Nemotron 3 Diarization replaces both with one homogeneous 31-layer Transformer, initialised from a Transformer-based NEST self-supervised checkpoint (Huang et al., 2024). It is smaller by about 18M parameters, has twice the speaker slots, and, as the next section shows, is cheaper per step. The GA .nemo is 198,676,480 bytes, about 2 bytes per parameter, where the gated preview's was 397,199,360; a q8_0 GGUF of 107,012,128 bytes ships alongside, presumably for the C++ runtime.

Training, per the card: 8 nodes of 8 A100-80GB, offline on simulated mixtures first, then streaming fine-tuning on real and simulated audio together. The data is about 10,000 hours of real conversations plus 82,611 hours of mixtures with one to eight speakers, simulated with the FastMSS toolkit from Polok et al., much of the source audio licensed from David AI. NVIDIA's blog attributes a 0.77-point DER drop (11.19% to 10.42%) to adding the David AI data, without saying which test set that compound figure pools.

Streaming: a cache sorted by speaker

Streaming Sortformer (Medennikov et al., 2025) turns the arrival-order property into memory. Each step, the model sees four things concatenated along time, all as 512-wide post-stacking embeddings:

1. the speaker cache (Arrival-Order Speaker Cache, AOSC): up to 264 frames kept from the whole past, grouped by speaker slot in arrival order;
2. the FIFO: the most recent past frames, in order;
3. the chunk: the new frames to be scored;
4. the right context: a few frames of look-ahead, attended to but scored on the next step.

All of it goes through all 31 layers with full attention. Because the cache is laid out speaker by speaker in arrival order, the first voices the model reads every step are slot 1's, then slot 2's, and a returning speaker lands in their old slot without a matching step. That is what the Sortformer training bought.

When the FIFO overflows, its oldest frames (222 of them in the streaming configs) are merged into the cache and the cache is compressed back to 264. Reading _compress_spkcache in NeMo's sortformer_modules.py, each frame gets a per-speaker score

with every probability clamped at 0.25 before the logs. It is positive when speaker $k$ is confidently on and everyone else is off: clean, single-speaker evidence. Non-speech is dropped, overlapped frames are dropped once a speaker has 16 clean ones, the newest frames get a 0.05 nudge, and two boosting passes guarantee each of the eight speakers up to 24 of the 264 slots (1.92 s of their voice) before the rest go to the highest scores. With eight speakers, 264 slots is 33 per speaker: 32 for voice, 2.56 s of it, and one filled with the learned silence embedding. The cache is a curated voice sample of everyone heard so far, not a transcript of the past.

Latency is set by how much new audio a step waits for. The card's formula is $(\text{chunk} + \text{right context}) \times 80\,\text{ms}$, which gives the four recommended settings: 380 frames for 30.4 s ("offline-style"), 13 for 1.04 s, 8 for 0.64 s, and 4 for 0.32 s. The model can technically run with an 80 ms buffer; 0.32 s is the lowest NVIDIA recommends. None of these include compute.

The table in that widget is where "real-time" can be checked against NVIDIA's own throughput numbers. At batch size 1 on an RTX PRO 5000 in BF16, the card reports an RTFx (audio seconds per processing second) of 12.5 eager and 54 compiled at 0.32 s. Each step there scores 3 new frames, 240 ms of audio, while reading 532 frames. Dividing, a step costs about 19 ms eager or 4.4 ms compiled, and the same division at 1.04 s and 0.64 s lands on the same 19 ms. Per-step cost is flat because the cache and FIFO dominate the input. Lower latency does not make steps cheaper, it makes more of them. That reading is mine, derived from the card's numbers, not a measurement. It also explains some of the speedup over the predecessor at 1.04 s: the old model reached that latency with a 6-frame chunk and 7 frames of look-ahead, the new one with 9 and 4, so it takes a third fewer steps per minute of audio on top of each step being cheaper.

So: real-time, yes, with 12.5× headroom at the most demanding setting on that GPU, before torch.compile. On a CPU or a phone, the card says nothing. The official hardware list is NVIDIA GPUs from Ampere to Blackwell; the GGUF and NVIDIA's NeMo-Speech.cpp runtime suggest CPU use, and Argmax ships the model in its on-device SDK, but no CPU real-time factor is published and I did not measure one.

The numbers NVIDIA reports

The card evaluates on 901 recordings across eight conditions, with forced-alignment reference labels for AMI, AliMeeting and NOTSOFAR1 (Horiguchi et al., 2025 explains why the original ASR-style labels over-count speech). The only comparison model is NVIDIA's own predecessor. Full-set DER, lower is better, reported:

I recomputed the blog's headline from these rows: the relative reductions at 1.04 s run from 9.0% (CALLHOME) to 65.2% (NOTSOFAR1 headset mix), and their unweighted mean is 41.0%, as stated. The blog is careful to call it an average of per-dataset ratios, not a pooled DER. The biggest gains are on NOTSOFAR1, where 90 of the 160 recordings have five to seven speakers and a four-slot model is simply too small. Going from offline to the 0.32 s setting costs Nemotron 3 between 0.8 and 3.5 points; the predecessor at 0.32 s is worse than the new model at every setting.

expand

Three things in the same tables are less flattering, and the first two the blog does not highlight.

• Two speakers is not better. On two-speaker CALLHOME, 5.98 against the predecessor's 5.68 at 30.4 s, and 6.98 against 6.83 at 1.04 s. The blog does mention the offline number. For phone calls, the old four-speaker model is still marginally ahead.
• Speaker counting got worse on AMI. The card also reports speaker-counting accuracy (SCA). On both AMI conditions the predecessor gets the count right on 93.75% of meetings at every latency; Nemotron 3 gets 87.50% offline and 81.25% on the headset mix at 1.04 s. My reading, labelled as one: AMI meetings have three or four speakers, and a model with four slots cannot over-count a four-person meeting. A model with eight can. That is the failure PhantomLoss is aimed at, and it has not closed it. At low latency the counting also slips on NOTSOFAR1 single-channel, from 78.12% offline to 55.00% at 0.64 s and 0.32 s.
• The 5-9 speaker DIHARD bucket includes nine-speaker audio, one more than the model can represent. The blog says so; it means 27.58 is partly a measurement of a hard cap.

Throughput, reported, batch 32 compiled: 15,113× against 2,619× offline and 865× against 136× at 1.04 s. Those are batched-server numbers, and the blog itself warns against reading them as single-stream latency.

"#1": one leaderboard, and a closer second one

The #1 claim comes from VoiceArena's Diarization-Bench v1: 139 English conversations, about 22 hours, 115 in person and 24 over VoIP, sessions up to 26 minutes, overlap scored, no collar, and the speaker count never given to the system. The operator calls these "early results".

expand

On that board the claim holds with room to spare: 14.72% against 19.3% for BUT's open-source DiariZen and 20.6% for pyannoteAI's commercial Precision-3, about 24% relative. It also leads with 100 ms and 250 ms collars. Two caveats. The corpus is English conversation only, and VoiceArena's page does not say where the 139 recordings came from. And the transcription APIs ranked 7th to 12th are general speech-to-text products with diarization attached, not dedicated diarizers.

The second public comparison is Argmax's OpenBench, which NVIDIA's blog also quotes. It runs the model through Argmax's own SDK (SpeakerKit, a 684_74MB variant) on an M2 Ultra Mac Studio, not through NeMo. Argmax ships the model in its commercial SDK, so it is a partner, not a neutral party. Three of the six columns of its DER table, rounded to two decimals as published, lowest of the three in bold:

Lowest average, yes, by 0.01. Lowest on 5 of 11 datasets, as Argmax says, but only by counting the CallHome tie; pyannoteAI's API is lowest or tied on 7. The losses cluster where speaker counts and recording conditions are wildest: in-the-wild video (AVA-AVD, EGO4D) and long earnings calls. And one of the five wins is not evidence. OpenBench's voxconverse.yaml scores diarizers-community/voxconverse on its test split, while the model card lists VoxConverse v0.3 "Development and test" among its training data. That is presumably why VoxConverse, the benchmark you might expect in the card, is absent from NVIDIA's own evaluation. Held-out evidence should not include it either.

So the Korean repost's third claim needs a qualifier. First on VoiceArena by a wide margin, yes. Overwhelmingly first on speaker-diarization benchmarks, no: on the broadest public comparison it is level with a commercial API, and ahead of NVIDIA's own previous model by a lot.

Licence, the preview, and running it

The GA weights are under OpenMDW-1.1: use, modify and redistribute, commercially, with notices retained, a patent-retaliation clause, and no restriction on outputs. That is looser than the predecessor's NVIDIA Open Model License. The earlier Nemotron-3-Diarization-preview repo (created August 24) is different: gated, under NVIDIA's Software and Model Evaluation License, internal testing only, no production use, and no disclosure of evaluation results without NVIDIA's consent. I did not request access, and nothing here is about the preview's weights.

For a speaker-attributed transcript, NVIDIA pairs the diarizer with a streaming ASR model: either a multitalker Parakeet that consumes the speaker activity as conditioning, or Nemotron 3.5 ASR run once per active speaker with the diarizer's activity as a mask. Qwen3.8-LiveTranslate took the other route and built diarization into the translation model itself, and that piece made the point that offline diarization is easier because it sees the whole recording. This model's own rows put a size on it: 0.8 to 3.5 DER points between the 30.4 s and 0.32 s settings. The VAD, STT, LLM, TTS cascade in speech-to-speech is the kind of pipeline a separate diarizer plugs into, between the audio and the transcript. Separation, the harder problem of pulling overlapped voices apart rather than labelling them, is what FLASepformer attempts. Diarization only tells you both people were talking.

What holds

The architecture is the part worth keeping. One plain Transformer, a loss that makes output slots mean "order of arrival", and a cache that stores each arrival's cleanest seconds: that combination lets a 99.2M-parameter model diarize a stream at a third of a second of latency with no clustering and no per-chunk permutation search. The benchmark story is good and narrower than the headline.