# Nemotron 3.5 Super VL: 88 layers, eight of which remember every token

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/nemotron-3-5-super-vl
> date: 2026-10-15
> tags: mixture-of-experts, state-space-models, vision-language-models, quantization, long-context, nvidia, gguf

On 2026-10-15 LM Studio posted: "Nemotron 3.5 Super VL from @NVIDIAAI is live in LM Studio! 120B total / 12B
active MoE. Supports vision input, reasoning. Trained for visual understanding, tool use. Requires ~70GB RAM to
run locally."

Behind the post is a real release from NVIDIA, in four Hugging Face repositories:

- `NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16`, the post-trained reference weights.
- `-NVFP4`, the same model quantized to 4-bit experts, for deployment.
- `-Base-BF16`, the pretrained checkpoint before any post-training.
- `-NVFP4-LiLiCorr`, which is not the model at all. It is a 1.13B-parameter speculative drafter (more on it below).

There is a 79-page technical report, [Nemotron 3.5 Super and Lightning](https://research.nvidia.com/labs/nemotron/files/NVIDIA-Nemotron-3.5-Super-and-Lightning-Technical-Report.pdf),
dated 2026-10-14. It is not on arXiv. The licence is the [OpenMDW License Agreement 1.1](https://openmdw.ai/license/1-1/),
which grants permission "to deal in the Model Materials without restriction" and "does not impose any restrictions
or obligations with respect to any use, modification, or sharing of any outputs". That is closer to MIT than to the
NVIDIA Open Model License earlier Nemotrons used. Its one sharp clause is a patent-retaliation termination.

I did not run the model. Everything below comes from files: `config.json`, the safetensors headers of the BF16 and
NVFP4 repos (read by HTTP range request, about 51 MB of headers instead of 324 GB of weights), the NVFP4 repo's
`hf_quant_config.json`, the GGUF headers of LM Studio's build, the remote modeling code, and the report. Numbers are
labelled **reported** (NVIDIA or LM Studio says so), **measured** (I counted it in a file) or **reasoned** (my
arithmetic on the other two, assumptions stated).

<ModelCard repo="nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16" claimed="120B / 12B active" note="Mamba-2 + attention + latent MoE language model with a C-RADIO vision encoder. config.json's auto_map points at remote code in the repo (trust_remote_code). Licence OpenMDW-1.1." />

## What kind of model this is

The card calls it a "Mamba-2–Transformer hybrid Mixture of Experts with a C-RADIO vision encoder". The report says
the language backbone is initialized from Nemotron 3 Super Base, so this is the same family as Nemotron-H and
Nemotron 3: most sequence mixing is done by Mamba-2 state-space layers, a few attention layers are kept for exact
recall, and the feed-forward work goes to a sparse mixture of experts. The new parts in 3.5 are the training
(a trillion tokens of distillation from Nemotron 3 Ultra Base, an Indic stage, agentic software-engineering
mid-training, a two-stage SFT that teaches it vision), a recursively shared multi-token-prediction head, and the
quantized and drafter releases.

The interesting question for anyone who wants to run it is not the training. It is what the architecture costs in
memory, and that falls straight out of the config.

## The stack, from config.json

`llm_config.layers_block_type` lists 88 blocks one by one (**measured**): 40 `mamba`, 40 `moe` and 8 `attention`.
Each block is a single mixer or feed-forward behind an RMSNorm and a residual, so "88 layers" here means 88
half-layers in the usual transformer sense. The order is almost, but not quite, periodic. Mamba-2 and MoE alternate,
and an attention block is dropped in at indices 7, 16, 25, 36, 47, 58, 69 and 78: gaps of 9, 9, 11, 11, 11, 11 and
9. Attention sits only in the middle of the stack, never at either end.

<HybridBudget />

Each kind of block, from the BF16 headers (**measured**):

- **Mamba-2 mixer, 109,640,064 parameters.** 128 heads of head dimension 64, a state of 128 per head, 8 groups for
  the input-dependent B and C matrices, and a width-4 causal convolution. The input projection is
  `[18560, 4096]`, and the 18,560 decomposes exactly: 8,192 for the gate z, 8,192 for x, 1,024 each for B and C
  (8 groups of 128) and 128 for the per-head step size. The mixer's inner width is twice the 4,096 hidden size.
- **Attention mixer, 35,655,680 parameters.** 32 query heads and 2 key-value heads of dimension 128. With only two
  KV heads, K and V together are 512 values per token per layer.
- **Latent MoE block, 2,873,102,848 parameters.** 512 routed experts, 22 used per token, plus one shared expert. This
  is where the parameters live. It gets its own section.

If you have read the [ReplaySSM article](/articles/replayssm), the Mamba-2 shapes are the same as Nemotron 3 Super's;
if you have read [Rigel](/articles/rigel-hybrid-mamba), this is the same recipe at fifty times the size, with a much
lower ratio of attention: 8 of 48 mixers here, against 10 of 40 there. For the mechanism of the state-space layer
itself, see the [Mamba architecture page](/architectures/mamba-ssm).

## Why eight attention layers decide the memory bill

An attention layer has to keep a key and a value for every token it has seen. A Mamba-2 layer compresses everything
it has seen into a fixed state and throws the tokens away. That is the whole trade: attention can look anything up
exactly; Mamba-2 pays a constant amount of memory but can only remember what fit through its state.

The arithmetic for one sequence (**reasoned** from config fields):

- **KV cache.** 8 layers × 2 KV heads × 128 dims × 2 (K and V) = 4,096 values per token. In BF16 that is 8 KiB per
  token, so a million-token context (the config's `max_position_embeddings` is 1,048,576) holds 8.6 GB of KV. In FP8,
  which the NVFP4 release serves by default, half that.
- **Mamba-2 state.** 40 layers × 128 heads × 64 × 128 = 41,943,040 values, plus a tiny convolution state of
  10,240 channels × 3 per layer. The config asks for an FP32 SSM cache, which makes it 168 MB per sequence (plus
  5 MB of convolution state), the same at token 10 and at token 1,000,000. The NVFP4 serving recipe keeps it in FP16 with stochastic rounding, half that.
- **The crossover.** The constant Mamba state equals the growing KV cache at about 20,480 tokens in BF16. Past that,
  the eight attention layers are the larger cost, and they keep growing.

Now replace the 40 Mamba-2 layers with the same narrow attention. Every one of the 48 mixers keeps 1 KiB per token,
the cache grows six times faster, and the million-token context needs 51.5 GB of KV per sequence (**reasoned**).
That is the hybrid's argument in one number. The widget above runs it for every context from 4K to 1M and for each
release's cache precision, against the memory of an H100, a DGX Spark and an H200.

There is a nice consistency check in the report. It says that in vLLM "one Mamba cache page occupies at least as many
bytes as approximately 4K tokens of Attention KV cache". One Mamba-2 layer's state is 1,048,576 values; one attention
layer's KV is 512 values per token; at FP32 state and BF16 KV, 4 MiB divided by 1 KiB per token is 4,096 tokens
(**reasoned**). The report's figure is the config's arithmetic.

What the constant state costs instead is in the report too. When prefix caching and speculative decoding run together,
vLLM has to recover a Mamba state at an arbitrary position, and Mamba states are only checkpointed every few
thousand tokens, so the engine replays tokens to rebuild one. NVIDIA added an extra checkpoint near the replay boundary
and the `--enable-mamba-shared-prefix-checkpoint` flag that appears in every serving recipe on the card. The state is
cheap to hold and awkward to rewind.

## Latent MoE: experts that work in a quarter of the width

The MoE block is not a standard one, and its tensor shapes say exactly what it does (**measured**):

| Tensor | Shape | Role |
|---|---|---|
| `gate.weight` | `[512, 4096]` | the router, scoring all 512 experts on the full-width hidden state |
| `fc1_latent_proj` | `[1024, 4096]` | projects the 4,096-wide token down to a 1,024-wide latent |
| `experts.N.up_proj` | `[2688, 1024]` | each expert's up projection, in the latent |
| `experts.N.down_proj` | `[1024, 2688]` | each expert's down projection, back to the latent |
| `fc2_latent_proj` | `[4096, 1024]` | projects the expert sum back to 4,096 |
| `shared_experts.up/down` | `[5376, 4096]` / `[4096, 5376]` | one shared expert at full width |

There is no gate matrix in the experts: the activation is ReLU², `mlp_hidden_act: relu2`, so each expert is two
matrices, 5,505,024 parameters. The router picks 22 of 512 and normalises their weights (`norm_topk_prob`), and the
config's `routed_scaling_factor` of 5.0 scales the sum.

<LatentMoe />

Two things follow. First, a routed expert is small: 5.5M parameters, against 44M for the shared expert, so the 22
routed experts a token visits add up to 121M per layer, less than three shared experts' worth (**reasoned**). The
routed pool is about storage, not compute. Second, the tokens that cross the network to experts on other GPUs are
1,024 values wide, not 4,096. The report credits this, "LatentMoE exchanges latent vectors that are smaller than the
Qwen models' hidden states", for part of its throughput lead. Per token, per MoE layer, 22 dispatched copies at BF16
are 44 KiB in the latent against 176 KiB at full width (**reasoned**, before any FP8 dispatch). A quarter of the
all-to-all.

## Does 120B total, 12B active hold?

Summing every tensor in the 64 BF16 shards (**measured**):

| Component | Parameters |
|---|---:|
| Routed experts (40 layers × 512) | 112.74B |
| Mamba-2 mixers (40) | 4.39B |
| MTP draft head | 2.94B |
| Shared experts (40) | 1.76B |
| Vision encoder (C-RADIO) | 0.65B |
| Embedding table | 0.54B |
| Output head | 0.54B |
| Latent projections, routers, norms | 0.42B |
| Attention mixers (8) | 0.29B |
| Vision connector | 0.19B |
| **Total** | **124.45B** |

The language model without the MTP head and the vision side is 120.67B, so "120B total" is an honest round-down of the
backbone. The whole file, which the report itself uses to compute bits per element, is 124.45B.

Active per text token: every Mamba-2 and attention mixer, every shared expert, latent projection and router, 22 of the
512 routed experts in each MoE layer, and the output head. That is 12.23B (**measured** from shapes, with the
selection rule from config). It leaves out the embedding table (a lookup reads one row), the MTP head (it only runs
when speculative decoding is on) and the vision tower (it runs once per image). Both headline numbers hold. Routed
experts are 90.6% of the stored checkpoint and 39.6% of the active compute; the always-on Mamba-2 mixers are 35.9%.

## The vision side: C-RADIO, a pixel shuffle, and video packing

The vision tower is C-RADIO (LM Studio's page says C-RADIO v4-H), configured as `vit_huge_patch16_224`: 32 layers,
hidden size 1,280, patch size 16, with three summary tokens from its distillation teachers and seven registers
(**measured**, from `vision_config`). It accepts between 1,024 and 13,312 patches per image, so resolution is
dynamic.

The connector is the InternVL-style one. A pixel shuffle with `downsample_ratio: 0.5` folds each 2×2 block of patch
features into one 5,120-wide vector, and a two-layer MLP (RMSNorm, 5,120 to 20,480, ReLU², 20,480 to 4,096) maps it
into the language model's width. So an image costs a quarter as many tokens as it has patches: 256 at the minimum,
3,328 at the maximum (**reasoned** from the processor config and the modeling code).

Video goes through the same tower with two frames stacked on the channel axis (`video_temporal_patch_size: 2`), so a
pair of frames is one ViT pass. The processor targets 1,024 patches per frame, which is 256 tokens per pair. The card's
vLLM recipe samples up to 256 frames at 2 FPS, which would be 128 pairs and 32,768 tokens, and then
`--video-pruning-rate 0.5` drops half of them (**reasoned**). The pruning is Efficient Video Sampling, readable in
`modeling_nemotron_h_omni.py`: it scores every token by cosine dissimilarity to the same position one frame-pair
earlier, always keeps the first pair, and keeps the most-changed tokens up to the budget. A static slide deck costs
almost nothing after the first frame. A handheld walkthrough costs nearly all of it.

The report is clear that vision came late: the base model is text-only, the projector is warmed up, and vision data
arrives in SFT. That matters for the benchmarks.

## What it scores, and where it loses

The headline figure from the card and the report puts it against six models:

<Figure src="https://media.thesatyajit.com/ai/articles/nemotron-3-5-super-vl/fig1.png?v=4b9f24375652f153" alt="Grouped bar charts of twelve benchmarks comparing Nemotron 3.5 Super against Nemotron 3 Ultra, Qwen3.5 122B-A10B, Qwen3.8 Flash-Next, DeepSeek-V4 Flash, Inkling Small and GLM-5.3 Flash, with a panel on the right showing GB200 output throughput of 1,854, 2,283 and 4,332 tokens per second per GPU for Qwen3.5, Qwen3.8 Flash-Next and Nemotron 3.5 Super." caption="Accuracy across open models on twelve benchmarks, and maximum GB200 throughput at 50 or more tokens per second per user on a 20-turn agentic workload. Every number is NVIDIA's (Nemotron 3.5 Super technical report, Figure 1; also on the model card)." />

I counted the card's own table, 36 non-offline rows, treating hallucination rate as lower-is-better (**measured** from
the card; every score in it is **reported**):

- Against **Qwen3.5-122B-A10B**, the closest model by size (122B total, 10B active), it wins 22, loses 13, ties 1. The
  wins are large on agentic and reasoning rows: SWE-bench Pro 51.78 against 44.9, Terminal-Bench 2.1 66.70 against
  48.70, BrowseComp 78 against 58.1, and an AA-Omniscience hallucination rate of 30.0 against 84.5.
- Against **Qwen3.8-Flash-Next** it wins 6 and loses 29. Against **GLM-5.3-Flash**, 10 and 25. Against
  **DeepSeek-V4-Flash**, 4 and 21. NVIDIA put three models it mostly loses to in its own chart, which I count in its
  favour.
- **The vision rows are the weak half.** On the 11 multimodal rows it loses 8 of 11 to Qwen3.5-122B-A10B, the model it
  otherwise beats: V\* 85.60 against 91.1, MathVision 78.16 against 83.3, OCRBench v2 Chinese 59.15 against 67.3,
  BabyVision 29.60 against 35.7. It wins only OCR-Reasoning, HLE full set and Video-MME v2, the last by 0.85 points.
  Against Qwen3.8-Flash-Next it loses 10 of 11. For a release whose name ends in VL, vision is not where it leads.
- **MMLU-ProX, a multilingual knowledge test, is its worst row**: 76.91, the lowest of all seven models, even after
  the Indic continued pretraining stage. And it loses τ2-bench telecom by 14 points to Qwen3.5.

The card's harness chart is the honest one for agentic coding. NVIDIA's headline SWE-bench Pro score of 51.78 comes
from its own NeMo Gym harness; in six third-party agent harnesses it scores between 41.04% and 49.45%.

<Figure src="https://media.thesatyajit.com/ai/articles/nemotron-3-5-super-vl/fig2.png?v=a7ec8124087f7084" alt="Two bar charts. SWE-Bench Pro across six agent harnesses: Claude Code 49.45%, Mini-SWE agent 47.40%, Codex 47.26%, Hermes 47.20%, Pi 46.92%, OpenCode 41.04%. Terminal-Bench 2.1: Codex 61.80%, Hermes 61.42%, Claude Code 59.55%, OpenCode 55.06%, Mini-SWE agent 54.68%, Pi 48.31%." caption="The same model in six third-party agent harnesses, with unmodified harness system prompts. Every one scores below the 51.78 headline on SWE-Bench Pro (Nemotron 3.5 Super VL model card, agentic coding benchmarks)." />

### Terminal-Bench does not agree with itself

Terminal-Bench 2.1 appears four times, with three different BF16 scores (**reported**, all of them):

| Source | BF16 | NVFP4 |
|---|---:|---:|
| Model card benchmark table | 66.70 | 60.80 |
| Report, Table 23 (BF16 vs NVFP4) | 60.67 | 62.78 |
| Report, Figure 1 and the card's chart | 60.7 | |
| Report, test-time scaling baseline | 62.7 | |

The card's NVFP4 table says quantization costs 5.9 points on this benchmark. The report's table, which NVIDIA says
holds "all configs, harnesses, and other non-model parameters" fixed, says NVFP4 gains 2.11. Both cannot describe the
same runs. My guess is that the card's 66.70 is a later or differently configured run, but nothing in either document
says so. Two smaller slips: the report's abstract and Section 4.1 give a 99.62% median accuracy recovery for NVFP4, its
conclusion says 99.52%; the card says 67B tokens of software-engineering mid-training, the report says 65.4 billion.

## NVFP4, and what "~70GB RAM" is

The NVFP4 release is a mixed-precision checkpoint, and `hf_quant_config.json` lists every layer (**measured**): the
20,480 routed-expert matrices and the 80 shared-expert matrices are NVFP4 with one FP8 scale per 16 values; the 40
Mamba-2 in/out projections, the 8 attention layers' q/k/v/o and the output head are FP8; the router, the latent
projections, the convolutions, the embeddings and the whole vision path stay BF16. The KV cache is FP8. The files total
74.99 GB. The report computes 4.82 bits per element over all 124.453 billion parameters, which matches the files.

<Figure src="https://media.thesatyajit.com/ai/articles/nemotron-3-5-super-vl/fig3.png?v=874b4b196657ca94" alt="Bar charts comparing BF16 and NVFP4 serving. Maximum throughput at 50 tokens per second per user: H100 522 versus 970 tokens per second per GPU (1.86 times), GB200 1,778 versus 4,324 (2.43 times). Generation speed at minimum latency: H100 509 versus 576 tokens per second per user, GB200 587 versus 765." caption="BF16 against NVFP4 on the 20-turn agentic workload. On H100 the NVFP4 checkpoint runs as 4-bit weights with 16-bit activations; on GB200 both are 4-bit (Nemotron 3.5 Super technical report, Figure 10)." />

On accuracy the report gives a 36-benchmark table (Table 23). Most rows move by under a point. The largest drop is
MMLU-ProX, 76.91 to 73.76; the card also shows WebVoyager falling from 77.72 to 66.31 and DeepSWE from 24.65 to 18.2,
both offline rows that are not in the report's table.

LM Studio does not ship either of NVIDIA's checkpoints. Its page links `lmstudio-community/...-GGUF`, which holds two
quantizations (**measured** from the file listing):

- `Q4_K_M`: two shards totalling **70.10 GB**. That is LM Studio's "~70GB".
- An NVFP4 GGUF: 75.24 GB.
- A separate `mmproj` file for the vision tower and connector, in BF16: 1.73 GB.

I range-read the first shard's GGUF header. It declares `nemotron_h_moe` with `block_count` 89: the 88 backbone layers
plus the MTP layer, so the draft head is in the file. Expert up-projections are Q4_K, but expert down-projections use
ggml type 20, a 32-value 4-bit block format, because their 2,688-wide rows are not a multiple of the 256 that K-quants
need (**measured** types, **reasoned** cause). The output head is Q6_K and the Mamba-2 parameters other than its
projections are F32. Over the 123.6B parameters in the file, 70.10 GB is about 4.5 bits per weight (**reasoned**).

So "~70GB RAM" is the size of the weights for text only. With vision you add the 1.73 GB projector. Then the caches:
at a 128K context, in llama.cpp's F16 KV and F32 recurrent state, about 1.07 GB of KV and 0.17 GB of Mamba-2 state
(**reasoned**). Call it 73 GB for one long conversation with images, plus compute buffers. A 96 GB machine is
comfortable, a 64 GB one is not, and NVIDIA's own target, the 128 GB DGX Spark, even fits the full million-token
context. See [continuous batching on a DGX Spark](/articles/dgx-spark-batching) for what that box does with many
sequences at once.

## LiLiCorr, the fourth repository

The `-NVFP4-LiLiCorr` repo is 1.32 GB and holds one `model.safetensors`. The card calls it "the LiLiCorr speculative
decoding checkpoint", for DGX Spark and low-concurrency serving: a 1.13B-parameter dense drafter (5 layers, 32 query
and 8 KV heads, an 8,192-token sliding window) that proposes seven tokens in one forward pass, plus a 37M-parameter
head that picks a mutually compatible sequence out of the top 8 candidates at each slot (**reported**).

It solves the same problem [DFlash 2](/articles/dflash2) attacks. A parallel drafter predicts each slot's marginal
independently, so its picks can each be plausible and still not fit together. LiLiCorr's head gives every candidate an
"out" and an "in" vector, scores neighbours by cosine similarity, and walks the block left to right taking the best fit.
Verification by the target is unchanged, so the output distribution is exactly the target's.

Does it beat the built-in MTP head? On SPEED-Bench at draft length 7 (**reported**), LiLiCorr accepts 4.74 tokens per
step greedy and 4.36 sampled. The boosted MTP head accepts 4.718 greedy and 4.612 sampled. So LiLiCorr does not accept
more tokens; it drafts them in one pass instead of seven recursive ones, which is what matters when a single user on a
memory-bound box is waiting.

## What I would take away

- The architecture is sized for long agentic sessions on a box with fixed memory. Forty Mamba-2 layers make the
  per-sequence state a constant 168 MB; eight narrow attention layers make the growing part 8 KiB per token. Its
  throughput claims rest on that, not on the experts.
- The 120B/12B labels hold. The routed experts are 90.6% of the parameters and well under half the per-token work.
- The local build is real: 70.10 GB of Q4_K_M plus 1.73 GB of vision projector, on a 96 or 128 GB machine.
- Read the benchmarks with care: it is strong on agentic coding and hallucination, weaker on vision than its name
  suggests, and the Terminal-Bench numbers need an explanation from NVIDIA.

For the earlier Nemotron work on 4-bit formats, see [Nemotron in NVFP4](/articles/nemotron-nvfp4) and
[SGLang's NVFP4 KV cache](/articles/nvfp4-kv-cache). For a mixture-of-experts primer, the
[mixture-of-experts architecture page](/architectures/mixture-of-experts).