Nemotron 3.5 Super VL: 88 layers, eight of which remember every token

mdjsonmcpcopy↗ai

2026-10-15 · 17 min · mixture-of-experts · state-space-models · vision-language-models · quantization · long-context · nvidia · gguf

Why read this

Notable· top 60%

Recounts 120B/12B from the headers, prices the hybrid's cache from config, explains LM Studio's 70 GB, and flags where the VL model loses.

• Original analysis
• Widely used
• A lasting reference

·LLM architecture·Needs a workstation GPU·OpenMDW-1.1·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

2 of 3: A reference for a year or more

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 66 of 100, ranked 205 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 2026-10-15 LM Studio posted: "Nemotron 3.5 Super VL from @NVIDIAAI is live in LM Studio! 120B total / 12B active MoE. Supports vision input, reasoning. Trained for visual understanding, tool use. Requires ~70GB RAM to run locally."

Behind the post is a real release from NVIDIA, in four Hugging Face repositories:

• NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-BF16, the post-trained reference weights.
• -NVFP4, the same model quantized to 4-bit experts, for deployment.
• -Base-BF16, the pretrained checkpoint before any post-training.
• -NVFP4-LiLiCorr, which is not the model at all. It is a 1.13B-parameter speculative drafter (more on it below).

There is a 79-page technical report, Nemotron 3.5 Super and Lightning, dated 2026-10-14. It is not on arXiv. The licence is the OpenMDW License Agreement 1.1, which grants permission "to deal in the Model Materials without restriction" and "does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs". That is closer to MIT than to the NVIDIA Open Model License earlier Nemotrons used. Its one sharp clause is a patent-retaliation termination.

I did not run the model. Everything below comes from files: config.json, the safetensors headers of the BF16 and NVFP4 repos (read by HTTP range request, about 51 MB of headers instead of 324 GB of weights), the NVFP4 repo's hf_quant_config.json, the GGUF headers of LM Studio's build, the remote modeling code, and the report. Numbers are labelled reported (NVIDIA or LM Studio says so), measured (I counted it in a file) or reasoned (my arithmetic on the other two, assumptions stated).

What kind of model this is

The card calls it a "Mamba-2–Transformer hybrid Mixture of Experts with a C-RADIO vision encoder". The report says the language backbone is initialized from Nemotron 3 Super Base, so this is the same family as Nemotron-H and Nemotron 3: most sequence mixing is done by Mamba-2 state-space layers, a few attention layers are kept for exact recall, and the feed-forward work goes to a sparse mixture of experts. The new parts in 3.5 are the training (a trillion tokens of distillation from Nemotron 3 Ultra Base, an Indic stage, agentic software-engineering mid-training, a two-stage SFT that teaches it vision), a recursively shared multi-token-prediction head, and the quantized and drafter releases.

The interesting question for anyone who wants to run it is not the training. It is what the architecture costs in memory, and that falls straight out of the config.

The stack, from config.json

llm_config.layers_block_type lists 88 blocks one by one (measured): 40 mamba, 40 moe and 8 attention. Each block is a single mixer or feed-forward behind an RMSNorm and a residual, so "88 layers" here means 88 half-layers in the usual transformer sense. The order is almost, but not quite, periodic. Mamba-2 and MoE alternate, and an attention block is dropped in at indices 7, 16, 25, 36, 47, 58, 69 and 78: gaps of 9, 9, 11, 11, 11, 11 and 9. Attention sits only in the middle of the stack, never at either end.

Each kind of block, from the BF16 headers (measured):

• Mamba-2 mixer, 109,640,064 parameters. 128 heads of head dimension 64, a state of 128 per head, 8 groups for the input-dependent B and C matrices, and a width-4 causal convolution. The input projection is [18560, 4096], and the 18,560 decomposes exactly: 8,192 for the gate z, 8,192 for x, 1,024 each for B and C (8 groups of 128) and 128 for the per-head step size. The mixer's inner width is twice the 4,096 hidden size.
• Attention mixer, 35,655,680 parameters. 32 query heads and 2 key-value heads of dimension 128. With only two KV heads, K and V together are 512 values per token per layer.
• Latent MoE block, 2,873,102,848 parameters. 512 routed experts, 22 used per token, plus one shared expert. This is where the parameters live. It gets its own section.

If you have read the ReplaySSM article, the Mamba-2 shapes are the same as Nemotron 3 Super's; if you have read Rigel, this is the same recipe at fifty times the size, with a much lower ratio of attention: 8 of 48 mixers here, against 10 of 40 there. For the mechanism of the state-space layer itself, see the Mamba architecture page.

Why eight attention layers decide the memory bill

An attention layer has to keep a key and a value for every token it has seen. A Mamba-2 layer compresses everything it has seen into a fixed state and throws the tokens away. That is the whole trade: attention can look anything up exactly; Mamba-2 pays a constant amount of memory but can only remember what fit through its state.

The arithmetic for one sequence (reasoned from config fields):

• KV cache. 8 layers × 2 KV heads × 128 dims × 2 (K and V) = 4,096 values per token. In BF16 that is 8 KiB per token, so a million-token context (the config's max_position_embeddings is 1,048,576) holds 8.6 GB of KV. In FP8, which the NVFP4 release serves by default, half that.
• Mamba-2 state. 40 layers × 128 heads × 64 × 128 = 41,943,040 values, plus a tiny convolution state of 10,240 channels × 3 per layer. The config asks for an FP32 SSM cache, which makes it 168 MB per sequence (plus 5 MB of convolution state), the same at token 10 and at token 1,000,000. The NVFP4 serving recipe keeps it in FP16 with stochastic rounding, half that.
• The crossover. The constant Mamba state equals the growing KV cache at about 20,480 tokens in BF16. Past that, the eight attention layers are the larger cost, and they keep growing.

Now replace the 40 Mamba-2 layers with the same narrow attention. Every one of the 48 mixers keeps 1 KiB per token, the cache grows six times faster, and the million-token context needs 51.5 GB of KV per sequence (reasoned). That is the hybrid's argument in one number. The widget above runs it for every context from 4K to 1M and for each release's cache precision, against the memory of an H100, a DGX Spark and an H200.

There is a nice consistency check in the report. It says that in vLLM "one Mamba cache page occupies at least as many bytes as approximately 4K tokens of Attention KV cache". One Mamba-2 layer's state is 1,048,576 values; one attention layer's KV is 512 values per token; at FP32 state and BF16 KV, 4 MiB divided by 1 KiB per token is 4,096 tokens (reasoned). The report's figure is the config's arithmetic.

What the constant state costs instead is in the report too. When prefix caching and speculative decoding run together, vLLM has to recover a Mamba state at an arbitrary position, and Mamba states are only checkpointed every few thousand tokens, so the engine replays tokens to rebuild one. NVIDIA added an extra checkpoint near the replay boundary and the --enable-mamba-shared-prefix-checkpoint flag that appears in every serving recipe on the card. The state is cheap to hold and awkward to rewind.

Latent MoE: experts that work in a quarter of the width

The MoE block is not a standard one, and its tensor shapes say exactly what it does (measured):

There is no gate matrix in the experts: the activation is ReLU², mlp_hidden_act: relu2, so each expert is two matrices, 5,505,024 parameters. The router picks 22 of 512 and normalises their weights (norm_topk_prob), and the config's routed_scaling_factor of 5.0 scales the sum.

Two things follow. First, a routed expert is small: 5.5M parameters, against 44M for the shared expert, so the 22 routed experts a token visits add up to 121M per layer, less than three shared experts' worth (reasoned). The routed pool is about storage, not compute. Second, the tokens that cross the network to experts on other GPUs are 1,024 values wide, not 4,096. The report credits this, "LatentMoE exchanges latent vectors that are smaller than the Qwen models' hidden states", for part of its throughput lead. Per token, per MoE layer, 22 dispatched copies at BF16 are 44 KiB in the latent against 176 KiB at full width (reasoned, before any FP8 dispatch). A quarter of the all-to-all.

Does 120B total, 12B active hold?

Summing every tensor in the 64 BF16 shards (measured):

The language model without the MTP head and the vision side is 120.67B, so "120B total" is an honest round-down of the backbone. The whole file, which the report itself uses to compute bits per element, is 124.45B.

Active per text token: every Mamba-2 and attention mixer, every shared expert, latent projection and router, 22 of the 512 routed experts in each MoE layer, and the output head. That is 12.23B (measured from shapes, with the selection rule from config). It leaves out the embedding table (a lookup reads one row), the MTP head (it only runs when speculative decoding is on) and the vision tower (it runs once per image). Both headline numbers hold. Routed experts are 90.6% of the stored checkpoint and 39.6% of the active compute; the always-on Mamba-2 mixers are 35.9%.

The vision side: C-RADIO, a pixel shuffle, and video packing

The vision tower is C-RADIO (LM Studio's page says C-RADIO v4-H), configured as vit_huge_patch16_224: 32 layers, hidden size 1,280, patch size 16, with three summary tokens from its distillation teachers and seven registers (measured, from vision_config). It accepts between 1,024 and 13,312 patches per image, so resolution is dynamic.

The connector is the InternVL-style one. A pixel shuffle with downsample_ratio: 0.5 folds each 2×2 block of patch features into one 5,120-wide vector, and a two-layer MLP (RMSNorm, 5,120 to 20,480, ReLU², 20,480 to 4,096) maps it into the language model's width. So an image costs a quarter as many tokens as it has patches: 256 at the minimum, 3,328 at the maximum (reasoned from the processor config and the modeling code).

Video goes through the same tower with two frames stacked on the channel axis (video_temporal_patch_size: 2), so a pair of frames is one ViT pass. The processor targets 1,024 patches per frame, which is 256 tokens per pair. The card's vLLM recipe samples up to 256 frames at 2 FPS, which would be 128 pairs and 32,768 tokens, and then --video-pruning-rate 0.5 drops half of them (reasoned). The pruning is Efficient Video Sampling, readable in modeling_nemotron_h_omni.py: it scores every token by cosine dissimilarity to the same position one frame-pair earlier, always keeps the first pair, and keeps the most-changed tokens up to the budget. A static slide deck costs almost nothing after the first frame. A handheld walkthrough costs nearly all of it.

The report is clear that vision came late: the base model is text-only, the projector is warmed up, and vision data arrives in SFT. That matters for the benchmarks.

What it scores, and where it loses

The headline figure from the card and the report puts it against six models:

expand

I counted the card's own table, 36 non-offline rows, treating hallucination rate as lower-is-better (measured from the card; every score in it is reported):

• Against Qwen3.5-122B-A10B, the closest model by size (122B total, 10B active), it wins 22, loses 13, ties 1. The wins are large on agentic and reasoning rows: SWE-bench Pro 51.78 against 44.9, Terminal-Bench 2.1 66.70 against 48.70, BrowseComp 78 against 58.1, and an AA-Omniscience hallucination rate of 30.0 against 84.5.
• Against Qwen3.8-Flash-Next it wins 6 and loses 29. Against GLM-5.3-Flash, 10 and 25. Against DeepSeek-V4-Flash, 4 and 21. NVIDIA put three models it mostly loses to in its own chart, which I count in its favour.
• The vision rows are the weak half. On the 11 multimodal rows it loses 8 of 11 to Qwen3.5-122B-A10B, the model it otherwise beats: V* 85.60 against 91.1, MathVision 78.16 against 83.3, OCRBench v2 Chinese 59.15 against 67.3, BabyVision 29.60 against 35.7. It wins only OCR-Reasoning, HLE full set and Video-MME v2, the last by 0.85 points. Against Qwen3.8-Flash-Next it loses 10 of 11. For a release whose name ends in VL, vision is not where it leads.
• MMLU-ProX, a multilingual knowledge test, is its worst row: 76.91, the lowest of all seven models, even after the Indic continued pretraining stage. And it loses τ2-bench telecom by 14 points to Qwen3.5.

The card's harness chart is the honest one for agentic coding. NVIDIA's headline SWE-bench Pro score of 51.78 comes from its own NeMo Gym harness; in six third-party agent harnesses it scores between 41.04% and 49.45%.

expand

Terminal-Bench does not agree with itself

Terminal-Bench 2.1 appears four times, with three different BF16 scores (reported, all of them):

The card's NVFP4 table says quantization costs 5.9 points on this benchmark. The report's table, which NVIDIA says holds "all configs, harnesses, and other non-model parameters" fixed, says NVFP4 gains 2.11. Both cannot describe the same runs. My guess is that the card's 66.70 is a later or differently configured run, but nothing in either document says so. Two smaller slips: the report's abstract and Section 4.1 give a 99.62% median accuracy recovery for NVFP4, its conclusion says 99.52%; the card says 67B tokens of software-engineering mid-training, the report says 65.4 billion.

NVFP4, and what "~70GB RAM" is

The NVFP4 release is a mixed-precision checkpoint, and hf_quant_config.json lists every layer (measured): the 20,480 routed-expert matrices and the 80 shared-expert matrices are NVFP4 with one FP8 scale per 16 values; the 40 Mamba-2 in/out projections, the 8 attention layers' q/k/v/o and the output head are FP8; the router, the latent projections, the convolutions, the embeddings and the whole vision path stay BF16. The KV cache is FP8. The files total 74.99 GB. The report computes 4.82 bits per element over all 124.453 billion parameters, which matches the files.

expand

On accuracy the report gives a 36-benchmark table (Table 23). Most rows move by under a point. The largest drop is MMLU-ProX, 76.91 to 73.76; the card also shows WebVoyager falling from 77.72 to 66.31 and DeepSWE from 24.65 to 18.2, both offline rows that are not in the report's table.

LM Studio does not ship either of NVIDIA's checkpoints. Its page links lmstudio-community/...-GGUF, which holds two quantizations (measured from the file listing):

• Q4_K_M: two shards totalling 70.10 GB. That is LM Studio's "~70GB".
• An NVFP4 GGUF: 75.24 GB.
• A separate mmproj file for the vision tower and connector, in BF16: 1.73 GB.

I range-read the first shard's GGUF header. It declares nemotron_h_moe with block_count 89: the 88 backbone layers plus the MTP layer, so the draft head is in the file. Expert up-projections are Q4_K, but expert down-projections use ggml type 20, a 32-value 4-bit block format, because their 2,688-wide rows are not a multiple of the 256 that K-quants need (measured types, reasoned cause). The output head is Q6_K and the Mamba-2 parameters other than its projections are F32. Over the 123.6B parameters in the file, 70.10 GB is about 4.5 bits per weight (reasoned).

So "~70GB RAM" is the size of the weights for text only. With vision you add the 1.73 GB projector. Then the caches: at a 128K context, in llama.cpp's F16 KV and F32 recurrent state, about 1.07 GB of KV and 0.17 GB of Mamba-2 state (reasoned). Call it 73 GB for one long conversation with images, plus compute buffers. A 96 GB machine is comfortable, a 64 GB one is not, and NVIDIA's own target, the 128 GB DGX Spark, even fits the full million-token context. See continuous batching on a DGX Spark for what that box does with many sequences at once.

LiLiCorr, the fourth repository

The -NVFP4-LiLiCorr repo is 1.32 GB and holds one model.safetensors. The card calls it "the LiLiCorr speculative decoding checkpoint", for DGX Spark and low-concurrency serving: a 1.13B-parameter dense drafter (5 layers, 32 query and 8 KV heads, an 8,192-token sliding window) that proposes seven tokens in one forward pass, plus a 37M-parameter head that picks a mutually compatible sequence out of the top 8 candidates at each slot (reported).

It solves the same problem DFlash 2 attacks. A parallel drafter predicts each slot's marginal independently, so its picks can each be plausible and still not fit together. LiLiCorr's head gives every candidate an "out" and an "in" vector, scores neighbours by cosine similarity, and walks the block left to right taking the best fit. Verification by the target is unchanged, so the output distribution is exactly the target's.

Does it beat the built-in MTP head? On SPEED-Bench at draft length 7 (reported), LiLiCorr accepts 4.74 tokens per step greedy and 4.36 sampled. The boosted MTP head accepts 4.718 greedy and 4.612 sampled. So LiLiCorr does not accept more tokens; it drafts them in one pass instead of seven recursive ones, which is what matters when a single user on a memory-bound box is waiting.

What I would take away

• The architecture is sized for long agentic sessions on a box with fixed memory. Forty Mamba-2 layers make the per-sequence state a constant 168 MB; eight narrow attention layers make the growing part 8 KiB per token. Its throughput claims rest on that, not on the experts.
• The 120B/12B labels hold. The routed experts are 90.6% of the parameters and well under half the per-token work.
• The local build is real: 70.10 GB of Q4_K_M plus 1.73 GB of vision projector, on a 96 or 128 GB machine.
• Read the benchmarks with care: it is strong on agentic coding and hallucination, weaker on vision than its name suggests, and the Terminal-Bench numbers need an explanation from NVIDIA.

For the earlier Nemotron work on 4-bit formats, see Nemotron in NVFP4 and SGLang's NVFP4 KV cache. For a mixture-of-experts primer, the mixture-of-experts architecture page.