# Comet 60B: Nemotron 3.5 Super VL with half of every expert pool cut > Satyajit Ghana — Head of Engineering @ Inkers Technology > canonical: https://ai.thesatyajit.com/articles/comet-60b > date: 2026-10-16 > tags: mixture-of-experts, pruning, distillation, vision-language-models, quantization, nvidia On 2026-10-15 Multiverse Computing posted: "Meet Comet 60B, in collaboration with @nvidia. Built from Nemotron 3.5 Super VL: 50% fewer total parameters, Within 5% of original accuracy, Text, image & video. Open-source multimodal AI for agentic workloads." The parent is the model I took apart a day earlier in [Nemotron 3.5 Super VL: 88 layers, eight of which remember every token](/articles/nemotron-3-5-super-vl): a hybrid of 40 Mamba-2 layers, 8 attention layers and 40 latent mixture-of-experts layers, with a C-RADIO vision tower, 124.45B parameters in the checkpoint and 12.23B active per text token. I reuse its counts here rather than redo them. Comet ships as three Hugging Face repositories, `Comet_60B_A12B_BF16`, `_FP8` and `_NVFP4`, with one model card for all three and a [press release](https://multiversecomputing.com/resources/multiverse-computing-launches-comet-60b-in-collaboration-with-nvidia). There is no paper. The model is credited to CompactifAI, Multiverse's "quantum-inspired" compression, which their 2024 paper ([arXiv 2401.14109](https://arxiv.org/abs/2401.14109)) describes as replacing weight matrices with tensor networks. So the question is mechanical: what was removed to halve the parameter count, and do the claims hold? I did not run the model. Everything below comes from files: both `config.json`s, both BF16 safetensors headers (22,087 tensors against 43,079), the router weights of every MoE block in both and a few whole experts (about 300 MB of HTTP range reads out of 371 GB of weights), the quantization configs and the card. Numbers are labelled **measured** (I counted it in a file), **reported** (Multiverse or NVIDIA says so) or **reasoned** (my arithmetic on the other two). ## What halving an MoE can mean In the parent, every MoE block has 512 routed experts, each a pair of 1,024 × 2,688 matrices working in a 1,024-wide latent, and a router that sends each token to 22 of them. Those routed experts are 112.74B of the 124.45B parameters, 90.6% of the checkpoint (**measured**, in the parent article). Anything that halves the model has to go through them. There are four obvious ways in: - **Drop layers.** Cuts active and total parameters alike. - **Drop experts**, keeping the number picked per token. Cuts total, leaves active nearly alone. - **Narrow experts** by removing hidden channels. Cuts total and active in proportion. - **Factorise matrices** into low-rank or tensor-network products. That would show up as new tensors with new shapes: factors, cores, bond dimensions. The headers say which of these happened. ## The diff, tensor by tensor Comet's `config.json` differs from the parent's in exactly two fields (**measured**): `n_routed_experts` goes from 512 to 256, and `moe_intermediate_size` from 2,688 to 2,432. Every other field is identical: 88 blocks in the same order, hidden size 4,096, 128 Mamba-2 heads, 32 query and 2 key-value heads, 22 experts per token, the same vision config, the same one-layer MTP head, the same 1,048,576 maximum positions. The safetensors headers agree. After collapsing expert indices, the two checkpoints have the same set of tensor names, and only four kinds of tensor change shape, in each of the 40 backbone MoE blocks and in the MTP block (**measured**): | Tensor | Parent | Comet | |---|---|---| | `gate.weight` (router) | `[512, 4096]` | `[256, 4096]` | | `gate.e_score_correction_bias` | `[512]` | `[256]` | | `experts.N.up_proj` | `[2688, 1024]` × 512 | `[2432, 1024]` × 256 | | `experts.N.down_proj` | `[1024, 2688]` × 512 | `[1024, 2432]` × 256 | No layer is missing. No new tensor appears. The Mamba-2 mixers, the attention mixers, the shared experts (still 5,376 wide), the latent projections, the embedding table, the output head and the whole vision path keep their shapes. There are no factor tensors anywhere: every weight in Comet is a plain dense matrix with either the parent's shape or a slice of it. Whatever "tensor network" means in this release, it is not in the weights. So Comet is two of the four options at once: half the experts, and each surviving expert 9.5% narrower. ## The config says how The same `config.json` carries a block the parent does not have, `compactifai_pruning`, which reads like an export of the pipeline's own settings (**measured**; it also holds internal file paths, which I leave out). The fields that matter: - `"method": "REAP post_up, domain protection over 6 groups (reasoning/code/general/agentic/image/video), …"` - `"experts": 256` and `"intermediate": 2432` - for channels, `"criterion": "reap (act_wabs * ||fc2 @ down_col||)"` and `"compensate": "ls ridge 0.0001"` - for the channel refit, a calibration subset of 745 samples, 5,005,192 tokens, of a 32K-context text set - `"dropped_tensors": 20992` REAP is Cerebras' Router-weighted Expert Activation Pruning, [arXiv 2510.13999](https://arxiv.org/abs/2510.13999). The idea derives in a few lines. When a token goes through an MoE block, the output is the sum, over the experts the router picked, of the router's gate value times that expert's output. If you delete an expert, the error you introduce on a token is roughly that gate value times the norm of that expert's output, and only on the tokens where the router would have picked it. So run calibration data through the model, and for each expert average gate × output-norm over the tokens routed to it. The experts with the smallest scores contribute least to the block's output; delete those. The paper argues this beats merging experts for generative tasks, and reports near-lossless code generation on Qwen3-Coder-480B and Kimi-K2 after removing half the experts. "Domain protection over 6 groups" reads as: the scores were computed separately on reasoning, code, general, agentic, image and video data, so that an expert that matters only for video is not pruned because text calibration never wakes it. The config does not say how the six were combined, and there is no write-up to check it against. The channel step applies the same reasoning one level down. Inside an expert, a hidden channel's contribution is its activation times the norm of the matching column of the down projection, pushed through the latent's up-projection (`fc2`). Channels with the smallest average contribution go, and the down projection is then refitted by ridge-regularised least squares so that the narrower expert reproduces the wider one's output on calibration tokens as closely as it can. That is the `ls ridge 0.0001`. The arithmetic checks out. 40 blocks × 256 removed experts × 2 matrices is 20,480 tensors, and the MTP block's 256 × 2 make 20,992, the `dropped_tensors` count (**reasoned**). ## Which experts went The checkpoint renumbers the surviving experts 0 to 255, so it does not say which of the parent's 512 they were. The routers do. A router row is the vector the router dots with a token's hidden state to score one expert, and fine-tuning moves it only slightly. For every MoE block I read both routers in full and matched each of Comet's 256 rows to the parent row with the highest cosine similarity (**measured**): - In all 41 blocks every match is unique: 256 different parent experts. - Every match is order-preserving: Comet's expert 0 is the lowest-numbered survivor, and so on. The pruning kept the parent's order and dropped the rest. - The backbone matches are unambiguous. Matched cosines have a median between 0.988 and 0.999 per block and never fall below 0.898; the best non-matching parent row never passes 0.703. In the MTP block the margin is narrower, 0.866 against 0.843, but still clean. No block keeps a contiguous range; the survivors are scattered differently in every block, which is what a data-driven score looks like. Two more things fall out of the match. First, the router rows drifted further in deeper blocks: the median matched cosine is 0.999 in the first blocks and 0.988 to 0.992 in the last few (**measured**). Healing moved the late layers more. Second, the parent's `e_score_correction_bias`, the per-expert bias the router uses for load balancing, does not predict survival. In every block, kept and dropped experts have the same mean bias to within 0.03 (**measured**). REAP is measuring something the load balancer was not. The channels behave the same way. For four experts I spot-checked (blocks 1, 45 and 87), the 2,432 up-projection rows match 2,432 distinct parent rows in order, with cosines above 0.98 (**measured**). The kept rows moved 5% to 7% in relative norm from the parent's, and the down projections 9% to 15%, which is the least-squares refit plus healing (**measured** on those four, **reasoned** cause). ## Everything else moved a little Unchanged shape is not unchanged weights: the card says the pruned model was healed by distillation from the parent, then trained with GRPO-style reinforcement learning. I compared the first 2 MB of a dozen tensors that kept their shape (**measured**): - The vision tower is close to untouched. In the first and last C-RADIO blocks, 84% and 98% of the BF16 values are bit-identical to the parent's, with relative changes of 0.04% and under 0.01%. - The language model's shared parts moved more but not much: 0.06% to 0.09% relative change in the latent projection, a Mamba-2 input projection and an attention query projection; 0.22% in a shared expert. A layer norm is bit-identical. - The MTP head's input projection moved 6.6%. So the vision tower is intact: same architecture, same 0.65B parameters, nearly the same weights. Whatever Comet lost on vision, it lost in the experts, not the encoder (**reasoned**). The MTP number has an explanation in a second file. `_convert.json`, which records how the Megatron checkpoint was exported, says the main weights came from a reinforcement-learning checkpoint while all 528 MTP tensors were "reference graft": copied from the healed, pre-RL model (**measured**). The config also names a separate MTP source dated 2026-09-21. The draft head was therefore never trained against the RL'd model it drafts for. MTP decoding is lossless, so this cannot change outputs; it can lower the acceptance rate, which the card does not report (**reasoned**). ## 60B total, 12B active? Summing every tensor in both BF16 headers (**measured**): | Component | Nemotron 3.5 Super VL | Comet 60B | |---|---:|---:| | Routed experts | 112.74B | 51.00B | | Mamba-2 mixers | 4.39B | 4.39B | | MTP draft head | 2.94B | 1.40B | | Shared experts | 1.76B | 1.76B | | Vision encoder and connector | 0.84B | 0.84B | | Embedding table | 0.54B | 0.54B | | Output head | 0.54B | 0.54B | | Latent projections, routers, norms | 0.42B | 0.38B | | Attention mixers | 0.29B | 0.29B | | **Total** | **124.45B** | **61.13B** | 61,126,735,112 against 124,453,395,976: 50.9% fewer parameters (**reasoned**). The card's 61.1B is exact. The language model alone falls from 120.67B to 58.89B. The 60 in the name rounds down the whole checkpoint; the press release's "120 billion" for the parent rounds down only its language model. Active per text token, counted the same way as in the parent article (22 routed experts per block plus everything always on and the output head; not the embedding lookup, the MTP head or the vision side): **12.23B for the parent, 11.73B for Comet** (**measured** from shapes, with the selection rule from config). The card's 12.3B and 12.8B add the embedding table; its 11.2B and 11.7B drop the output head too, and match mine less its 0.54B. A reply under the post asked whether this is "basically more active parameters". It is slightly fewer: the token still visits 22 experts, but each is 9.5% narrower, so the routed share of a token's compute falls from 121.1M to 109.6M per block and the whole active count by 4.1% (**reasoned**). This is the trade the design makes. Memory to hold the model halves; compute per token barely moves. That is the right trade for an MoE, where the routed pool is storage and the rest of the network is the work. ## What the half buys Checkpoint sizes, summed from the repositories' safetensors files (**measured**): | Format | Nemotron 3.5 Super VL | Comet 60B | |---|---:|---:| | BF16 | 248.91 GB | 122.26 GB | | FP8 | not published | 66.28 GB | | NVFP4 | 75.01 GB | 40.48 GB | The FP8 checkpoint follows NVIDIA's FP8 layout for Nemotron 3 Super: experts and Mamba-2 projections in FP8, everything else in BF16. The NVFP4 checkpoint copies the parent's NVFP4 layout, and its `hf_quant_config.json` lists 20,563 NVFP4 modules and 113 FP8 ones (**measured**); 113 is the 80 Mamba-2 projections, the 32 attention projections and the output head (**reasoned**). Because the experts were halved and the BF16 and FP8 parts were not, Comet's NVFP4 file averages 5.3 bits per parameter (**reasoned**), against the 4.82 NVIDIA's report computes for the parent (**reported**). The card's minimums follow from these sizes: BF16 on 2 H200s, FP8 on 1 H200, NVFP4 on 1 B300 (**reported**). The attention and Mamba-2 layers are identical, so the KV cache is still 8 KiB per token in BF16 and the recurrent state a constant 168 MB per sequence, as priced in the parent article. What changes is the room left for them: on 4 H200s the card reports KV capacity of 24.0M tokens against 14.1M (**reported**). The card's throughput table reports Comet serving 20% more chat and 23% more agentic output tokens per GPU on the same 4 H200s, with inter-token latency down from 23.4 ms to 19.1 ms on chat (**reported**, vLLM with MTP, NVIDIA's Dynamo traces). Active parameters fell only 4.1%, so where does 20% come from? Two places, I think (**reasoned**). One is the cache room: more concurrent sequences, more prefix-cache hits. The other is batching. One token reads 22 experts either way, but a decode step for a batch reads every expert any token in the batch picked. With 32 tokens per step and uniform routing, the parent touches about 386 of its 512 experts per block and Comet about 242 of 256, so the expert weights read per step fall from 4.26 GB to 2.41 GB per block in BF16, about 43% less. At 128 tokens per step both models read nearly every expert, and the saving approaches the full 55%. Real routing is skewed, so true counts are lower, but the direction holds. The widget's batch slider runs this. ## "Within 5%": which 5% The card has the right table: Multiverse ran the parent and Comet on the same vLLM build, harnesses, prompts, judges and sampling, 35 scores in all. Every score is **reported**; the counts are mine (**measured** from the card). The card's own summary is that Comet BF16 is "within 3 points of the original on 28 of the 35 scores below and within 5 points on 32". Both hold. It is ahead on 9 rows: all three τ²-Bench domains and their average, AutomationBench-AA, SciCode, SWE-bench Pro, GDP.pdf mean and BabyVision. The post says "Within 5% of original accuracy". Five points and five percent are far apart on low-scoring benchmarks. As a relative drop, Comet BF16 is within 5% of the parent on 26 of 35 rows. The nine that are not: | Benchmark | Parent | Comet BF16 | Change | |---|---:|---:|---:| | AA-Omniscience Index | 5.08 | 2.83 | −44.3% | | GDP.pdf all-pass | 6.20 | 5.20 | −16.1% | | HLE, no tools | 28.59 | 25.35 | −11.3% | | OCRBench v2 CN, EvalScope | 61.27 | 55.88 | −8.8% | | Terminal-Bench 2.1 | 60.67 | 55.76 | −8.1% | | Video-MME v2, group score | 18.26 | 16.82 | −7.9% | | GDPval-AA v2.1 | 74.51 | 69.05 | −7.3% | | V\* | 90.58 | 84.82 | −6.4% | | Video-MME v2, linear score | 33.82 | 31.74 | −6.2% | AA-Omniscience is an index that can go negative (Comet's NVFP4 scores −8.29 against the parent NVFP4's 5.38), so its percentage means least. By area (**reasoned** from the card's rows): - **Agents gain.** Five of the seven agent and tool-use rows go up, by 0.75 to 2.53 points; τ³-Banking slips 0.38 and GDPval-AA falls 5.46. The card credits the RL stage. - **Hard knowledge and terminal work lose.** HLE −3.24, Terminal-Bench −4.91, SWE-bench Verified −2.07. - **Vision and video lose almost everywhere.** 16 of the 17 image, OCR, document and video rows are lower, by 2.35 points on average; only BabyVision rises. The worst are V\* (−5.76), Chinese OCR (−5.39 and −2.48) and both Video-MME variants. Vision is where half the experts cost most, though the encoder was barely touched. That fits the mechanism (**reasoned**): pruning removes capacity the calibration set does not exercise, and the channel refit's calibration was text. If image tokens routed to experts text rarely woke, they lost them.
### The charts use different numbers from the table The model card and the press release both lead with bar charts, and the charts do not plot the table. - **Every Comet bar is the FP8 checkpoint**, labelled only "Comet 60B": GPQA 88.57, HLE 24.51, Terminal-Bench 57.30 (**measured** against the table). - **The parent bars are a third set of numbers.** The charts show the parent at 90.40 on GPQA Diamond, 30.03 on HLE, 64.19 on Terminal-Bench, 52.44 on SWE-bench Pro, 74.80 on Video-MME. The card's own table has 89.90, 28.59, 60.67, 50.30 and 74.48. NVIDIA's card has 89.52, 31.79, 60.67 and 51.78. None of the three agree, and nothing says where the chart baseline came from (**measured**, against both cards). This baseline is higher than the table's on all eight text rows and lower on five of the six vision rows, so it does not consistently flatter Comet; it is just unexplained. - **The "Where Comet 60B Leads" chart picks the better checkpoint per row.** It shows SWE-bench Multilingual 69.33, τ²-Bench telecom 83.11 and BabyVision 30.93, all FP8, where the BF16 scores are 66.56 (below the parent's 67.00), 79.28 and 28.09. And it labels τ³-Banking "τ²-Bench banking". - **The press release then confuses the two.** It says the largest gains are in "tau2-Bench banking (76.75 vs 83.11)", which are the telecom numbers, and three paragraphs later says telecom "scores above the original (81.91)", a different baseline for the same row.
None of this changes the table, the one like-for-like comparison. On it, FP8 is as good as BF16: within 3 points on 27 of 35 rows, a mean change of −1.10 points against −1.46. The card itself says single-run benchmarks move 1 to 2 points between runs, so gaps under about 3 points are noise.
## Licence, and loose ends The card puts Comet under the NVIDIA Nemotron Open Model License and says it is "Licensed by NVIDIA Corporation". The parent's public repository is under OpenMDW 1.1, which, as I noted in the parent article, is closer to MIT and places no restriction on how a derivative is licensed. The stricter licence is permitted, but it is not inherited from the public parent; it may come with the early access Multiverse says it had (**reasoned**). "Open source" here means open weights under a custom licence; the model code is NVIDIA's, under Apache 2.0. The card still has loose ends (**measured**): an unfilled release date, `TODO` markers in the citation block, format links to `Comet-1.0-60B-*` repositories that do not exist (the organisation lists only the three `Comet_60B_A12B_*` repos), and an FP8 command that spells the organisation `MMultiverseComputingCAI`. None of it touches the weights. ## The sibling: Hypernova 60B Multiverse's earlier `Hypernova-60B-2605` did the same thing to OpenAI's gpt-oss-120b, and the two configs show the recipe (**measured**): 36 layers became 32, 128 experts per layer became 80, and the expert width went from 2,880 to 2,560, with 4 experts per token kept. Hypernova also dropped layers; Comet did not. Both are structured pruning plus healing, and neither checkpoint contains a tensor network. ## What I would take away - Comet is Nemotron 3.5 Super VL with REAP expert pruning: 256 of 512 experts kept per MoE block, each 9.5% narrower, then distillation and RL. Nothing else was cut, and the routers confirm it expert by expert. - The headline counts hold: 61.13B total, 50.9% fewer, with 11.73B active against 12.23B. The memory halves; the compute per token does not. - "Within 5%" is five points, not five percent. On the card's table, agent benchmarks improve, hard knowledge and terminal work lose 3 to 5 points, and vision loses on 16 of 17 rows. - Read the table, not the charts. The charts mix in FP8 scores and a parent baseline that appears nowhere else. - If tensor networks were used, they decided what to cut; they do not represent what is left. The press release says as much: "Tensor-network profiling" first, then a step that "removes the least relevant parts". That profiling is the one step I cannot check. For more on pruning experts by router statistics, see [runtime dynamic compression](/articles/runtime-dynamic-compression), which swaps REAP selections at runtime, and the Soft-REAP experiment in [Qwen3.8-Flash-Next](/articles/qwen3-8-flash-next). For the NVIDIA tooling the FP8 and NVFP4 checkpoints were made with, [NVIDIA Model Optimizer](/articles/nvidia-model-optimizer) and [Nemotron in NVFP4](/articles/nemotron-nvfp4). For the basics, the [mixture-of-experts architecture page](/architectures/mixture-of-experts).