Le Chonk's weights, read from the files: 1.05T, 52B active, and a 41 KB-per-token cache

mdjsonmcpcopy↗ai

2026-11-01 · 21 min · mixture-of-experts · kv-cache · quantization · benchmarks · security · agentic-coding

Why read this

Solid· top 85%

Counts parameters, cache and NVFP4 size from the 54 shard headers, scores my preview predictions, and reads Mistral's chart data against its prose.

• Original analysis
• Widely used
• Concrete numbers to act on

·LLM architecture·Needs datacenter GPUs·Custom licence·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

1 of 3: API-only, gated or restrictive licence

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 58 of 100, ranked 437 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

On 29 October Mistral posted that it was releasing "the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity". Three claims follow in the post: it "ranks #1 on Cognition's FrontierCode among open models", it "sets a new benchmark for Pareto efficiency" on Artificial Analysis, and it is "state-of-the-art in cyberdefense". The thread has one more post, a link to the blog and to the Hugging Face repository.

I wrote about the launch preview three weeks ago, when there was an API and no files. That article ended with a list of things I would open first when the weights landed. This one opens them. The repository is 1.06 TB, so I did not download it: each safetensors shard starts with an 8-byte length and a JSON header, and an HTTP range request reads exactly that. Fifty-four range requests gave me the name, dtype and shape of every tensor in the model (measured). Everything below is labelled measured (from those files, or from a local run), reported (a number Mistral or another party published, which I did not reproduce) or reasoned (arithmetic or inference from the first two).

What shipped

The model card calls it "Mistral Large 4 - Le Chonk" (reported). The headline facts it gives, which the next sections check against the files:

• about 1T total and 49B active parameters, 52B including embeddings and the output layer;
• 128 routed experts, 4 active per token, plus 1 shared expert;
• a 1M-token context, with a Pixtral-style vision encoder for image input and text output;
• three reasoning efforts, low, medium and high (the default);
• an embedded speculative-decoding draft head called DSpark, with "up to 1.9x" throughput;
• trained from scratch on 3,800 NVIDIA Grace Blackwell GPUs in Mistral's own datacentres in Europe.

The weights are not the preview model. The blog says three more weeks of reinforcement learning went in between, and gives the preview's own numbers next to the release's on several charts. So everything the preview article could say about benchmarks was about a different checkpoint.

An NVFP4 sibling repository, Mistral-Large-4-1T-A52B-NVFP4, is also published; I used only its file sizes.

Counting the parameters

The index file says total_size is 1,060,548,427,328 bytes. Summing every tensor in the 54 headers gives the same number to the byte (measured). Counting elements instead of bytes, and leaving out the 63.8M FP32 scale values that sit next to the FP8 weights, gives 1,053,161,553,184 parameters.

Three things stand out in the ledger.

The experts are the model. There are 71 sparse layers (layer 0 is dense), each with 128 experts of three 6,144 x 6,144 matrices, so one expert is 113,246,208 parameters and the routed pool is 71 x 128 x 113,246,208 = 1,029,181,538,304. That is 97.7% of everything stored (measured). The shared expert has the same shape, one per sparse layer, and is 8.04B in total.

"49B active" and "52B active" are both right. One text token goes through all the attention weights, the dense layer-0 FFN, the shared experts, the output head and 4 of 128 routed experts per layer. That is 51.08B parameters, which is 49.82B without the output head and 52.34B with the embedding table counted too (measured, then reasoned for the 4/128 factor). The card's "49B / 52B" matches if 49.82 is truncated. The embedding table is a lookup of one 12 KB row, so a bandwidth estimate should count the output head and not the table.

The vision encoder is 1.637B, the draft head 2.10B. The preview article quoted a 1.6B vision encoder from Mistral's docs; the tensors say 1,637,107,712, plus 67M in the projector (measured). The DSpark head is stored in BF16 inside the same checkpoint: three dense MLA layers, an embedding and an output matrix of 204,800 x 512 each, and a small recurrent mixing layer. It is 4.2 GB and idle unless you turn speculation on.

The formats are mixed. Experts, attention projections and the shared expert are FP8 E4M3 with FP32 scales on 128 x 128 blocks. The embedding, output head, router gates, wkv_a, wkv_b and norms stay BF16. That matches the modules_to_not_convert list in config.json.

My preview article's predictions, scored

The preview article estimated from Large 3's configuration, because that was all that existed. Here is how that went.

The NVFP4 line is worth a second look, because the miss turned into a check. If only the routed experts are 4-bit, at 4.5 bits each (4 bits plus one 8-bit scale per 16 values), and everything else is left as it is in the FP8 checkpoint, the expected size is 1,029,181,538,304 x 0.5625 + 31,115,624,000 bytes of non-expert tensors = 610,030,239,296. The Hub reports 610,096,072,460 for the NVFP4 repository. The difference is 65.8 MB, or 0.011%, and the two tokenizer files in that repo are 53.7 MB of it (measured, then reasoned). So the 4.77 bits I calibrated on Large 3 was too high for this model: the shared expert and the attention are a smaller share here, and those are the tensors the quantizer leaves alone.

The attention, and why the cache is small

params.json and config.json agree on the shape of the attention, and it is not Large 3's.

• MLA with no query compression. q_lora_rank is null: wq is a full 8,192 x 6,144 matrix (64 heads x 128). The key-value path is compressed to a 512-wide latent plus a 64-wide rotary part, so wkv_a_with_mqa is 576 x 6,144 (measured).
• Cache sharing across layers. kv_source_layers points layers 39, 43, 47, 51, 55, 59, 63, 67 and 71 at layer 35's cache. Nine layers have no wkv_a_with_mqa tensor at all, so 63 of 72 layers own a cache (measured).
• A sparse-attention indexer. Forty-four layers carry an indexer (wq_b, wk, a weights projection) with 32 heads of 128 dimensions and index_topk 2,048. Twenty-eight layers are marked shared and reuse the selection of an earlier layer (measured). I read the indexer as a DeepSeek-style lightning indexer that scores every cached key and keeps the best 2,048 for the real attention. That reading is reasoned from the tensor names and config, not from the modelling code, which I did not read.
• Three RoPE layers, then one without. no_rope_layers repeats 1,1,1,0, so every fourth layer has no rotary embedding, and YaRN scaling is set with factor 2,048 from an original 512 (measured).
• An FP8 cache. params.json sets fp8_kv_cache and fp8_indexer to true.

Per token, the cache holds 576 bytes in each of 63 layers, 36,288 bytes, plus 128 bytes of indexer key in each of 44 layers, 5,632 bytes. Total 41,920 bytes, 40.9 KiB (measured layer counts and widths, reasoned that the indexer key is cached once, at one byte per value). The same model with ordinary multi-head attention in BF16 would store 72 x 64 heads x (128 key + 128 value) x 2 bytes = 2,359,296 bytes a token. The ratio is 56.3. At the full 1,048,576 tokens, that is 44.0 GB against 2.47 TB.

The widget counts weights plus one sequence's cache against 85% of GPU memory, because 0.85 is the --gpu-memory-utilization in the card's own vLLM command. It gives one result I did not expect. The card's recipe is --tensor-parallel-size 8 --gpu-memory-utilization 0.85. On eight 141 GB cards that budget is 958.8 GB, and the FP8 weights alone are 1,060.5 GB, so it does not fit (reasoned, with nominal memory sizes). Eight B200 at 192 GB give 1,305.6 GB and leave 245 GB of cache, which is 5.8M tokens of single-sequence cache, so the recipe reads as written for that class of node. On H200 you need the NVFP4 checkpoint, or a higher utilization setting and almost no room for anything else.

The cache is cheap, but not free of scaling: the indexer has to look at every cached key for every new token. At 1M tokens that is 44 layers x 128 bytes x 1,048,576 = 5.9 GB per sequence per step, against 74 MB for the selected 2,048-key attention itself (63 x 2,048 x 576 bytes) (reasoned). The weights are shared between sequences in a batch and the indexer scan is not, so at long context the scan is what stops batching from helping.

What a decode step reads

Decoding is mostly a memory-bandwidth problem. For one token the step reads about 53.15 GB in FP8: 20.99 GB that every token needs, and 32.16 GB for its 4 chosen experts in each of 71 layers (measured sizes, reasoned sum). A node of eight H200 has about 38.4 TB/s of nominal aggregate bandwidth, so the ceiling for one stream is around 720 steps a second. Real engines get a fraction of that.

The catch is batching. Two requests in the same step rarely pick the same experts, so the step reads more of the pool. If each token picks its 4 experts independently and uniformly, the share of a layer's experts touched by a batch of B is 1 - (1 - 4/128)^B.

The numbers from that formula (reasoned, FP8):

Past a batch of about 100 every expert is read on every step, and the cost per token falls as 1/B from there. Below it, a MoE model is wasteful: at B = 32 the step reads 12.8 times the bytes of B = 1 and produces 32 tokens, a 2.5x gain in throughput from 32x the streams. This is why "49B active" describes compute, and not what a busy server pays.

It also bears on DSpark. The card reports "up to 1.9x" throughput. A verification step over a 3-token draft plus one real token is a batch of 4 to the experts: 15.3 distinct experts a layer, 143.7 GB, or 2.7 times the bytes of a single-token step. To be 1.9x faster on a purely bandwidth-bound model, each step would have to accept 5.1 tokens, and a 3-token draft cannot give more than 4. So the uniform-routing model cannot reach 1.9x at batch 1. Either consecutive tokens share experts a lot more than random picks would (likely, since a draft head is proposing text from one local context), or the reported figure comes from a regime where attention and communication cost more than the weights. I cannot tell which from the files. What I can say is that the uniform assumption in my widget is pessimistic for speculation, and that the "up to" in the claim is carrying weight.

The tokenizer and the chat template

I ran both locally, with tokenizers 0.23.3 and jinja2 3.1.6 in a throwaway virtual environment, and not transformers, which does not yet know this architecture. That means the template ran in plain Jinja and not in the sandboxed environment transformers uses. The differences are in tojson escaping of characters such as <, which my test tool schema did not contain.

Tokenizer (measured). tokenizer.json is a byte-level BPE with a vocabulary of 204,800. Its 1,000 added tokens are all special: <s> is id 1, </s> is 2, [INST] 3, [/INST] 4, [AVAILABLE_TOOLS] 5, [TOOL_RESULTS] 7, [TOOL_CALLS] 9, [IMG] 10, [SYSTEM_PROMPT] 17, [ARGS] 32, [THINK] 34, [/THINK] 35 and [MODEL_SETTINGS] 36. Digits are split one per token. A tiny sample of bytes per token, single sentences each so treat it as an anecdote:

The non-English rows are the point. Hindi at 7.2 bytes per token is 2.8 characters a token, so a 160-language claim is at least consistent with a vocabulary that spends tokens on scripts. Code and JSON are the expensive ones, and those are what an agent's context fills with. Tokens per task, in the Pareto argument below, depend on the tokenizer as well as on how the model talks.

Template (measured). For a plain user message, the template produces:

<s>[MODEL_SETTINGS]{"reasoning_effort": "high"}[/MODEL_SETTINGS][INST]Fix the failing test.[/INST]

That is 20 tokens. Three things in it are easy to miss. There is no default system prompt (the template's default is an empty string). reasoning_effort is always written, defaulting to high, so a client that never sets it gets the most expensive mode and the most reasoning tokens. And the generation prompt adds nothing after [/INST]: the model starts straight into its reply, with [THINK]...[/THINK] as the reasoning block. A system message goes first as [SYSTEM_PROMPT]...[/SYSTEM_PROMPT]; with tools, [AVAILABLE_TOOLS][...] precedes the settings; a tool call is rendered [TOOL_CALLS]run[ARGS]{"cmd": "ls"}</s> and its result [TOOL_RESULTS]a.py[/TOOL_RESULTS]. A conversation that starts with an assistant turn raises "Conversation must start with a user or system message", which is the template's own guard.

What the benchmarks are, and what the charts show

The blog's charts are not images. Each carries a JSON spec with its labels and values in the page, so I read the numbers from those and not by eye. These are reported by Mistral, with the margins computed by me.

FrontierCode. Cognition's page (8 June 2026) describes a set of 150 tasks in three nested subsets (Diamond, Main, Extended) from 36 open-source repositories, written by more than 20 maintainers. A solution passes only if it clears every "blocker", a criterion "a maintainer would consider hard stops during code review". A failing solution scores 0 and a passing one gets a weighted rubric score; each model runs five times per reasoning effort and the best effort is reported. The tasks are private and "we don't currently plan to release the tasks publicly". The Cognition page does not use the name "2.0". Artificial Analysis describes its implementation as 132 private tasks, three attempts per task, the Stirrup harness capped at 500 turns, and a model as judge. The blog says "Cognition's private FrontierCode 2.0 benchmark" and does not say who ran it, with which harness or how many attempts. I could not tell which of these definitions the 19.7% uses.

What it is, though, is a mean score and not a pass rate: failures count as zero and passes are weighted. In the chart Large 4 is at 19.7. The next bars are DeepSeek V4.1 Flash at 19.3, GLM-5.3 at 17.8, Kimi K3 at 17.3, GLM-5.3 Flash at 14.2 and Beam 1.0 at 2.9. The lead over second place is 0.4 points. If I treat it as a pass-rate-like quantity on 132 tasks, one standard error at a score near 0.2 is sqrt(0.2 x 0.8 / 132) = 3.5 points (reasoned; the true spread depends on a rubric score I cannot see, and per-task correlation between models would make a difference between two of them tighter). A 0.4-point gap on a private set is a statement that Large 4 is in the leading group, and "ranks #1" is true to the decimal and not much beyond it.

Terminal-Bench 4.0 (via the Artificial Analysis Coding Agent Index). Large 4 41.4, GLM-5.3 40, DeepSeek V4.1 Flash 31, Kimi K3 21, Muse Spark Open 6, Beam 1.0 3. The preview, labelled "self reported", was 28. The lead is 1.4 points. The 13-point gain over the preview is the larger, and better-supported, story.

Coding Agent Index and the Pareto claim. The scatter chart gives six points, score against average millions of tokens per task:

Checking the prose against the points: 1 - 8.5 / 15.9 = 46.5%, which rounds to the "47% fewer tokens" in the card (measured from the spec, so the claim is internally consistent). On this chart only GLM-5.3 beats high effort on score, by one point, at 1.9 times the tokens; Kimi K3, DeepSeek Flash and Muse Spark all sit behind it on both axes. So "Pareto efficiency" holds in tokens per task. Tokens are not dollars. A 1T-parameter model costs more per token to serve than a flash-class one, and the chart does not carry a price axis, so I would read it as "cheap to run an agent loop on, per task" and not "cheap". The medium-to-high step is +13 points for +2.7M tokens (+47%), and the default is high.

The blog also shows a Surge Code human-preference chart: Large 4 wins 64% of prompts against GLM-5.3 and Kimi K3, 56% against DeepSeek V4.1 Flash, and 22% against Claude Opus 5.5 (reported).

Cyber: what is claimed

I leave the demonstrations alone and cover what is reported.

The blog says Large 4 scores 51% on the Artificial Analysis Cyber Index, which "measures a model's ability to discover vulnerabilities in a codebase, reproduce and validate them, and patch them without breaking existing functionality". It reports that the model "handles these legitimate defensive security tasks without unnecessary refusals", and an average refusal rate of 96.6% on malicious cyber requests, "the highest we measure among open models", taken over the cyber-related prompts of JailbreakBench, StrongREJECT and AgentHarm. The preview was at 95.3. On the B3 agent-security benchmark it reports 97.1% attack resistance, up from 93.3%. The partner quotes (CrowdStrike, Cloudflare, Rubrik, Zscaler, Elastic, ENISA, Orange Cyberdefense and others) are about the approach: open weights, auditability, hosting under your own control. The only result in them is Booz Allen's, that the model "was able to match vulnerability detection recall rates against other frontier cyber models" on an internal benchmark, with no number given.

The card says "state of the art among open-weight models for security work". The chart's data says something narrower:

By Mistral's own numbers Large 4 is fifth of seven on the Cyber Index, 6 points behind the top. (I am assuming the models on the chart are open-weight; Mistral does not label them.) The refusal chart uses a different comparison set (DeepSeek-V4-Pro at 50.7, Gemma-4-31B 84.1, GLM-5.3 87.1, Kimi K3 93.6), and neither chart puts capability and refusal on one axis for the same models. So "strong at defence and refuses attacks" is two separate measurements on two sets of competitors. It is a defensible position: capability within the leading group plus the highest refusal rate Mistral measured. It is not what "state-of-the-art" on its own would suggest, and the combination has not been shown to be best for any common set.

Mistral also writes plainly about a limit. After the preview it saw "the model refusing to comply with black-box attacks against live targets" in its traffic, yet "a potential for uplift to conduct real attacks", and says it is changing its guardrails. I read "guardrails" as including moderation around Mistral's own API, which a downloaded checkpoint does not come with (reasoned; the post does not say). The 96.6% is a property of the weights, and what happens to it after fine-tuning, with Mistral Forge or anything else, is the owner's responsibility. A vetted "specialized variant" for penetration testing and red-teaming is offered only to partners under contract and is not part of this release (reported).

The licence

The repository's LICENSE is the "Mistral Ecosystem Open-Weight License", "a modified version of the MIT license" (measured, read in full). Anyone may use, copy, modify, further train and sell copies of the model. Two conditions matter. First, notices must be kept. Second, if you or an affiliate "operates a Model as a Service business" you need a separate commercial licence from Mistral before using the model or derivatives "for any commercial purpose". Model as a Service means giving a third party access for inference or post-training, by API or hosted endpoint, in a way that lets them control inputs, parameters or training data. Excluded: end-user products with the model embedded in specific features or harnesses, mere relaying of requests to models hosted by others, internal use, and use through Mistral's own products. In practice you can run it internally, build a product on it and fine-tune it; you cannot resell a hosted endpoint without asking. Large 3 was Apache 2.0, so this is tighter. I am not a lawyer, and the line between "embedded in a harness" and "hosted endpoint" is the one to take advice on.

What I did not run

I did not run the model. At 1.06 TB FP8 or 610 GB NVFP4 it does not fit on this machine, so no output, speed or benchmark here is mine. I did not download any tensor; every size comes from headers and the Hub's file listing. I did not read the vLLM or transformers modelling code for this architecture, so my description of the indexer's behaviour is inferred from tensor names and config. The bandwidth numbers use nominal vendor figures from memory and a uniform-routing assumption that real routers violate. I did not reproduce any benchmark; the chart values are Mistral's.

How I checked

• Post and thread: fetched through the fxtwitter mirror; the thread is two posts, the second holding the links.
• Model card, config.json, params.json, generation_config.json, LICENSE, chat_template.jinja, tokenizer.json: downloaded and read in full.
• Headers: for each of the 54 consolidated-*.safetensors shards, a range request for bytes 0-7 (the header length) and one for the header itself, 6.16 MB in all. I summed bytes per group (matches the index total_size exactly), parameters per group, per-layer tensor presence (63 layers with wkv_a_with_mqa, 44 with indexer.wk), and the shapes quoted above.
• NVFP4 size: usedStorage of the NVFP4 repository from the Hub API against my prediction from the FP8 file, above.
• Tokenizer and template: tokenizers 0.23.3 and jinja2 3.1.6, CPU, seconds. Token counts are from the real vocabulary; the template was rendered with bos_token <s> and eos_token </s>.
• Charts: parsed the JSON chart specs embedded in the blog page, not the pictures. The Cognition and Artificial Analysis definitions are from their pages as I read them on 1 November; the Artificial Analysis page text came through a summarising fetch tool, so check the numbers there before quoting them.