ChessLFM: 408 of the 552 Elo are not in the weights

"ChessLFM: 2000 Elo in 230M params." Chess is the rare subject where a headline like that is fully checkable, because chess strength is a measured quantity with a well-understood methodology, and because the whole system shipped: a model, a 4-bit ONNX export, and a browser app you can read. So I read it. I rebuilt the action space from scratch, summed the safetensors header, ported the prompt builder out of the minified bundle, and ran the exact artifact the demo downloads with its legal-move mask taken off.

Two things come out of that. The encoding is better than the headline suggests — the model really has learned legality, and I can put a number on it. And the strength number is a wrapper number: of the 552 Elo between the first trained checkpoint and the final one, 408 come from code that never touched a weight, and 106 of those 408 live in a row of the project's own results table that the write-up never mentions.

One move, one token

The design question the project opens with is the right one: how should a language model emit a chess move? Free-form text makes the model spend capacity tracking the board, spreads a move over several tokens, and leaves nothing to stop an illegal one. ChessLFM takes DeepMind's ChessBench action space instead — every geometrically possible move becomes one vocabulary entry, like <m:e2e4> — and keeps a decoder-only model so the thing exports to ONNX and runs under WebGPU.

The claim underneath is that chess has 1,968 geometrically possible moves. That is a self-contained assertion about an 8×8 board, so I rebuilt the set from geometry alone: every queen-like and knight-like (from, to) pair, plus every promotion move for all four pieces on both back ranks. Then I diffed it against the <m:*> keys in the released token_ids.json.

It matches, move for move, with exactly one extra: the tokenizer ships 1,969 move tokens, and the extra one is <m:0000>, the UCI null move. That is not a move — it is the left-padding for the history window, which I will come back to. Every other group in the surgery lines up to the unit, and 1969 + 64 + 73 = 2106, which is the num_added that surgery_meta.json reports against a base vocabulary of 64,402 and a final vocab_size of 66,508.

What I like about this action space is what it does not need. Castling is not a special token: e8g8 is already a member of the king's move set. En passant is not a special token either; it is an ordinary diagonal pawn capture. The set is closed under the rules of chess without a single exception, which is why 1,968 entries is enough.

The prompt is the other half. This is a port of Gy() out of the shipped Space bundle — same field order, same halfmove bucketing, same padding — rendered for a real position:

Eighty tokens, always. The model never sees free text, and the position of every field is fixed, so the network can learn "the square at index 37" rather than parsing. Note <hm:0>: the halfmove clock is bucketed into fours (floor(min(hm,100)/4), 26 buckets), so the model can tell a fresh clock from a 50-move-rule scare but not 3 plies from 2. And note that the history window is eight plies, padded left with the null move — which is exactly enough to see a repetition coming and not much more.

expand

Legal moves were never the hard part

The pitch opens with "LLMs are famously bad at chess... sooner or later they play an illegal move." The system's answer is a mask: "illegal moves are impossible, because we mask them before selecting our move." The model card is more precise about who does it — "the host masks the move logits to the legal moves" — and the released code is more precise still. evaluateBoard calls t.moves({verbose:true}) on a chess.js board, looks up each legal move's index in the action space, gathers only those logits, and softmaxes over that subset. Legality never passes through the network at all.

So: how much work is that mask actually doing? Nobody published the number, and it is the exact number that separates "the encoding taught the model chess" from "the wrapper is holding the board together." I measured it.

I ran onnx/model_q4.onnx — the 4-bit export the browser downloads — under onnxruntime on CPU, two forward passes per position exactly as evaluateBoard does, and took the plain argmax over all 2,106 vocabulary entries with no mask of any kind. Positions came from 150 master games, sampled across all phases, plus a second stratum of sparse endgames.

Over 1,700 positions the unmasked argmax was a move token 1,700 times out of 1,700 — it never once tried to emit a value token or a structural token in the move slot — and it was a legal move 1,694 times. 99.65%. The mask changed the move it would have played six times in seventeen hundred. Mean probability mass on the legal set was 0.981.

That is a real result for the encoding, and I want to state it plainly because this site usually ends up on the other side of these: the "LLMs play illegal moves" problem is solved here by the board encoding and the one-token action space, not by the mask. The mask is a safety net that fires about once every 280 moves.

It is also worth looking at what the six failures are, because they are not what the framing predicts. Not one of them is a model that lost track of the position. All six are tactical:

8/8/p3r3/1k3pp1/1P6/1R4K1/6P1/8 w - - 12 55
  top move: b3b5  (Rxb5 — the rook captures the black king)
  not even pseudo-legal; 84% of the move mass sits on illegal moves
 
8/4KQk1/7p/6p1/r5P1/8/8/8 b - - 16 84
  top move: g7f7  (Kxf7 — grabs the queen, which the king on e7 defends)
  pseudo-legal, illegal; 96% of the move mass sits on illegal moves
 
2rnqb2/1b1n1N1k/p2PB1Pp/1p6/4Q3/8/PP4PP/R4RK1 b - - 0 29
  top move: h7g6  (Kxg6 — the pawn is defended by the queen on e4, down the diagonal)
  pseudo-legal, illegal; the masked pick is the model's own second choice

Two are kings walking into a defended capture; one is a rook trying to take the enemy king outright. These are misjudgements about who defends what, which is a strength failure, not a board-state failure — and the mask turns a strength failure into a legal move, which is a different and smaller service than "it stops the model losing the position." The illegal rate roughly doubles in sparse endgames (0.25% → 0.60%), which is where the training mixture needed synthetic drills in the first place, so that tracks.

Keep the two axes separate, because the rest of this piece depends on it. Legal-move rate: 99.65% unmasked, 100% masked, and the mask is nearly free. Playing strength: a different claim entirely, measured a different way, and this is where it gets interesting.

2004 against what?

An Elo number is meaningless without the pool. The announcement says "2004 Elo (95% CI 1972-2039) on a Stockfish-anchored ladder" and leaves it there — no opponent settings, no time control, no game count in the prose. The recipe table's footnote adds "the two full 720-game ladders." The actual methodology is in the demo, in a tooltip string sitting in the bundle:

// assets/index-BPxYr76G.js — the Elo badge's tooltip
"Anchored Elo from a ~720-game ladder vs node-limited Stockfish (UCI_Elo 1320-2400),
 with test-time tree search (depth 3, width 6) + repetition-aware backup.
 The demo plays that exact configuration. Engine-anchored scale, not directly
 comparable to FIDE or Lichess ratings."

That is an honest disclosure, and it is buried in a minified JavaScript bundle rather than in the post whose title is the number. Unpacking it:

• The pool is one engine, dialled down. Not a field of engines, not humans on a server. Every opponent is Stockfish with UCI_LimitStrength on and UCI_Elo set somewhere between 1320 and 2400.
• UCI_Elo is a CCRL Blitz scale, not a human one. This is checkable at the source: the Stockfish commit that set the current parameterisation (a08b8d4) describes the range as "CCRL Blitz Elo from 1320 to 3190, approximately," anchored "± 100 Elo to CCRL Blitz," fitted from games against ranked versions of the Stash engine. So the anchor itself carries ±100 Elo of stated slop, roughly three times the half-width of the 1972-2039 interval ChessLFM reports around 2004.
• The dial was calibrated at a time control the ladder did not use. That same commit pins the calibration to "a time control of 120s+1.0s." The ladder ran Stockfish node-limited, and the node budget is not published anywhere I can find. A node limit is a reasonable choice for reproducibility, but it is off-calibration, and without the number nobody can rebuild the ladder.
• There is no public rating history. Neither the post nor the model card links a bot account, and I found nothing under ChessLFM, LFM2 or mlabonne in Lichess's bot listing. The only strength evidence anywhere is the project's own harness, which is not released.

To be fair to the project: the tooltip and the model card both refuse the comparison the headline invites. "Engine-anchored scale, not directly comparable to FIDE or Lichess ratings" is exactly right, and the model card leads with "roughly 2004 Elo with a shallow depth-3 search on top, and roughly 1500 with the raw one-pass policy," which is the most useful sentence anybody wrote about this model. The headline is "2000 Elo in 230M params." A reader who stops there will come away thinking a 230M file plays like a strong club player. Both halves of that sentence are doing work they have not earned.

The one strength claim that travels well is the head-to-head: "the final model scores 49% head-to-head against the 2000-rated anchor." That is a direct, interpretable statement — it played the 2000 dial roughly even. Whether 2000 on that dial is 2000 anywhere else is a separate question with a published answer of ±100.

Where the 552 Elo came from

The last figure in the announcement is the honest one, and it is the figure the prose contradicts by omission. Six rows, each stacked on the one above:

expand

I checked the arithmetic first, because a stacked table is the easiest place for a rounding error to hide: 1452 + 302 + 106 + 67 + 61 + 16 = 2004. Exact. The table is internally consistent, and it is the only place the full accounting appears.

Now read the prose next to it. The write-up describes SFT (about 1450), then search (about +300), then a rerun of SFT over all 126M positions (+67), then HL-Gauss (+60), then GRPO (+16). Add the prose's own numbers up and you land near 1893, not 2004. Most of the gap is row 03 — "+ Wider search, repetition fix", +106 — which appears in the table, in the figure, and nowhere in the text; the rest is the prose rounding 1452 to 1450, 302 to 300 and 61 to 60. That +106 is the second largest single gain in the entire project, larger than HL-Gauss and GRPO combined, and it is a search-width increase plus a bug fix. It is also the only row written with a tilde (~1860).

While I am in the table: only two of the six rows carry a confidence interval, and the footnote says those intervals "come from the two full 720-game ladders." So four of the six configurations were not measured on a full ladder, which is a reasonable way to spend engine time and a reason not to read the intermediate deltas as precisely as they are printed.

Attributing every row to what it changed:

Two rows changed the search wrapper and bought 408 Elo. Three rows changed the network and bought 144. And the 144 is softer than it looks: rows 04 and 05 both train the value head, which the one-pass policy never consults — it exists to score leaves — and the post says GRPO "barely changes greedy play (within the confidence interval)" and contributes its +16 "whenever the policy is used as a search prior." So every Elo point earned after the first SFT checkpoint is realised through the search loop. The model card's own framing agrees from the other direction: the final network, raw, is "roughly 1500."

That is not a criticism of the work. It is a well-known result being re-derived cleanly at small scale: a policy-plus-value network is a search prior, and the search is where the strength comes out. AlphaZero is the same shape. The criticism is of the sentence "2000 Elo in 230M params," which puts the strength inside the parameters. A truer version is "1452 Elo in 230M params, and 2004 with 432 leaf evaluations per move."

expand

What the demo actually runs

Three things in the released bundle differ from the description, and one of them matters.

te(rt => ({ ...rt,
  useSearch:   Ne.device === "webgpu",
  searchDepth: Ne.device === "webgpu" ? 3 : 0
}))

The device gate. On load, the app resolves a device and then does this:

// assets/index-BPxYr76G.js — immediately after the model loads
te(rt => ({ ...rt,
  useSearch:   Ne.device === "webgpu",
  searchDepth: Ne.device === "webgpu" ? 3 : 0
}))

device is "webgpu" if navigator.gpu.requestAdapter() resolves and "wasm" otherwise. So on any browser without a WebGPU adapter, the demo turns the search off and plays the raw policy — the ~1500 configuration — while the tooltip in the same bundle says "the demo plays that exact configuration." The UI does change its mode chip to "Greedy," so an attentive reader can notice, but nothing says the Elo badge no longer applies. There is a second path to the same place: if the search throws, the handler retries with {useSearch: false, searchDepth: 0, topK: 1} and reports it to the console only.

This is the pattern I keep finding in this class of release: not a fabricated number, but a default that quietly switches off the mechanism the number depends on. If you have played ChessLFM in the browser and thought it felt weak, check whether your browser has WebGPU before concluding anything about the model.

The search is cheaper and more expensive than described. Cheaper: the code is alpha-beta, not plain minimax — negamax breaks out of its child loop when alpha >= beta and marks the node cutoff: true — so the real leaf count is below 432, helped further by a FEN-keyed value cache. More expensive: the post says the value head "scores the ~430 positions this reaches," but 432 is only the leaves. Every interior node needs two forward passes, because the move logits are only available after the value token has been appended to the prompt. Full budget at the shipped defaults is 517 nodes and 602 forward passes per move, against 2 for the raw policy.

Which raises a question the code makes unavoidable: the value head has 64 levels, and the search compares leaves by the bin's midpoint. How often can it actually tell two lines apart? I measured the depth-1 frontier — 116 positions, the policy's top 12 legal moves each, one evaluateValue pass per child, which is exactly how a leaf gets scored. Those roughly 12 candidates came back with a mean of 4.4 distinct value bins, and in 56.9% of positions the best bin was shared by two or more moves (2.9 on average; in one position all twelve tied). The deeper backup breaks some of those ties, but when it does not, the decision falls through to the policy prior, because the root candidate sort is a plain descending sort and JavaScript's Array.prototype.sort is stable. The network's two heads are more entangled than the diagram suggests.

The value decode throws away what HL-Gauss bought. HL-Gauss replaces a one-hot target with a Gaussian over neighbouring bins, and the point of it in Stop Regressing is to read the scalar back out as the expectation of the softmax. The shipped decode is bestValueBin, a hard argmax, and the win probability is the winning bin's midpoint: (k + 0.5) / 64. So inference resolves position value to 1.6 percentage points and discards the distribution the training objective was shaped to produce. It still bought +61 Elo — better-ranked argmax bins are worth having — but there may be free Elo sitting in a two-line change.

The 230M is not the 164.8M

One more thing worth checking, because the parameter count is half the headline. I range-read the safetensors header rather than trusting the label:

The checkpoint really is 230,688,512 parameters. But there is no lm_head.weight among its 132 tensors, so the embedding is tied, and model.embed_tokens.weight is [66508, 1024] — 68,104,192 parameters, 29.52% of the model, serving as both input table and output head. Of its 66,508 rows, 64,402 are the base model's English vocabulary. The prompt is built only from the 2,106 added tokens; the only outputs ever read are value and move tokens. Those 64,402 rows are unreachable in both directions.

The project agrees, in code. The ONNX export's surgery_meta.json records "vocab_len": 2106 and "pruned_from": ".../checkpoints/grpo-hlg/step-1000", and its fp32 data file is 659,025,920 bytes — 164,756,480 float32 values. The browser downloads the 4-bit version: 223,439,008 bytes. So the thing you actually play against is a 164.8M-parameter, 4-bit model with 2,106 tokens in its vocabulary, which cannot emit a single English word and is not, in any operational sense, a language model any more. It is a chess network that inherited a transformer's file format.

That is a good engineering decision — and it makes "230M params" the number for a file nobody runs. The honest version is smaller and more impressive: a 165M network, quantised to 4 bits, playing a strong club game in a browser tab.

I will also note that nothing published re-measures the quantised export. The 2004 came from the bf16 checkpoint; the demo runs q4. Quantisation to 4 bits usually costs a little strength. My legality probe ran on q4 and found 99.65%, which tells you the quantisation did not break the encoding, but says nothing about the Elo.

What is not published

For a project whose headline is a measured quantity, the measuring apparatus is the part I most wanted and the part that does not exist publicly. There is no training code, no data-preparation code, no ladder harness, no UCI_Elo schedule, no node budget, no time control, no PGN archive of the 720-game ladders, and no bot account with a rating history. The GitHub repository named in the older Chess-LLM collection was last updated in January 2024 and is a different project. The only code released for ChessLFM is the inference side: a minified browser bundle, which to its credit is complete enough to answer most of the questions above.

That means the 2004 is not independently reproducible today. It is not unreproducible in principle — the ingredients are all standard, and anyone with Stockfish and a weekend could build a comparable ladder — but they would be building a different ladder, and the two numbers would not be comparable. This is a personal project published on a Substack, not a paper, and it is more forthcoming than most papers. I am recording the gap, not demanding a tech report.

What I would take from this

The parts of ChessLFM that deserve to be copied are the encoding and the honesty of the recipe table. A fixed-length prompt, a closed one-token action space, and a value token in the same vocabulary give you a model that is 99.65% legal without a mask, exports to ONNX cleanly, and can be wrapped in a search loop without a custom head. That is a genuinely clean design and the measurements back it.

The part that should not travel is the headline. "2000 Elo in 230M params" compresses four separate things — an engine-anchored scale with ±100 of stated slop, a search wrapper worth 408 of 552 Elo, a bug fix the prose omits, and a parameter count for a file the demo does not run — into a sentence that reads as a property of the weights. The recipe table, the model card and the demo tooltip each say the truer thing. The title says the louder one, and the title is what gets quoted.

Primary sources. The announcement, maximelabonne.substack.com/p/chesslfm-2000-elo-in-230m-params; the model, mlabonne/LFM2.5-230M-Chess; the browser export, mlabonne/LFM2.5-230M-Chess-ONNX; the demo and its code, hf.co/spaces/mlabonne/ChessLFM; and, for the anchor, Stockfish commit a08b8d4. Figures are the announcement's own, reproduced here for commentary.