Contrastive Language Model (CLM): the action never sees the state, and 81.6% is a pick from four

mdjsonmcpcopy↗ai

2026-09-26 · 17 min · explainer · llm · agents · benchmarks · evaluation · architecture

On September 23, Jacky Kwok and colleagues at Stanford and NVIDIA Research posted Contrastive Language Models as a new kind of System One model: a model that takes a state and a closed set of options and returns a probability for each option, which is the category Jev opened a week earlier. The release makes three claims. CLM-8B is "on par with Jev" on computer-use, gaming and tool-calling tasks with "up to 9× lower latency". Fine-tuned as a verifier, it sets "a new SOTA" on DeepSWE (81.6%) and Terminal-Bench 2.1 (87.6%), where Jev "fails". And splitting states from actions makes training and serving cheap.

I read the repository, the Notion post (including its collapsed toggles), both model repositories and every dataset they point to. I downloaded the two 75.6 MB head checkpoints, checked that their pickles reference nothing but tensor-rebuild globals, and loaded them with PyTorch's weights-only loader. The evaluation data is 681 MB, so I read only its metadata columns, 313 KB of it, by HTTP range request. I ran no code from the release and I have no GPU, so nothing here is a rerun of the model.

What is in the checkpoint

CLM_v0.1-8B.pt is 75,557,149 bytes, uploaded on September 21 and unchanged since (the LFS hash at the first commit matches today's). It holds two heads, a scalar and a config. There is no language model in it.

"CLM-8B" means 8B of borrowed encoder and 18.9M of CLM. The "20M-parameter projection head" in the post is the two heads together, rounded.

A retriever, used as a decider

expand

A CLM is two encoders that never talk to each other. The state text goes through Qwen3-8B; the hidden state of its last token is L2-normalised and pushed through the state head. Each candidate action goes through the same frozen Qwen3-8B, separately, and through the action head. Both outputs are normalised again, and the logit for a pair is

where $E$ is the frozen encoder, $f_s$ and $f_a$ the two heads and $\theta$ the learned logit_scale. A softmax over the candidates you supplied is the answer. Training is the symmetric InfoNCE loss over a batch of $B$ matched pairs, each state pulled toward its own action and pushed from the other $B-1$, and the same the other way round:

That is CLIP's loss down to its constants: train/finetune.py initialises the scale at $\log(1/0.07)$ and clamps it at 100, as CLIP did. The released head's scale has drifted past the clamp, so the temperature in use is exactly 0.01. A side effect I can see in the files: a clamped value gets no gradient, and the DeepSWE head's logit_scale after fine-tuning is bit-identical to the base's, 4.613248825073242.

The widget draws the readout in two dimensions instead of 512. The four action vectors are fixed; move the state and only the state is re-encoded. At the released scale of 100, a cosine gap of 0.05 becomes a logit gap of 5, about 148 to 1 in probability, so answers are sharp by construction. Drop the scale to 1 and the same geometry gives a nearly flat distribution.

What that buys, and what it costs

The bold cells are the same fact. An action is embedded without the state, so its vector depends on nothing else and can be computed once. clm-serve reserves a slab of device memory for these vectors, and for recently seen states, at start-up, the way vLLM reserves a KV cache. That is the "disaggregation" in the announcement, and it is real engineering.

The price is that the only place the state and an action meet is one dot product between two 512-d vectors. A readout that reads the option in the state's context can ask whether this action makes sense here. A two-tower model can only ask whether two points are near each other. This site's relational-choice result was about options that cannot see each other. CLM's options also cannot see the state they are answering. I would expect it to do no better on that benchmark, and possibly worse; nobody has run it (Reasoned).

The typed API sits on top of this. A choice embeds each option's description. A score takes the probability-weighted mean of the level indices. A noul, the yes/no type, is a softmax over two canned sentences, true: Yes. This is true: and false: No. This is false:, each followed by your question. The wire format is TypeSafe's /v1/systemone, so one client can call either model.

The training recipe, and how much of it shipped

Only the heads train, so every run reads precomputed encoder vectors; a full pre-training run is "about an hour on a single RTX 4090 GPU" (reported). The recipe results are reported and plausible. Top-1 against 10 hard negatives is 52.1% after pre-training alone and 69.2% after a short mid-training stage. Training on hard negatives from the start peaks at 62.4%. Replay holds the hard-negative score at 68.5% where agentic-only post-training drops it to 56.2%.

The scaling-law fits in the post are power laws in compute, data, head size and encoder size, with exponents of −0.144, −0.117, −0.061 and −0.172 read off the figure. The encoder-size fit, the one the post calls the strongest, has three points: Qwen3-1.7B, 4B and 8B.

Receipt one: "up to 9× faster than Jev"

expand

"Up to 9×" is the largest of four ratios. "On par" is two ties and two losses. For BFCL, WikiRacing and Mario there is no code, no log and no statement of where either model ran, so those rows are reported and nothing more.

T-Rex ships everything: examples/t_rex/, a harness taken from laya-vs-jev, and per-seed JSON for both models. Five seeded 60-second runs each:

Three things fall out of that table.

16.5 ms is about one frame. The game ticks at 60 FPS and Arena.tick collects answers once per tick. CLM's call returns in 2.6 ms and then waits for the next frame, 16.7 ms away. The headline ratio is floored by the harness's clock, not set by the model. The cache is visible in the same logs: CLM's encoder tokens per run fall from 47,528 on the first seed to 4,515 on the fifth as the server warms, while Jev's billed input tokens stay between 605,117 and 637,935.

The model-call ratio compares a local call with an internet one. CLM ran on a local clm-serve with a Qwen3-8B encoder on one RTX 4090; Jev is TypeSafe's hosted API. The 131.9 ms includes a round trip across the internet. Neither 9× nor the 50× in the raw call times isolates the model.

Survival measures the shield as much as the model. The README says so itself: "the survival number measures the combined system and the agreement and intervention rows measure the model." CLM proposed an option whose own text said Unsafe. … Collision. 364 times; Jev never did. As the laya-vs-jev piece showed, this harness builds each player's prompt from its own latency, and a faster player sees more options marked safe. That makes 65.8% against 98.7% partly two different exams. It does not explain the 364: a faster player has fewer unsafe options to pick.

The release does contain a like-for-like speed comparison, in a collapsed toggle on the Notion page (reported): a 200-token state on one RTX 4090, five trials. With 1 to 1,024 candidates of 15 tokens, CLM goes from 36 to 44 ms. Qwen3-8B constrained decoding on the same GPU reaches 4,285 ms at 1,024, about 97× slower. Jev goes from 131 to 579 ms. At five candidates of up to 2,048 tokens, CLM stays near 36 ms, constrained decoding reaches 3,402 ms and Jev 333 ms. That same-backbone, same-GPU row is the real size of the architectural win, and it is large. It is also a warm-cache number: the post says the candidate embeddings are cached, so the first time an action is seen is not in the curve. That is fair for a game's fixed action set. It is irrelevant for a verifier, whose candidates are new every time.

Receipt two: 81.6% on DeepSWE, 87.6% on Terminal-Bench 2.1

expand

Neither number is an agent's score. Each is a selection: sample several solutions per task, let a verifier choose one, report how often the chosen one passes. From the files:

• Candidates. mini-swe-agent driving Claude Opus 5, config mini_swe_agent_claude_opus_5_max: 449 rollouts over DeepSWE's 113 tasks, four per task with three missing (measured, dataset metadata).
• Held out. 38 tasks, 151 rollouts, best of four (measured).
• Trajectory score. Each step's (state, action) cosine; the mean of the last 12; highest wins, ties split evenly (evaluation/bon_eval.py). States are embedded at 8,192 tokens, keeping the last 8,191.
• "Lightweight fine-tuning." The head starts from CLM_v0.1-8B.pt and trains on 22,576 pairs. That count is exactly the steps of the passing Opus 5 rollouts on the other 66 tasks that have any: 20,117 training plus 2,459 validation, and the checkpoint's own n_val is 2,459 (measured). The best epoch is 15, 1.1 minutes into training by the checkpoint's clock.

So the verifier is trained on the same agent, the same model and the same benchmark, on other tasks, on successes only. My reading is that it learns what a passing Opus 5 step looks like, and a trajectory whose last 12 steps look like that wins (Reasoned). Step by step it is a weak retriever: the checkpoint's validation metrics put the true next action first within its task 4.5% of the time, at a mean normalised rank of 0.251 where 0.5 is chance.

That is where 81.6% comes from. The random pick is 28/38 = 73.7%, the dashed line, and the ceiling is 34/38 = 89.5% (both measured). 21 tasks pass on every rollout and 4 fail on every rollout, so a verifier decides 13. The fine-tuned head gets 10 of them, Jev gets 6, and a random pick expects 7.0. Over the exact pass fractions of those 13 tasks, a random picker does at least as well as the head 6.2% of the time and no better than Jev 37.9% of the time. "Jev fails as a verifier" means Jev is indistinguishable from a coin here. "CLM sets SOTA" means three more tasks than a coin, on a sample where a coin does that well about one time in sixteen.

There is a second, better-powered test, and it is sitting in the public data. The DeepSWE heads repository began on September 22 as a "Duplicate from Contrastive-LM/deepswe-prm-heads-8k", holding three checkpoints whose file hashes match the three task-disjoint fold heads now at tarsur385/deepswe-prm-heads-8k, and the seed-42 head replaced them on September 23. The fold heads start from a mid-trained head rather than the released one and train on the public pool of 4,701 passing trajectories from 23 other models on the same 113 tasks. The evaluation file carries every step's score from the fold head that never saw its task. Applying the release's own rule to all 113 tasks gives 89/113 = 78.8%, against 82.5 for a random pick and 100 for the ceiling (measured). A random picker gets that far only 1.6% of the time. On the 38 held-out tasks the same heads get 30.

The signal is real on 113 tasks. It is also narrow: averaging the last 8 to 16 steps beats random by 5.5 to 7.5 tasks; the last 1 to 7 are within 2.5 of random; averaging every step does worse than random. The released head's --window 12 is fixed by the evaluation script and the fine-tuning guide forbids changing it. Neither says how 12 was chosen.

The SOTA word. DeepSWE's own leaderboard reports the share of passing rollouts across all 113 tasks, per this site's reading of it. On that measure the candidates here score 327 of 449 = 72.8%. 81.6% is a different quantity, a pick on a 38-task subset whose ceiling is 89.5%.

Terminal-Bench 2.1 ships nothing: no task list, no rollouts, no head, no code. The chart says 30 held-out tasks, Fable 5 candidates, best of five, random 84.0%, CLM 87.6%, Jev 83.1%. As fractions of 30 those are 25.2, 26.28 and 24.93 tasks, so each is an average over something the release does not specify. The CLM-over-random gap is about one task. This site's reproduction of Alibaba's Qwen3.8-Max table lists GPT-5.6 Sol (max) at 88.8 on Terminal Bench 2.1 as a single model, a provider-reported number from a different harness. "SOTA" needs a comparison class, and the release gives none.

The latency panel, 79 against 449 ms and 32 against 131 ms on an H100, does not say what one sample covers: a step, a trajectory or a task. It cannot be a whole best-of-4 pick including the encoder. The last 12 states of a long rollout sit near the 8,191-token cap, and one such prefill on an 8B model is over 100 TFLOP, more than 100 ms even at the H100's peak dense BF16 rate (Reasoned). How Jev was asked to verify a trajectory, with what question and what truncation, is not published at all.

What is missing

• The reproduce command does not run as written. The README points at Contrastive-LM/deepswe-clm-embeddings-8k, which returns 401 and is absent from the organisation's dataset listing. The same rollouts are public as tarsur385/deepswe-prm-embeddings-8k.
• No code or logs for BFCL V4, WikiRacing, Super Mario, Terminal-Bench 2.1, or for Jev as a verifier.
• The research repository, which the README says holds the scaling experiments, data pipelines and figures, is not public. Neither are the mid-training negatives or the post-training embeddings.
• No calibration measurement for a model whose entire output is a probability, with its temperature pinned at 0.01.
• Encoder lock and truncation. Every head needs Qwen3-8B last-token vectors. clm-serve embeds 2,048 tokens by default; the DeepSWE head was trained on 8,192. Serving it at the default would feed it vectors it never trained on (Reasoned); the README says to raise both limits.

The model card promises a multimodal CLM-35B in early October.

What I would take away

The architecture is not new, and the release does not need it to be. It is a two-tower retriever, CLIP's loss on frozen LLM vectors, and that is what makes it fast. One encoder pass per state, cached vectors per action, and a dot product each is structurally cheaper than any readout that re-reads the options, and the same-GPU comparison against Qwen3-8B constrained decoding (4,285 against 44 ms at 1,024 candidates) is the number to quote.

The comparisons with Jev are weaker than the headlines. The 9× is a frame clock against a hosted API. "On par" is two ties and two losses, and in the one game with logs the shield carried CLM through 364 proposals its own prompt called a collision. The verifier result is a genuine, small, window-sensitive lift from a head trained in about a minute on the same agent's successes. It is honest enough to ship with its split, its hashes and its oracle in a JSON file. It is three tasks out of 38, and the file that would let anyone rerun it is at the wrong address.