BTL-4: reading a model card against its own weights

BTL-4 went up on Hugging Face on 2026-08-05: a 35B agentic reasoning model from Bad Theory Labs, Apache-2.0, 21 safetensors shards, and a benchmark table with 78.4% on SWE-bench Verified in it. There is no technical report. There is no arXiv paper. There is no third-party evaluation. At the time of writing the repository has 38 likes and, according to the Hugging Face API, zero downloads — all-time. Nobody has run this model.

That combination is common enough now that "how do I read this?" is a real question rather than a rhetorical one. The answer I want to argue for is that a model card is not the only evidence a release ships. The artifact itself — config.json, the tensor index, the bytes in the shards — is evidence too, it is machine-checkable, and it is often more informative than the prose. This piece is that check, run end to end on BTL-4. Some of the card holds up exactly. Some of it does not.

What checks out

Start with the parts that survive, because they are the majority and because a check that only ever finds problems isn't a check.

The base-model claim is exactly right. The card declares base_model: Ornith-1.0-35B, which is an unqualified string rather than a resolvable repository id, so nothing on Hugging Face verifies it for you. But ornith-ai/Ornith-1.0-35B exists — a real MIT-licensed model with 2.67M downloads — and its config.json matches BTL-4's on every architectural field:

The tensor maps are identical too: 31,666 tensors, same names, in both. Whatever else is true, BTL-4 is a derivative of Ornith-1.0-35B and not of something else wearing its name.

The weights are real and complete. 21 shards, 70.21 GB, 35.11B parameters in bf16, a coherent model.safetensors.index.json, vision tower included. This is not an empty repository with a good README.

The lineage is worth stating, because it puts BTL-4 next to work already covered here. Ornith-1.0-35B's architecture is qwen3_5_moe — the same family as Intern-S2-Mobius, which is 40 layers at hidden 2048 with the same 512-wide experts and the same every-fourth-layer full attention. Both descend from Qwen's Qwen3.5-35B-A3B, whose parameter count (35,951,822,704) is also the exact figure I measured for Macaron-V1-Tall's base checkpoint. Three unrelated labs, one 35B Qwen substrate. That is worth noticing on its own.

And one section of the card is genuinely useful, which I'll come back to at the end — it isn't the benchmark table.

The benchmark table doesn't close

Here is the LiveCodeBench v6 section of the card, quoted in full. Aggregate 66.1%, and:

followed by: "The set is 45% hard problems, which is what pulls the aggregate down."

Those four numbers cannot all be true, and you do not need the benchmark to see it. An aggregate pass rate is a weighted mean of the per-difficulty rates. Fix the hard share and the aggregate is pinned inside an interval — lowest when every remaining problem is medium, highest when every remaining problem is easy.

At 45% hard, the aggregate has to land between 74.9% and 81.7%. The card reports 66.1%, roughly nine points below the floor of what its own difficulty breakdown allows. Going the other way: to produce a 66.1% aggregate from a 60.5% hard bucket and an 86.7% medium bucket, you would need 78.6% hard problems and zero easy ones — which would leave the 99.1% easy row reporting a score for an empty set.

I want to be careful about what this does and does not establish. It does not tell you the model is bad, and it does not tell you which number is wrong. Any one of four edits reconciles it: the aggregate, the hard share, one of the bucket rates, or an unstated detail about how the aggregate was computed (a different problem set, a different pass@k, a subset that the difficulty table doesn't describe). What it does establish is that the table was never checked against itself, which is a fact about the release process rather than about the model. Numbers that were run, recorded, and then arithmetically verified do not do this.

The same section has a smaller tension worth flagging. The card says the runs used "full splits, no subsetting," and in the next breath specifies "442 problems, 2024-08 → 2025-05." A date window is LiveCodeBench's intended usage — the whole point of the benchmark is contamination-controlled time slices — so the window is legitimate. But a date-windowed 442-problem slice is, definitionally, a subset, and "no subsetting" is the wrong way to describe it. I could not independently confirm LiveCodeBench v6's true composition for that window, so I can't say which of the card's figures the real distribution would support.

What the config says that the card doesn't

config.json is written by the training code, not by the person writing the README, which makes it the more candid of the two documents. BTL-4's contains this:

{
  "model_name": "/vol/merged/btl4-pilot",
  "transformers_version": "5.13.1",
  "unsloth_version": "2026.7.6"
}

Three things leak out of five lines. The checkpoint was produced with Unsloth, a LoRA and QLoRA fine-tuning library. It was loaded from a directory called merged, which is what you call the output of folding an adapter back into its base. And the run was named btl4-pilot.

None of that is damning — LoRA is a completely normal way to fine-tune a 35B MoE, and merging is the normal way to ship one. But the card's training section says only: "Fine-tuned from Ornith-1.0-35B on an execution-gated reasoning corpus." A reader deciding whether a +4.3-point BFCL gain is likely to generalize would want to know it came from a merged adapter rather than a full fine-tune, and the card does not say.

The interesting question is whether the weights agree with the config. They do.

Reading 70 GB without downloading it

A safetensors file opens with 8 bytes giving a header length, followed by that many bytes of JSON describing every tensor: dtype, shape, and byte offsets into the rest of the file. That means two small HTTP range requests per shard buy you the complete layout of a 70 GB checkpoint. Once you have offsets, you can range-request one specific tensor out of the middle of a shard and compare it against the same tensor in another repository, having transferred a few hundred kilobytes.

I ran that against BTL-4 and Ornith-1.0-35B across ten groups of tensors.

The first pass was wrong, and the way it was wrong is the most useful thing in this article. Every normalization weight came back CHANGED — all forty layers' input and post-attention norms, the final norm, even norms inside the vision tower. That looked like a substantial finding. It was an artifact: BTL-4 stores norms as F32 where Ornith stores them as BF16, so I was comparing 8,192 bytes of one format against 4,096 bytes of another and reading the inevitable mismatch as training.

The check that settles it is arithmetic on file sizes. The two checkpoints differ in total size by 603,136 bytes. BTL-4's metadata reports exactly 301,568 parameters stored in F32; Ornith reports none. An F32 parameter costs two bytes more than a BF16 one, and 301,568 × 2 = 603,136. The entire size difference between the two models is the norm upcast and nothing else — which is what turns "I should exclude those rows" from a hunch into a fact.

What the change map means

With dtype-mismatched tensors excluded, the pattern is unusually clean:

• Changed, in every layer sampled: expert gate_proj and down_proj, the shared expert's up_proj, the linear-attention in_proj_qkv, and full-attention q_proj.
• Unchanged, in every window sampled: the MoE routers, token embeddings, the output head, the entire 27-layer vision tower, and the linear-attention A_log and dt_bias.

That is a LoRA target set, drawn from life. Adapters go on the projection matrices; routers, embeddings, output heads and frozen encoders are left alone. Combined with unsloth_version and /vol/merged/, the artifact is telling a consistent story that the prose omits.

Two of those frozen tensors deserve their own note.

A_log and dt_bias are untouched at every layer, and unlike the big matrices these are small enough to compare in full — 64 bytes each, byte-for-byte identical. In this architecture family A_log is the per-head base rate of the linear-attention decay gate, the parameter whose exponential sets how fast a channel forgets. KDA has a half-life works through what that number means: it converts directly into a memory horizon measured in tokens. So BTL-4's forgetting timescales are Ornith's, unmodified. Whatever the fine-tune taught the model about tool calling, it did not touch the mechanism that decides how long the model can hold something.

The vision tower is entirely unchanged, and entirely still there. BTL-4 ships processor_config.json, an image_token_id, a video_token_id, and 27 untouched vision layers. The card sets pipeline_tag: text-generation, describes a text-only training corpus, and reports no multimodal evaluation whatsoever. Nothing wrong with that — you inherit a capability you didn't train and don't claim. But a reader should know that roughly a tenth of what they'd be downloading is an unexercised, unevaluated image encoder, and that the model's multimodal behaviour is entirely Ornith's.

The number with the least behind it

Of the three headline benchmarks, note which one is documented and which is not.

BFCL v4 gets a full protocol sentence: official ast_checker, all 1240 cases, run in-house, and — best practice, this — an explicitly paired comparison against the base with "identical harness, identical decoding, only the weights differ." That is exactly how a fine-tuning claim should be stated, and the +4.3 points is the only number on the card that isolates what the training actually bought. LiveCodeBench gets a protocol sentence too, though the numbers under it don't close.

SWE-bench Verified 78.4% gets three words: "official harness." No base comparison, so there is no way to see what the fine-tune contributed. No statement of who ran it, while the two benchmarks above it are explicitly labelled in-house. No scaffold named, which for SWE-bench is most of the result — the agent loop around the model routinely moves that score more than the model does, a point the harness effect makes at length. No trajectory logs, no leaderboard submission.

It is also, by a distance, the biggest claim on the page. 78.4% would place a 35B model with roughly 3B active parameters within striking distance of the frontier systems this site has covered — the Macaron-V1 table has Claude Opus 4.8 at 88.6% on the same benchmark. Extraordinary is the wrong frame; unverifiable is the right one. The claim isn't refuted here. It's simply the one number with the least behind it, presented with the least detail, on a model that has been downloaded zero times.

The part of the card that's actually good

None of the above touches the most useful section, and it deserves to be lifted out because it is the kind of thing most cards leave you to discover in production:

Reasoning accumulates across agent turns. The chat template strips prior reasoning from older turns, but this only works if your harness separates it into reasoning_content. With vLLM, that means --reasoning-parser qwen3. Without it, thinking lands in content, accumulates every turn, and long agent runs degrade.

That is correct, specific, non-obvious, and expensive to learn by yourself. A reasoning model whose chat template prunes old thinking blocks depends on the serving layer routing them to the right field; get it wrong and you don't see an error, you see an agent that quietly gets worse over a long session while your context bill climbs. Whoever wrote that paragraph has actually run this thing in a loop. The same section is candid that the model is verbose, not a chat model, and token-hungry, and the generation-settings note — LiveCodeBench moving 60.9% → 66.1% purely by raising the output budget from 16K to 32K — is a real, useful observation about evaluating reasoning models even if the endpoint number is the one that doesn't reconcile.

There's a smaller inconsistency in the same neighbourhood: the card claims 262K native context and the vllm serve command it gives sets --max-model-len 131072. Both are defensible individually — you often can't fit the full window on the hardware you have — but nothing explains the gap.

The take

The reusable part here isn't the verdict on BTL-4, it's the sequence. Four checks, none of which require downloading a model or running a benchmark, in increasing order of effort:

1. Does the table close? Weighted means have to be consistent with their parts. This caught the LiveCodeBench contradiction in one line of arithmetic, before anything was fetched.
2. Does the declared base exist, and does the config match it? Field-by-field comparison confirmed BTL-4's lineage exactly, which is the strongest positive result in this whole piece.
3. What does the config leak? Training code writes provenance the README never mentions — library versions, working-directory paths, run names.
4. Do the weights agree with the story? Safetensors headers plus range requests turn "trust the training section" into a measurement, for a few hundred kilobytes of traffic.

Applied to BTL-4 the result is mixed rather than damning: a real fine-tune, of the model it says it is, with complete weights, shipped with one benchmark row that contradicts itself, one number that carries the most weight and the least evidence, and a training method the artifact discloses more honestly than the prose does. What I'd want before believing the headline is a base-model row for SWE-bench and the scaffold used to get it — the same paired-comparison discipline the card already applies to BFCL, extended to the number people will actually quote.

Sources: the BTL-4 model card (README, config.json, model.safetensors.index.json, safetensors headers), the Ornith-1.0-35B repository, and the Hugging Face models API for download, like, and parameter counts, all as of 2026-08-06. The tensor comparison was performed with HTTP range requests against both repositories' shards; the method, its dtype correction, and the limits of what a windowed comparison can prove are described in the second tab of the diff figure above. The LiveCodeBench arithmetic uses only figures printed on the card. No model was downloaded and no benchmark was re-run. Both interactives are mine; the repository ships no figures, so there are none to embed.