Carbon-X: teaching Qwen3-14B to read DNA in 6-base words

On 22 October Leandro von Werra posted the headline: "you can mid-train yourself to the frontier! At a fraction of the cost! Without distillation!" The model is Carbon-X, from Hugging Face's bio team with Zhongguancun Academy and TIGEM / University of Naples. You paste DNA into a chat message and it answers in English: benign or pathogenic, promoter or enhancer, which organism group, what the next bases are.

The post said it started from "Qwen3.5 14B". A reply pointed out that no such model exists, and he corrected it to Qwen3-14B, which is what config.json and the card's base_model say.

What shipped on 21-22 October: the weights (Apache-2.0), a GGUF repository, a release blog on a Space, and a model card that lists the training recipe as hyperparameters. What did not ship yet is the training code. More on that at the end.

I could not run a 29.6 GB model on a shared 4-core VM. What I could run is the part that makes this model different: the tokenizer. I also read the weights' safetensors headers and the GGUF metadata by HTTP range request, and rechecked every average and every piece of arithmetic in the card. Every number below is labelled measured (I computed it from a file or a run), reported (Hugging Face's figure, not re-run) or reasoned (my arithmetic on the other two).

One decoder, two alphabets

There are two existing ways to get a model that talks about DNA. Genomic foundation models such as Hugging Face's own Carbon-3B read raw sequence and score it, but they cannot explain a prediction. Chat models know biology from papers but have barely seen a genome. The usual bridge, used by BioReason, is a separate DNA encoder whose embeddings are projected into a language model.

Carbon-X skips the encoder. DNA is just more tokens in the same 40-layer decoder:

expand

The weights confirm the shape. I read both safetensors headers (measured): 443 tensors, all bfloat16, 14,822,046,720 parameters, 40 layers, hidden 5,120, FFN 17,408, q_proj 5,120 x 5,120 and k_proj/v_proj 1,024 x 5,120. That is 40 query heads and 8 KV heads of dimension 128, with q_norm and k_norm present. The embedding and LM head are separate tensors (untied), each 157,184 x 5,120. The architecture string is plain Qwen3ForCausalLM; nothing about the model body is custom.

At bfloat16 that is 29.6 GB of weights (measured, the two shard sizes sum to 29,644,145,152 bytes). The KV cache costs 40 layers x 2 x 1,024 x 2 bytes = 160 KiB per token (reasoned), which agrees with the GGUF card's "about 0.16 MB of KV cache per token". A full 40,960-token context is about 6.3 GiB per sequence on top of the weights. "Fits on a single 80 GB GPU" holds with room to spare.

The tokenizer, from the code

Everything new lives in tokenizer.py, a HybridDNATokenizer that wraps Qwen's BPE. I ran it on CPU with transformers 5.20.2 (the run exception for this article; peak RSS 210 MB). It is short enough to explain fully.

The vocabulary. The code builds every 6-letter string over A, C, G, T: $4^6 = 4096$ k-mers. It adds one tail token for every prefix of length 1-5, padded with N (CT becomes CTNNNN): $4 + 16 + 64 + 256 + 1024 = 1364$. Three markers, <dna>, </dna> and <oov>, make 5,463. The total is then padded to a multiple of 128 with 52 <unused_i> tokens. So the card's "5,515 DNA tokens" is 5,463 real ones plus 52 padding (measured: those are the counts the loaded tokenizer reports). The BPE part is Qwen's 151,669 ids, so the vocabulary is 157,184, matching the embedding tensor.

Where the new rows came from. Qwen3-14B's own config.json has vocab_size 151,936: Qwen pads its 151,669 real tokens with 267 spare rows. Carbon-X starts its DNA ids at 151,669, so the first 267 DNA tokens land on Qwen's spare rows and 5,248 rows are appended. That is $2 \times 5248 \times 5120 = 53{,}739{,}520$ new parameters across the embedding and LM head (reasoned from the two configs and the measured shapes), about 0.36% of the model. The card says all DNA embeddings start random and are trained for about 10B tokens with the body frozen before anything else moves.

Compression. I tokenized the same DNA both ways (measured):

Carbon-X counts include the two tags. The blog's "roughly a third as many tokens as Qwen3's tokenizer" holds on non-repetitive sequence. The one case BPE wins is a homopolymer, where it has merged long A-runs into single tokens. The 6-mer count is fixed by length: $\lceil n/6 \rceil + 2$, whatever the content. That predictability is the point. An 8,192-token training sequence holds at most 49,152 bases (reasoned; the card says "≈ 49 kbp"), and tokenizing 49,152 random bases gave 8,194 tokens in 13 ms (measured).

Auto-tagging. You do not have to write <dna> yourself. Any run of six or more A/C/G/T letters, either case, is wrapped; runs of twelve or more with up to 20% IUPAC ambiguity codes are wrapped too; space-separated codon blocks are joined first. I tried edge cases (measured). ACG TAC GAT TCA GGA CTT is joined into one sequence. Sequence ACGTA stays BPE, five letters being too short. And GATTACA is a 1997 film, in either case, comes out as <dna> GATTAC ANNNNN </dna> is a 1997 film. The detector has no notion of a word boundary. For a DNA chat model that is the right default, but pass auto_dna_tags=False if you feed it general text.

N costs six bases. Only canonical bases form k-mers. ACGTNACGTACGT becomes <oov> CGTACG TNNNNN: the one unknown base takes its whole 6-mer with it (measured). The card says the same; filter or mask before you tokenize.

The frame is fixed, and indels move it

The tokenizer cuts from the first base of each tagged run, every six bases, with no overlap. That has a consequence the card does not spell out. A substitution changes exactly one token. An insertion or deletion of anything other than a multiple of six shifts the frame. Every 6-mer downstream then becomes a different token id. In a 600-base random sequence I made each edit at base 301 and diffed the ids (measured). A substitution changed 1 of 102 tokens, and so did an N. A one-base insertion changed 52, a one-base deletion 50, and a three-base insertion 52. A six-base insertion added exactly one new token and left every downstream id intact, one slot later.

Try it on the card's own example. The reference is the 55-base HBB fragment from the usage snippet, and the variant there is A to T at base 20:

The model can still learn that CAGGAG shifted by one base is related to GCAGGA; 302B tokens of training is a lot of shifted DNA. But the two strings share nothing at the token level, so it has to learn that relation; it gets no help from the input. This is the obvious place to look when you notice that the one benchmark where Carbon-X trails the frontier by a wide margin is the non-SNV one: insertions, deletions and other non-substitution changes, 85.1 against Opus 5.5's 94.8 (reported). I cannot attribute the gap to tokenization from the outside. That would take an ablation Hugging Face has not published. It is a hypothesis, reasoned, not a finding.

The recipe

The card describes four stages. None of the code to run them is public yet; the numbers below are the card's (reported) unless marked.

1. Embedding warm-up. About 10B tokens with the transformer frozen, so only the new DNA rows learn.
2. Mid-training. 302B tokens at sequence length 8,192, global batch 512, 72,000 steps, cosine learning rate from 5e-5 to 5e-6 after 2,000 warmup steps. The arithmetic closes: $512 \times 8192 = 4{,}194{,}304$ tokens a step, times 72,000 steps is 301.99B (reasoned).
3. SFT. Three epochs over 20 dataset subsets, cosine 1e-5 to 1e-6, batch 32, examples up to 9,984 tokens, loss on assistant tokens only. About 30% is variant-effect reasoning, about 10% procedurally generated DNA skills (counting, searching, editing).
4. RL. Asynchronous GRPO in TRL with verifiable rewards, thinking off, in two rounds: 98,433 prompts for 150 steps, then 58,249 prompts re-selected from the first round's rollouts. 16 samples per prompt, completions up to 4,096 tokens, learning rate 1e-6, asymmetric clipping 0.2 / 0.28, gradient norm capped at 0.2.

The mid-training mixture is 55% raw DNA, 20% English, 15% biology text and 10% instruction data:

The blog says eukaryote-infecting viral genomes were excluded from the DNA. The English slice is the same kind of filtered web data I looked at in Ultra-FineWeb: what an education filter costs you, measured.

The learning-rate trap

The useful part of the blog is the account of what went wrong first. On Qwen3-4B, the learning rate that taught DNA fastest destroyed English, and the one that kept English barely taught DNA. The sweep's own plotted data (I read the arrays out of the blog's figure; reported): at 2e-4 the English score fell 22.6 points after about 10B English tokens while the DNA score rose from 0.33 to 0.45. At 2e-5 English fell only 2.3 points and DNA reached 0.36.

They also tried better embedding initialization, separate learning rates for embeddings and body, freezing layers whose DNA and English gradients conflict, merging DNA and English specialists, and more English replay. The blog says none of them recovered both. What worked was boring. A lower rate, a model 3.5x larger, a run long enough for slow DNA progress to add up, and cosine decay instead of warmup-stable-decay, which kept the rate high long enough to worsen forgetting.

expand

From the same figure's data (reported): HellaSwag ended at 0.792 against the starting model's 0.790. MMLU-Pro biology ended at 0.725 against 0.721. OpenBookQA ended at 0.858 against 0.882, a 2.4-point loss and the one real regression. This is the same trade I measured for a language adaptation in Jais 3: adapting K2 Horizon to Arabic without touching the tokenizer; Carbon-X has it harder, because it had to change the tokenizer.

Why mid-train at all

Could RL alone have done it? The blog ran the test. GRPO on FrontierVEP straight from Qwen3-14B never rose above the 50% chance line. From the mid-trained checkpoint, the same RL kept climbing:

expand

The base run's training reward does climb, to about 0.53, while its evaluations stay flat. That is the classic signature of RL finding format and prior-label shortcuts rather than reading the sequence. Mid-training is what gives RL something to sharpen. If you have read Scaling agentic RL: 365,000 environments behind one contract, this is the same lesson from the other side: the environment and the verifier matter, but RL cannot create a capability the policy never had.

"Without distillation"

A reply under the post put it well: "'without distillation' is doing a lot of heavy lifting — the supervision setup is the claim." Here is what the setup is. The blog says frontier models refused or hallucinated too often to be teachers, and self-distilling from their own RL rollouts carried unsupported claims into SFT. So they built deterministic verifiers with domain experts that check each claim in a training answer against the input sequence, and assembled SFT answers that pass. RL rewards are code-checked too, with a penalty on variant-effect answers whose rationale is not grounded.

That is a real distinction, and it is the right one. It is also narrower than the slogan. The mid-training mixture includes 1% Gemini-written descriptions of RefSeq sequences and 10% Nemotron post-training data, which NVIDIA built largely from other models' outputs (reasoned from the card's table). No teacher model wrote Carbon-X's DNA answers. Other models' text is still in the diet.

What the benchmarks show

Five DNA benchmarks, Carbon-X with thinking off and one sample per problem; each comparison model at whichever reasoning effort gave its best average:

The same table as text (reported; I recomputed every row's average from its five scores and all seven match, reasoned):

The headline figure plots average score against cost per problem:

expand

Four things to hold next to that chart.

The two benchmarks Carbon-X leads by the most are the two Hugging Face built. FrontierVEP and NextBase are in-house. On 23 October the HuggingFaceBio organization lists nine datasets, none of them FrontierVEP or NextBase (measured, from the Hub API). So nobody outside can rerun them yet. The data was decontaminated against all five with DNA overlap, text n-gram and homology checks (reported).

NextBase mostly measures refusals for the closed models. The blog says Opus 5.5 scored 17.7% on NextBase at low effort and 1.1% at max, with refusal rates rising with effort. Gemini 4 Argon fell from 58% to 38% on FrontierVEP as effort rose. Opus's 1.1 alone pulls its average about 15 points below the mean of its other four. Drop NextBase and the four-benchmark averages are GPT-6.1 Sol 76.2, Opus 5.5 75.9, Carbon-X 73.5 (reasoned). Carbon-X is third, and the gap is the non-SNV row.

66.7 against 67.5 is a tie. Carbon-X ran one sampled answer per problem at temperature 0.6. The GGUF card estimates the standard error of a single run at about 1.5 points on FrontierVEP and about 1 on NextBase (reported). A 0.8 gap in a five-way average is well inside that noise. "Close to GPT-6.1 Sol" is the honest phrasing, and it is the card's.

Where it is weak, the card says so. GenomeQA's six tasks average 63.6 (reasoned, from the per-task table: 70.0, 29.8, 93.2, 34.6, 64.0 and 90.0). Splice sites at 29.8 and histone marks at 34.6 are weak.

Two more reported results. IFEval prompt-level strict accuracy is 79.3%, with no number for Qwen3-14B beside it, so the size of any regression is not given. And used as a sub-agent that Kimi K3 was required to consult, Carbon-X raised Kimi's five-benchmark index from 49.4% to 60.5%, on 100 questions per benchmark with Python available in both arms.

The cost claim

The blog projects one pass over ClinVar's 2.4 million variants of uncertain significance. It extrapolates from each model's cost per completed FrontierVEP problem, refusals included. Its figure gives the exact numbers (reported): Carbon-X $271, GPT-6.1 Sol $32,380 at low effort and $658,632 at max, Opus 5.5 $110,724 at low and $4,027,185 at max. The card rounds these to "about $300" against "$32,000 to $4 million".

The comparison is self-hosted against list price. Carbon-X is priced at $5 per H200-hour from measured throughput, so $271 is about 54 H200-hours, or roughly 44,000 variants an hour (reasoned). The closed models are priced at standard API rates, and at max effort they think for thousands of tokens while Carbon-X answers directly. That is a real advantage of a direct-answer specialist, and it is also why the ratio moves so much with effort: 120x against GPT-6.1 Sol low, 2,430x against it at max (reasoned). The blog's opening says "1,000x lower cost"; the card says "about 1/1,800 of the inference cost per problem". Both are defensible readings of the same chart.

What the cost line leaves out is training. No GPU-hours are disclosed, and a reply asking for them went unanswered. A floor from the standard $6ND$ estimate: $6 \times 14.8\text{B} \times 302\text{B} \approx 2.7 \times 10^{22}$ FLOPs for mid-training alone. At an assumed 40% utilization of an H200's roughly 989 dense BF16 TFLOP/s, that is about 19,000 H200-hours, or about $94,000 at the same $5 rate (reasoned, with my assumptions, before SFT, RL and every ablation). "A fraction of the cost" of a frontier pretraining run, certainly. Not free.

Running it

The card says Carbon-X needs Transformers v5 with trust_remote_code=True for the tokenizer, or vLLM on one GPU. It also says "llama.cpp and other GGUF runtimes are not supported". That sentence is stale. The blog says the opposite, and a Carbon-X-GGUF repository published the same day ships BF16, Q8_0, Q6_K, Q5_K_M and Q4_K_M files. It needs llama.cpp b11742 or newer, which added the tokenizer.

I read the Q4_K_M file's metadata with a 24 MB range request (measured). tokenizer.ggml.model is hybriddna, tokenizer.ggml.dna_auto_tags is true, and there are 157,184 tokens. Each DNA k-mer is stored with a private-use character (U+E000) appended, so AAAAAA the DNA token cannot collide with a BPE string of the same spelling. The Python code flags that collision for CCCCCC, which exists in both vocabularies. The architecture block repeats the safetensors shapes exactly. The GGUF card reports the llama.cpp tokenizer matches Python's on 1,000 benchmark prompts (1.13M tokens) and 36 edge cases, and that Q8_0 through Q5_K_M score within about a standard error of the original on FrontierVEP and NextBase. Q4_K_M is a 9.6 GB file that fits a 16 GB consumer GPU, at 67.8 on FrontierVEP against 70.4 (reported).

I did not run the model itself: even Q4_K_M is over this article's 4 GB download cap.

The card's agent advice is worth following: pass sequences by name, because language models miscopy long DNA, and leave counting and coordinates to Python. The bundled carbon-dna skill also ranks variants by summed log-likelihood from vLLM's prompt_logprobs.

What is open

• Weights: yes, Apache-2.0, safetensors and GGUF.
• Recipe: as hyperparameters and data shares in the card, enough to plan a reproduction, not to run one.
• Training code: not yet. The card's "Code" link points to huggingface/carbon, whose main is commit 10bbc4b from 6 June 2026 (measured, shallow clone). It holds Carbon-3B-era evaluation and fine-tuning scripts and a tech report; no text file in it mentions Carbon-X or Qwen. In the replies von Werra says the team is "preparing the full release. It will be fully open and reproducible."
• Benchmarks: FrontierVEP and NextBase are not published.
• Pretraining DNA: the Carbon pretraining corpus from the Carbon-3B release is on the Hub. The 302B-token mid-training mix is not, as one artifact.

Carbon-X has not been clinically validated; the card and the blog both say so. Its other stated limits are miscounted bases and coordinates in long sequences, invented identities when context is missing, degraded long generations, and no evaluation of inputs much longer than training saw. The blog's opening line says you can "generate new DNA". The card says it "was not trained to design new ones". Believe the card.

The take

The idea is simple, and I think it is right. Give the decoder a vocabulary that fits DNA, then spend 302B tokens at a learning rate low enough to keep English. That puts a 14B open model within noise of the best closed model on these five benchmarks, cheaply enough to run all of ClinVar's uncertain variants. The tokenizer is the whole architectural change, and it checks out: a third of BPE's tokens, exact round trips, and a fixed frame that turns indels into a different sequence downstream.

What I would want before calling it frontier: FrontierVEP and NextBase published, more than one sample per problem, and the training code the post promised. Until then the 66.7 is Hugging Face's number on Hugging Face's terms, and it is a credible one.