# Carbon-X: teaching Qwen3-14B to read DNA in 6-base words

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/carbon-x
> date: 2026-10-23
> tags: pretraining, tokenization, reinforcement-learning, grpo, benchmarks, qwen, biology

On 22 October Leandro von Werra posted the headline: "you can mid-train
yourself to the frontier! At a fraction of the cost! Without distillation!" The
model is [Carbon-X](https://huggingface.co/HuggingFaceBio/Carbon-X), from
Hugging Face's bio team with Zhongguancun Academy and TIGEM / University of
Naples. You paste DNA into a chat message and it answers in English: benign or
pathogenic, promoter or enhancer, which organism group, what the next bases
are.

The post said it started from "Qwen3.5 14B". A reply pointed out that no such
model exists, and he corrected it to Qwen3-14B, which is what `config.json` and
the card's `base_model` say.

What shipped on 21-22 October: the weights (Apache-2.0), a GGUF repository, a
release blog on a Space, and a model card that lists the training recipe as
hyperparameters. What did not ship yet is the training code. More on that at
the end.

I could not run a 29.6 GB model on a shared 4-core VM. What I could run is the
part that makes this model different: the tokenizer. I also read the weights'
safetensors headers and the GGUF metadata by HTTP range request, and rechecked
every average and every piece of arithmetic in the card. Every number below is
labelled **measured** (I computed it from a file or a run), **reported**
(Hugging Face's figure, not re-run) or **reasoned** (my arithmetic on the other
two).

<ModelCard repo="HuggingFaceBio/Carbon-X" />

## One decoder, two alphabets

There are two existing ways to get a model that talks about DNA. Genomic
foundation models such as Hugging Face's own Carbon-3B read raw sequence and
score it, but they cannot explain a prediction. Chat models know biology from
papers but have barely seen a genome. The usual bridge, used by BioReason, is a
separate DNA encoder whose embeddings are projected into a language model.

Carbon-X skips the encoder. DNA is just more tokens in the same 40-layer
decoder:

<Figure
  src="https://media.thesatyajit.com/ai/articles/carbon-x/fig2.png?v=d61ed8afcc6b2e27"
  alt="Pipeline diagram: an input mixing English and DNA goes into a hybrid tokenizer that cuts the English with Qwen3 BPE and the DNA into 6-mers between dna tags; both token types index one shared embedding matrix, split into text rows from Qwen3 and new DNA rows; the Qwen3-14B decoder layers and LM head produce an output that mixes English words and tagged DNA."
  caption="The whole architecture is a tokenizer change: text and 6-mer DNA tokens share one embedding matrix and one Qwen3-14B decoder, and the output can contain either (Carbon-X release blog, Figure 3; rendered from the blog's HTML)."
/>

The weights confirm the shape. I read both safetensors headers (**measured**):
443 tensors, all bfloat16, **14,822,046,720 parameters**, 40 layers, hidden
5,120, FFN 17,408, `q_proj` 5,120 x 5,120 and `k_proj`/`v_proj` 1,024 x 5,120.
That is 40 query heads and 8 KV heads of dimension 128, with `q_norm` and
`k_norm` present. The embedding and LM head are separate tensors (untied), each
157,184 x 5,120. The architecture string is plain `Qwen3ForCausalLM`; nothing
about the model body is custom.

At bfloat16 that is 29.6 GB of weights (**measured**, the two shard sizes sum to
29,644,145,152 bytes). The KV cache costs 40 layers x 2 x 1,024 x 2 bytes =
160 KiB per token (**reasoned**), which agrees with the GGUF card's "about
0.16 MB of KV cache per token". A full 40,960-token context is about 6.3 GiB
per sequence on top of the weights. "Fits on a single 80 GB GPU" holds with
room to spare.

## The tokenizer, from the code

Everything new lives in `tokenizer.py`, a `HybridDNATokenizer` that wraps Qwen's
BPE. I ran it on CPU with `transformers` 5.20.2 (the run exception for this
article; peak RSS 210 MB). It is short enough to explain fully.

**The vocabulary.** The code builds every 6-letter string over A, C, G, T:
$4^6 = 4096$ k-mers. It adds one *tail* token for every prefix of length 1-5,
padded with N (`CT` becomes `CTNNNN`): $4 + 16 + 64 + 256 + 1024 = 1364$. Three
markers, `<dna>`, `</dna>` and `<oov>`, make 5,463. The total is then padded to
a multiple of 128 with 52 `<unused_i>` tokens. So the card's "5,515 DNA tokens"
is 5,463 real ones plus 52 padding (**measured**: those are the counts the loaded
tokenizer reports). The BPE part is Qwen's 151,669 ids, so the vocabulary is
157,184, matching the embedding tensor.

**Where the new rows came from.** Qwen3-14B's own `config.json` has
`vocab_size` 151,936: Qwen pads its 151,669 real tokens with 267 spare rows.
Carbon-X starts its DNA ids at 151,669, so the first 267 DNA tokens land on
Qwen's spare rows and 5,248 rows are appended. That is
$2 \times 5248 \times 5120 = 53{,}739{,}520$ new parameters across the
embedding and LM head (**reasoned** from the two configs and the measured
shapes), about 0.36% of the model. The card says all DNA embeddings start
random and are trained for about 10B tokens with the body frozen before
anything else moves.

**Compression.** I tokenized the same DNA both ways (**measured**):

| Sequence | Bases | Carbon-X tokens | Qwen3 BPE tokens | Ratio |
|---|---:|---:|---:|---:|
| HBB fragment from the card | 55 | 12 | 30 | 0.40 |
| Random, uniform ACGT | 600 | 102 | 308 | 0.33 |
| Random, uniform ACGT | 6,000 | 1,002 | 3,093 | 0.32 |
| CAG repeat | 600 | 102 | 400 | 0.26 |
| Poly-A | 600 | 102 | 75 | 1.36 |

Carbon-X counts include the two tags. The blog's "roughly a third as many
tokens as Qwen3's tokenizer" holds on non-repetitive sequence. The one case BPE
wins is a homopolymer, where it has merged long A-runs into single tokens. The
6-mer count is fixed by length: $\lceil n/6 \rceil + 2$, whatever the content.
That predictability is the point. An 8,192-token training sequence holds at
most 49,152 bases (**reasoned**; the card says "≈ 49 kbp"), and tokenizing
49,152 random bases gave 8,194 tokens in 13 ms (**measured**).

**Auto-tagging.** You do not have to write `<dna>` yourself. Any run of six or
more A/C/G/T letters, either case, is wrapped; runs of twelve or more with up to
20% IUPAC ambiguity codes are wrapped too; space-separated codon blocks are
joined first. I tried edge cases (**measured**). `ACG TAC GAT TCA GGA CTT` is
joined into one sequence. `Sequence ACGTA` stays BPE, five letters being too
short. And `GATTACA is a 1997 film`, in either case, comes out as
`<dna> GATTAC ANNNNN </dna> is a 1997 film`. The detector has no notion of a
word boundary. For a DNA chat model that is the right default, but pass
`auto_dna_tags=False` if you feed it general text.

**N costs six bases.** Only canonical bases form k-mers. `ACGTNACGTACGT`
becomes `<oov> CGTACG TNNNNN`: the one unknown base takes its whole 6-mer
with it (**measured**). The card says the same; filter or mask before you
tokenize.

### The frame is fixed, and indels move it

The tokenizer cuts from the first base of each tagged run, every six bases,
with no overlap. That has a consequence the card does not spell out. A
substitution changes exactly one token. An insertion or deletion of anything
other than a multiple of six shifts the frame. Every 6-mer downstream then
becomes a different token id. In a 600-base random sequence I made each edit at
base 301 and diffed the ids (**measured**). A substitution changed 1 of 102
tokens, and so did an N. A one-base insertion changed 52, a one-base deletion
50, and a three-base insertion 52. A six-base insertion added exactly one new
token and left every downstream id intact, one slot later.

Try it on the card's own example. The reference is the 55-base HBB fragment
from the usage snippet, and the variant there is A to T at base 20:

<KmerFrame />

The model can still learn that `CAGGAG` shifted by one base is related to
`GCAGGA`; 302B tokens of training is a lot of shifted DNA. But the two strings
share nothing at the token level, so it has to learn that relation; it gets no
help from the input. This is the obvious place to look when you notice that
the one benchmark where Carbon-X trails the frontier by a wide margin is the
non-SNV one: insertions, deletions and other non-substitution changes, 85.1
against Opus 5.5's 94.8 (**reported**). I cannot attribute the gap to
tokenization from the outside. That would take an ablation Hugging Face has not
published. It is a hypothesis, **reasoned**, not a finding.

## The recipe

The card describes four stages. None of the code to run them is public yet; the
numbers below are the card's (**reported**) unless marked.

1. **Embedding warm-up.** About 10B tokens with the transformer frozen, so only
   the new DNA rows learn.
2. **Mid-training.** 302B tokens at sequence length 8,192, global batch 512,
   72,000 steps, cosine learning rate from 5e-5 to 5e-6 after 2,000 warmup steps.
   The arithmetic closes: $512 \times 8192 = 4{,}194{,}304$ tokens a step, times
   72,000 steps is 301.99B (**reasoned**).
3. **SFT.** Three epochs over 20 dataset subsets, cosine 1e-5 to 1e-6, batch 32,
   examples up to 9,984 tokens, loss on assistant tokens only. About 30% is
   variant-effect reasoning, about 10% procedurally generated DNA skills
   (counting, searching, editing).
4. **RL.** Asynchronous GRPO in TRL with verifiable rewards, thinking off, in
   two rounds: 98,433 prompts for 150 steps, then 58,249 prompts re-selected
   from the first round's rollouts. 16 samples per prompt, completions up to
   4,096 tokens, learning rate 1e-6, asymmetric clipping 0.2 / 0.28, gradient
   norm capped at 0.2.

The mid-training mixture is 55% raw DNA, 20% English, 15% biology text and 10%
instruction data:

| Component | Share | Source |
|---|---:|---|
| Eukaryotic genomes | 38.5% | GENERator corpus |
| mRNA | 8.8% | OpenGenome2 |
| Prokaryotic genomes | 5.5% | OpenGenome2 |
| mRNA with splice sites and promoters | 2.2% | OpenGenome2 |
| FineWeb-Edu / DCLM / FinePDFs-Edu | 8% / 8% / 4% | web and PDFs |
| Biology documents with long DNA | 7% | FinePDFs, PubMed, bioRxiv, FineWeb, Biostars |
| Other biology documents | 7% | same sources |
| Synthetic RefSeq descriptions | 1% | written by Gemini |
| Instruction following and chat | 10% | Nemotron post-training data |

The blog says eukaryote-infecting viral genomes were excluded from the DNA.
The English slice is the same kind of filtered web data I looked at in
[Ultra-FineWeb: what an education filter costs you, measured](/articles/ultra-fineweb).

### The learning-rate trap

The useful part of the blog is the account of what went wrong first. On
Qwen3-4B, the learning rate that taught DNA fastest destroyed English, and the
one that kept English barely taught DNA. The sweep's own plotted data (I read
the arrays out of the blog's figure; **reported**): at 2e-4 the English score
fell 22.6 points after about 10B English tokens while the DNA score rose from
0.33 to 0.45. At 2e-5 English fell only 2.3 points and DNA reached 0.36.

They also tried better embedding initialization, separate learning rates for
embeddings and body, freezing layers whose DNA and English gradients conflict,
merging DNA and English specialists, and more English replay. The blog says
none of them recovered both. What worked was boring. A lower rate, a model
3.5x larger, a run long enough for slow DNA progress to add up, and cosine
decay instead of warmup-stable-decay, which kept the rate high long enough to
worsen forgetting.

<Figure
  src="https://media.thesatyajit.com/ai/articles/carbon-x/fig3.png?v=6c6234a50f60a544"
  alt="Six small line charts over 300 billion training tokens. Top row, DNA: sequence recovery rises smoothly from about 0.33 to 0.56; eukaryote species classification climbs noisily from about 0.28 to 0.37; SeqQA rises from about 0.38 to 0.71. Bottom row, English: MMLU-Pro biology, HellaSwag and OpenBookQA wander within a few points of the dashed Qwen3-14B line."
  caption="Mid-training Qwen3-14B on 302B tokens. DNA sequence recovery climbs from 0.33 to 0.56 against a 0.295 baseline; the English panels stay within about two and a half points of the starting model, OpenBookQA the furthest below (Carbon-X release blog, Figure 6; rendered from the blog's HTML)."
/>

From the same figure's data (**reported**): HellaSwag ended at 0.792 against
the starting model's 0.790. MMLU-Pro biology ended at 0.725 against 0.721.
OpenBookQA ended at 0.858 against 0.882, a 2.4-point loss and the one real
regression. This is the same trade I measured for a language adaptation in
[Jais 3: adapting K2 Horizon to Arabic without touching the tokenizer](/articles/jais-3);
Carbon-X has it harder, because it had to change the tokenizer.

### Why mid-train at all

Could RL alone have done it? The blog ran the test. GRPO on FrontierVEP straight
from Qwen3-14B never rose above the 50% chance line. From the mid-trained
checkpoint, the same RL kept climbing:

<Figure
  src="https://media.thesatyajit.com/ai/articles/carbon-x/fig4.png?v=ced471bdd6174ae1"
  alt="Training reward over 200 RL steps for two runs. The dashed base plus RL run rises to about 0.53 but its checkpoint evaluations stay near 0.5. The solid mid-training plus RL run rises to about 0.62, with checkpoint evaluations climbing from 0.49 to about 0.6."
  caption="FrontierVEP under GRPO. Base + RL (dashed) earns training reward but its held-out checkpoint evaluations sit at chance; mid-training + RL keeps improving to about 0.6 (Carbon-X release blog, Figure 10; rendered from the blog's HTML)."
/>

The base run's *training* reward does climb, to about 0.53, while its
evaluations stay flat. That is the classic signature of RL finding format and
prior-label shortcuts rather than reading the sequence. Mid-training is what
gives RL something to sharpen. If you have read
[Scaling agentic RL: 365,000 environments behind one contract](/articles/scaling-agentic-rl),
this is the same lesson from the other side: the environment and the verifier
matter, but RL cannot create a capability the policy never had.

### "Without distillation"

A reply under the post put it well: "'without distillation' is doing a lot of
heavy lifting — the supervision setup is the claim." Here is what the setup is.
The blog says frontier models refused or hallucinated too often to be teachers,
and self-distilling from their own RL rollouts carried unsupported claims into
SFT. So they built deterministic verifiers with domain experts that check each
claim in a training answer against the input sequence, and assembled SFT
answers that pass. RL rewards are code-checked too, with a penalty on
variant-effect answers whose rationale is not grounded.

That is a real distinction, and it is the right one. It is also narrower than
the slogan. The mid-training mixture includes 1% Gemini-written descriptions of
RefSeq sequences and 10% Nemotron post-training data, which NVIDIA built
largely from other models' outputs (**reasoned** from the card's table). No
teacher model wrote Carbon-X's DNA answers. Other models' text is still in the
diet.

## What the benchmarks show

Five DNA benchmarks, Carbon-X with thinking off and one sample per problem;
each comparison model at whichever reasoning effort gave its best average:

<BenchBoard />

The same table as text (**reported**; I recomputed every row's average from
its five scores and all seven match, **reasoned**):

| Benchmark | Carbon-X | GPT-6.1 Sol max | Opus 5.5 max | Gemini 4 Argon low | Kimi K3 low | DeepSeek V4.1 Flash low | GLM-5.3 low |
|---|---:|---:|---:|---:|---:|---:|---:|
| BioReason Coding (1,018) | **74.9** | 71.0 | 70.3 | 6.5 | 53.9 | 51.4 | 52.1 |
| BioReason Non-SNV (873) | 85.1 | 91.7 | **94.8** | 90.3 | 65.3 | 55.9 | 60.1 |
| GenomeQA (2,600) | 63.6 | **67.2** | 62.2 | 63.1 | 52.0 | 49.7 | 44.6 |
| NextBase (600) | **39.4** | 32.9 | 1.1 | 31.2 | 30.6 | 27.2 | 9.7 |
| FrontierVEP (1,000) | 70.4 | 74.9 | **76.2** | 58.0 | 50.1 | 51.3 | 32.7 |
| **Average** | 66.7 | **67.5** | 60.9 | 49.8 | 50.4 | 47.1 | 39.8 |

The headline figure plots average score against cost per problem:

<Figure
  src="https://media.thesatyajit.com/ai/articles/carbon-x/fig1.png?v=af8fd526eb07b0e4"
  alt="Scatter plot, average DNA benchmark score against cost per problem relative to Carbon-X on a log axis. Carbon-X sits at 67% and 1x; an arrow runs up to it from Qwen3-14B at 40%. GPT-6.1 Sol is at 68% near 1,800x; Kimi K3 50%, DeepSeek V4.1 Flash 47% and GLM-5.3 40% sit between 200x and 1,000x."
  caption="Average of the five DNA benchmarks against cost per problem, relative to Carbon-X. Opus 5.5 and Gemini 4 Argon, both in the table, are not plotted; the y-axis starts at 35% (Carbon-X model card, score-vs-cost figure)."
/>

Four things to hold next to that chart.

**The two benchmarks Carbon-X leads by the most are the two Hugging Face
built.** FrontierVEP and NextBase are in-house. On 23 October the
HuggingFaceBio organization lists nine datasets, none of them FrontierVEP or
NextBase (**measured**, from the Hub API). So
nobody outside can rerun them yet. The data was decontaminated against all
five with DNA overlap, text n-gram and homology checks (**reported**).

**NextBase mostly measures refusals for the closed models.** The blog says
Opus 5.5 scored 17.7% on NextBase at low effort and 1.1% at max, with refusal
rates rising with effort. Gemini 4 Argon fell from 58% to 38% on FrontierVEP
as effort rose. Opus's 1.1 alone pulls its average about 15 points below the mean of its other four.
Drop NextBase and the four-benchmark averages are GPT-6.1 Sol 76.2, Opus 5.5
75.9, Carbon-X 73.5 (**reasoned**). Carbon-X is third, and the gap is the
non-SNV row.

**66.7 against 67.5 is a tie.** Carbon-X ran one sampled answer per problem at
temperature 0.6. The GGUF card estimates the standard error of a single run at
about 1.5 points on FrontierVEP and about 1 on NextBase (**reported**). A 0.8
gap in a five-way average is well inside that noise. "Close to GPT-6.1 Sol" is
the honest phrasing, and it is the card's.

**Where it is weak, the card says so.** GenomeQA's six tasks average 63.6
(**reasoned**, from the per-task table: 70.0, 29.8, 93.2, 34.6, 64.0 and 90.0).
Splice sites at 29.8 and histone marks at 34.6 are weak.

Two more reported results. IFEval prompt-level strict accuracy is 79.3%, with
no number for Qwen3-14B beside it, so the size of any regression is not given.
And used as a sub-agent that [Kimi K3](/articles/kimi-k3) was required to
consult, Carbon-X raised Kimi's five-benchmark index from 49.4% to 60.5%, on
100 questions per benchmark with Python available in both arms.

## The cost claim

The blog projects one pass over ClinVar's 2.4 million variants of uncertain
significance. It extrapolates from each model's cost per completed FrontierVEP
problem, refusals included. Its figure gives the exact numbers (**reported**):
Carbon-X \$271, GPT-6.1 Sol \$32,380 at low effort and \$658,632 at max,
Opus 5.5 \$110,724 at low and \$4,027,185 at max. The card rounds these to
"about \$300" against "\$32,000 to \$4 million".

The comparison is self-hosted against list price. Carbon-X is priced at \$5
per H200-hour from measured throughput, so \$271 is about 54 H200-hours, or
roughly 44,000 variants an hour (**reasoned**). The closed models are priced
at standard API rates, and at max effort they think for thousands of tokens
while Carbon-X answers directly. That is a real advantage of a direct-answer
specialist, and it is also why the ratio moves so much with effort: 120x
against GPT-6.1 Sol low, 2,430x against it at max (**reasoned**). The blog's
opening says "1,000x lower cost"; the card says "about 1/1,800 of the inference
cost per problem". Both are defensible readings of the same chart.

What the cost line leaves out is training. No GPU-hours are disclosed, and a
reply asking for them went unanswered. A floor from the standard $6ND$
estimate: $6 \times 14.8\text{B} \times 302\text{B} \approx 2.7 \times 10^{22}$
FLOPs for mid-training alone. At an assumed 40% utilization of an H200's
roughly 989 dense BF16 TFLOP/s, that is about 19,000 H200-hours, or about
\$94,000 at the same \$5 rate (**reasoned**, with my assumptions, before SFT,
RL and every ablation). "A fraction of the cost" of a frontier pretraining run,
certainly. Not free.

## Running it

The card says Carbon-X needs Transformers v5 with `trust_remote_code=True`
for the tokenizer, or vLLM on one GPU. It also says "llama.cpp and other GGUF
runtimes are not supported". That sentence is stale. The blog says the
opposite, and a
[Carbon-X-GGUF](https://huggingface.co/HuggingFaceBio/Carbon-X-GGUF) repository
published the same day ships BF16, Q8_0, Q6_K, Q5_K_M and Q4_K_M files. It needs
llama.cpp b11742 or newer, which added the tokenizer.

I read the Q4_K_M file's metadata with a 24 MB range request (**measured**).
`tokenizer.ggml.model` is `hybriddna`, `tokenizer.ggml.dna_auto_tags` is true,
and there are 157,184 tokens. Each DNA k-mer is stored with a private-use
character (U+E000) appended, so `AAAAAA` the DNA token cannot collide with a
BPE string of the same spelling. The Python code flags that collision for
`CCCCCC`, which exists in both vocabularies. The architecture block repeats the
safetensors shapes exactly. The GGUF card reports the llama.cpp tokenizer
matches Python's on 1,000 benchmark prompts (1.13M tokens) and 36 edge cases,
and that Q8_0 through Q5_K_M score within about a standard error of the
original on FrontierVEP and NextBase. Q4_K_M is a 9.6 GB file that fits a
16 GB consumer GPU, at 67.8 on FrontierVEP against 70.4 (**reported**).

I did not run the model itself: even Q4_K_M is over this article's 4 GB
download cap.

The card's agent advice is worth following: pass sequences by name, because
language models miscopy long DNA, and leave counting and coordinates to
Python. The bundled `carbon-dna` skill also ranks variants by summed
log-likelihood from vLLM's `prompt_logprobs`.

## What is open

- **Weights:** yes, Apache-2.0, safetensors and GGUF.
- **Recipe:** as hyperparameters and data shares in the card, enough to plan a
  reproduction, not to run one.
- **Training code:** not yet. The card's "Code" link points to
  `huggingface/carbon`, whose `main` is commit `10bbc4b` from 6 June 2026
  (**measured**, shallow clone). It holds Carbon-3B-era evaluation and
  fine-tuning scripts and a tech report; no text file in it mentions Carbon-X or Qwen. In
  the replies von Werra says the team is "preparing the full release. It will
  be fully open and reproducible."
- **Benchmarks:** FrontierVEP and NextBase are not published.
- **Pretraining DNA:** the Carbon pretraining corpus from the Carbon-3B release
  is on the Hub. The 302B-token mid-training mix is not, as one artifact.

<RepoCard repo="huggingface/carbon" />

Carbon-X has not been clinically validated; the card and the blog both say so. Its
other stated limits are miscounted bases and coordinates in long sequences,
invented identities when context is missing, degraded long generations, and
no evaluation of inputs much longer than training saw. The blog's opening
line says you can "generate new DNA". The card says it "was not trained to
design new ones". Believe the card.

## The take

The idea is simple, and I think it is right. Give the decoder a vocabulary that
fits DNA, then spend 302B tokens at a learning rate low enough to keep English.
That puts a 14B open model within noise of the best closed model on these five
benchmarks, cheaply enough to run all of ClinVar's uncertain variants. The
tokenizer is the whole architectural change, and it checks out: a third of
BPE's tokens, exact round trips, and a fixed frame that turns indels into a
different sequence downstream.

What I would want before calling it frontier: FrontierVEP and NextBase
published, more than one sample per problem, and the training code the post
promised. Until then the 66.7 is Hugging Face's number on Hugging Face's terms,
and it is a credible one.