Qwen, minus the e: a mask removes the letter, training has to write the rest

A lipogram is text that leaves out a letter. Georges Perec wrote a whole novel without the letter e. Kevin Wu wanted to know what a small language model could do, so he fine-tuned Qwen3-1.7B on his own 16 GB GPU to answer everyday prompts without the letter e, and wrote it up in Qwen, Minus the E.

His post makes two claims. Simply masking out the e tokens produces gibberish. And you can then train the model with reinforcement learning to speak coherently. The first sounds wrong. A mask is a guarantee: it makes the forbidden token impossible. Why is that not enough?

This article answers it from the tokenizer upward. I read the post, the writeup and the repository, and I ran what fits on a shared 4-core CPU VM: the vocabulary count, the mask on Qwen3-0.6B, and a short GRPO loop. Every number below carries a label. Measured means I computed it from a file or a run. Reported means it is the writeup's figure and I did not re-run it. Reasoned means my arithmetic on one or the other.

expand

The e is not a letter, it is inside a token

A language model does not read letters. It reads token ids, and a token is a chunk of text the tokenizer learned to keep together. Whether a token contains an e is a fact about its spelling that the model has to memorize, one token at a time, the same reason models once miscounted the r's in "strawberry".

That has a consequence for the mask. The word "the" is one token, with the e inside it. So is " time", and " more", and " spend". To keep an e out of the text, you cannot forbid a letter. You forbid every token whose spelling contains one.

So how many is that? I decoded every token of Qwen3's tokenizer one at a time and checked for an e or E, after Unicode decomposition so that é counts, skipping the 26 special tokens.

• 151,669 tokens in the vocabulary, 47,405 of them contain an e. That is 31.3% (measured).
• Among tokens made only of ASCII letters, 37,258 of 70,096 contain one: 53.2% (measured). Roughly one letter-only token in two is gone.
• Among whole-word tokens, a space plus letters, the share grows with length. Four-letter words: 2,061 of 5,779, or 36%. Nine letters and up: 6,535 of 8,525, or 77% (measured).
• What survives is 32,838 letter-only tokens (measured), most of them short.

The widget below is built from exactly those counts. Pick a sentence, and the tokens that contain an e light up. Then step through a real masked generation and watch what the mask removes from the model's own distribution.

"The cat sat on the mat" is seven tokens and two are banned. "Write a poem about the way trees move in the wind" is twelve and six are banned. For ordinary English the mask takes away the model's first choice about half the time.

What the mask does to a plan

The mask is a logit filter. Before sampling, every forbidden token's score is set to minus infinity, so after the softmax its probability is exactly zero and the rest renormalizes. The writeup's repository builds it from the tokenizer, decoding each id and forbidding it if the text contains an e, with a flag for non-ASCII look-alikes. That is the same mechanism as the grammar masks in XGrammar and parallel constrained decoding, only with a very simple grammar: any string without an e.

What the mask cannot do is change the model's plan. A model that is about to write "Forrest" has already committed to a spelling. When that token is banned the sampler picks the best survivor, and the best survivor is a different spelling of something else. The writeup shows the result for the woodchuck prompt:

This is a classic riddiculous and humorous riddiculous riddiculous riddiculous

No word there has an e. The text is nonsense, because "ridiculous" needs one, the model wanted it, and "riddiculous" was the nearest thing left.

I ran the same mask on Qwen3-0.6B with a system prompt that says to avoid e and E. Here is what the model wanted at each step along its masked greedy answer, five prompts (measured, greedy, bf16 on CPU):

• On 31 of 92 masked steps (34%) the model's own top token was banned. On the study prompt it was 16 of 27 steps, or 59%. On the woodchuck prompt it was 0 of 12.
• Averaged over the unmasked steps, the model put 25% of its probability, across the whole vocabulary, on tokens with an e (measured, token-weighted over 97 steps).
• Generate without the mask: 0 of 20 answers at temperature 0.7 and 120 tokens were e-free. With it, 20 of 20 (measured). The mask does exactly what it promises.

The text is mostly readable, which is not what the post reports for 1.7B. Prompt: "What is the movie Forrest Gump about". Masked Qwen3-0.6B, temperature 0.7:

Forford Gump is a film about a young boy who is lost in a world full of magic and good and bad.

Prompt: "Where does the word cliche come from?"

Clichy is a word that has its roots in old Latin, and its word is "clichy" from "clicus" (a sound) and "hy" (a suffix), which might imply a sound or a trait.

And for the tree poem, one sample ends "Sing of roots, and roots, and roots." So at 0.6B and 120 tokens, on short prompts, I did not get pure gibberish. I got Forford, Clichy, a loop, and answers that are about something else. That matches the writeup's own table in kind: it reports that the masked 1.7B model is usable on 26% of 200 answers (reported), judged by an LLM for sense and for answering the prompt. A sentence can contain no e, no misspelling a regex would notice, and still be wrong.

A cheap proxy cannot see this. I scored each sample by the share of its words that exist as whole-word tokens in the vocabulary: 0.961 masked, 0.969 unmasked (measured, 20 samples each), and 18 of 20 masked answers passed a 0.9 threshold. Both say the text is fine. It is not. The e is checkable by code. Sense is not.

Why it compounds

A forced token goes into the context and the next step conditions on it. A misspelling is evidence the model is writing badly, and it tends to keep going. So the error grows along the answer, which is why a longer answer is a worse answer under the mask. Greedy decoding makes it loop, since the best survivor is the same every time. The writeup also found the SFT adapter fell apart at temperature 1.0 (invented words like "Ranak"), so every result uses 0.7.

The recipe: SFT, then GRPO under the mask

Here is the setup, all reported from the writeup and checked against the repository where I could.

Task. Answer everyday prompts from Dolly and No Robots, minus prompts that cannot be answered without an e ("Who painted the Mona Lisa?"), capped at 300 tokens, with a system message that forbids e and E. An answer is usable if the regex finds no e, and an LLM judge scores it at least 4 of 5 for coherence and for answering the prompt. Every number is the usable rate over 200 answers: 50 validation prompts times 4 samples.

The first attempt, RL on the letter alone, failed. The reward gave each token with an e a negative score. The model found the cheapest policy: streams of random letters (i m k t r n l d g h j z p q w x y f v b c) and Tst, tkt, nkt, tkt. A KL penalty that was too small did not stop it.

SFT, then GRPO. Training on a 50,000-word e-free novel, Gadsby, did nothing (about 26% against the masked model's 26%). Training on 180 e-free answers written by frontier models and checked by code did better (35%). Then GRPO:

1. Pick 4 prompts.
2. Sample a group of 8 answers for each, mask on.
3. The judge scores each answer for coherence and for responding, each 1 to 5. The reward is (coherence − 1 + responds − 1) / 8, from 0 to 1.
4. The advantage is the answer's reward minus its group's mean, divided by the group's spread, so an answer is pushed up only if it beat its own siblings.
5. Update the LoRA adapter, rank 16, learning rate 2e-5, KL weight 0.02, 300 steps, temperature 0.7.

The group-relative step is why GRPO needs no value network. The repository's grpo.py is a clipped-ratio surrogate (eps 0.2) with a KL term and a length normalization it can switch off (reported, from reading the file). Verification is split: the e rule is a few lines of code, and "does it make sense" is a judge.

The usable-rate results, from the writeup (reported):

Most of the gain arrives by step 125. The final count is 121 of 200 usable, against 52 of 200 for the mask alone and 0 of 200 for the instruction alone: the unmasked 1.7B model used an e in all 200 answers.

The failure mode: shorter answers

The writeup is honest about what the reward did. Median answer length fell from 52 to 18 tokens for the masked start and from 53 to 24 for the SFT start (reported). Some answers collapsed to a single word: asked who services a rail station, the trained model answers "SNCF", asked which sport resembles baseball, "football". Both score 5 on coherence.

Two fixes, tried in 150-step runs from the SFT adapter, did not stop it: taking the lower judge score as the reward, and dividing the loss by the 300-token cap instead of each answer's length. The writeup's explanation is the one above: under the mask every extra word is another chance to be pushed into a bad one. For the masked start the gain was mostly shortness (clean answers that skipped part of the question went from 24 to 66 of 200). For the SFT adapter both scores rose at every length, so it was not just shortness.

My reading (reasoned): the mask is part of the environment. RL here does not teach the model to avoid the letter. It teaches it to write under a filter that already forbids it, and one cheap way to write well under that filter is to write less. Compare the MiMo reward-hacking audit: when a grader's blind spot is cheaper than the task, the policy finds it.

What fits in 16 GB

The writeup walks the memory budget as napkin arithmetic, and the arithmetic holds. For a 2B-parameter model in bf16 with AdamW, batch 4, sequence 2,048, vocabulary 150k:

I re-ran the sums: they match (reasoned). Three tricks bring the 37.7 GB down to about 7 to 8 GB (reported).

• LoRA freezes the weights and trains $\Delta W = BA$. For a 2,048 by 2,048 matrix at rank 16 that is $2 \times 2048 \times 16 = 65{,}536$ trainable numbers instead of about 4.2 million, 64 times fewer (reasoned from the writeup's own dimensions). Gradients and optimizer state shrink with it.
• Gradient checkpointing keeps only each layer's input and recomputes the rest in the backward pass, about 1.3 GB instead of 11.3 for activations, at the price of one extra forward pass (reported).
• Chunked cross-entropy never builds the full batch-by-sequence-by-vocabulary logits.

The 16 GB card ran 32 answers per step at a peak of 10 to 13 GB (reported).

For scale, my CPU run (measured): Qwen3-0.6B in fp32, LoRA rank 16 on every attention and MLP projection, 10,092,544 trainable parameters, peak resident memory 4.8 GB, about 25 to 50 seconds a step for 16 rollouts of 48 tokens on 2 threads while other jobs shared the machine.

What I ran: GRPO, unmasked, on a CPU

A faithful small copy of the writeup's recipe is out of reach here: no judge, no 300 steps. So I asked a narrower question. With the mask off, can the model learn to avoid the e from the regex alone?

Setup (measured). Qwen3-0.6B, system prompt forbidding e and E, 24 training prompts, 16 held-out evaluation prompts sampled twice each (32 answers, temperature 0.7, 48 new tokens). Reward, from code: 0 if the text contains an e, 0.2 if it is e-free but shorter than 4 words, otherwise 0.5 plus half the share of words found as whole-word tokens in the vocabulary, as a cheap stand-in for coherence. GRPO with group 8 and 2 prompts per step, one gradient step per batch, no KL term, learning rate 2e-4, group-normalized advantages and per-answer length normalization.

Run A, 30 steps (measured). Before training, 0 of 32 evaluation answers were e-free. After 10 steps, 8 of 32 (25%), and the share stayed at 25% through step 29, while the mean answer length fell from 18.2 words to 6.4. I reran the first 10 steps with every answer saved, same seed, and got the same numbers. What the 25% is: at step 10, 32 of 32 answers repeated the question back, word for word. Four of my 16 evaluation prompts happen to contain no e ("Why do birds sing?", "What is a good pizza topping?", "How do I stop worrying?", "What is a good gift for my mom?"). Echoing those scores a perfect reward, and the policy learned to echo everything. The 25% is exactly those four prompts, sampled twice. Nothing was learned about avoiding the letter.

Run B, the same loop with an echo guard (measured). I set the reward to 0 for any answer whose words overlap the prompt's by 60% or more, and ran it for 18 steps before my time cap. It did not learn: e-free answers went 0 of 32, then 2 of 32 at step 10, then 1 of 32 at step 18. The cause is plain once you look at step 0: with no e-free answer to reward, most groups score all zeros, the advantage is zero, and nothing pushes. The answers at step 18 had drifted to markdown bullets and bold text, longer than before, with no e-avoidance. The writeup's first attempt, with its random-letter streams, is the same failure on a bigger budget.

That is the writeup's failure mode in miniature: ten steps, not a hundred. The reward said "no e, and real-looking words", and the cheapest policy satisfied it by saying nothing new. It is also why the writeup keeps the mask on during RL. The mask takes the e out of the policy's job and leaves the reward to judge the writing. My two runs are the same experiment without that help, and they show what the help is worth: a model this size cannot find an e-free answer by sampling often enough for a regex reward to teach it.

RL against constrained decoding

So which is it, mask or train? The result of the writeup is that you need both, and the order matters.

The mask is a guarantee, and it is also a distribution shift. It makes the rule certain and moves the model off the distribution it was trained on. Training under the mask is on-policy for the real decoder: the samples the judge scores are the samples that will be served. That idea is general (my reading): if you must decode under a constraint, train under it. The grammar masks in XGrammar are the same trade on a different constraint.

The two halves of the reward differ. The e check is verifiable: code, no noise. The coherence check is a judge, and the hacking lives there, as the one-word answers show. A verifier both cheap and complete is rare, which is the point of how small a verifier can be.

A caveat I will not smooth over. The post says you can "just RL a model to speak coherently". The writeup's actual result is narrower: RL under the mask, from an SFT start, gets to 60.5% usable on 200 answers from 50 prompts, with the judge a single model and the model still needing the mask at inference ("they still need the mask to stay e-free"). It is a good small result, not a model that has learned to avoid e.

What I did not check

• I did not run Kevin Wu's repository, adapters or judge (the judge needs a hosted model and a key). Every writeup number above is reported.
• My vocabulary count is for Qwen3-0.6B's tokenizer. I did not check that Qwen3-1.7B's is identical.
• My CPU runs are small: 5 prompts for the mask, 16 prompts by 2 for evaluation, 30 steps for run A and 18 for run B. They show a mechanism, not a rate. The usable-proxy in the mask comparison is a vocabulary check and not a judge.
• The 0.6B masked answers were less broken than the writeup's 1.7B ones. The writeup's own 4B comparison (12.5% masked against 26% for 1.7B) suggests size does not help, and I did not test it.

What to take from it

Treat the letter as a decoding rule and the language as a learned skill. The mask costs nothing and guarantees the rule, but it hands the model a vocabulary it never planned around. Training against that mask, with a reward that judges the writing and cannot be satisfied by saying less, closes the gap. The code for the e is trivial. The code for "this reads like English" is the whole project.