# On-policy self-training: a model writes its own problems, and the answer goes in the teacher's context
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/on-policy-self-training
> date: 2026-10-27
> tags: self-training, self-distillation, synthetic-data, post-training, llm, reinforcement-learning
Most post-training recipes still start from somebody else's work. SFT needs reference solutions. RLVR needs gold answers. Distillation needs a stronger teacher. Even the recent self-distillation methods need a curated set of problems.
[On-Policy Self-Training](https://arxiv.org/abs/2610.32770) (OPST, Sick et al., Google and Ulm University, 23 October 2026) removes the problems too. You give the model a list of topic names. It writes questions, writes answers to them, filters its own output, and trains on what is left. On Gemma 4 E2B without thinking it reports MMLU going from 37.5 to 47.7 (reported).
I read the paper, the appendix and the prompt templates. I checked the arXiv abstract page, the Hugging Face paper page and the [project page](https://leonsick.github.io/opst/) on 27 October 2026. The project page lists code as "Coming Soon", so nothing here is a rerun of their results. What I could do was rebuild the loop at toy scale and measure the claims that are checkable in a toy. The toy does not reproduce the headline gain, and I will show why. It does reproduce the mechanism the paper leans on hardest. Throughout, every claim is labelled **measured** (I ran it), **reported** (the paper says it) or **reasoned** (my inference).
## The ladder this sits on
The paper frames post-training along two axes: whose trajectories you train on, and how dense the signal is (reported).
- **SFT** imitates curated demonstrations. Dense signal, but on trajectories the model never visits at inference.
- **RLVR** trains on the model's own rollouts, but compresses supervision to one scalar per rollout.
- **On-policy distillation** samples from the student and has a stronger teacher score every token.
- **On-policy self-distillation (OPSD)** removes the stronger teacher. Student and teacher are the same weights. The teacher also reads some *privileged information* the student does not see, usually the reference solution.
Each step relocates the external ingredient rather than deleting it. OPSD still needs a problem set with solutions. OPST's claim is that the privileged information can be self-generated and the problem set can go away.
## What the loop does
Five steps, repeated for N rounds. Student weights $\theta$ and teacher weights $\phi$ both start at the base model $\theta_0$.
**1. Generate.** For a topic $\tau$ such as "Abstract Algebra" or "rate, time, and distance word problems", a generator writes a query $x$ plus privileged information $z=(r,a)$: a rationale $r$ and a candidate answer $a$. The generator is either the EMA teacher or a frozen copy of the base model. Queries can be multiple choice or open ended (reported).
**2. Pseudo-verify.** Two checks, both by the generator. It solves $x$ in a fresh context, without $z$, at low temperature, and must reproduce $a$. Then it paraphrases $x$ into $x'$, solves that, and must reproduce $a$ again. The point is to separate authoring from solving: a flawed question can still get a confident answer in the pass that wrote it.
**3. Consensus gate.** The student samples $G$ rollouts on each verified sample. If all of them reproduce the candidate answer, the sample is discarded, because a question the student already answers reliably gives a teacher conditioned on $z$ almost nothing to add. What survives sits at the student's frontier, and since the gate uses the *current* student, the frontier moves as training proceeds (reported).
**4. Distil on-policy.** For each surviving $(x,z)$ the student samples its own response $y$. At every token the teacher reads $(x, z, y_{
The paper's data volumes are small: its ablation peaks at 285 generated samples per round and 7 rounds (reported, Table 6), so roughly two thousand problems in total. The main text I read does not give $G$, the learning rate, the batch size or the hardware, which is one more reason a rerun is not possible from the page alone.
## The gate is a cliff
The consensus gate is the only part of the loop with a closed-form shape, so it is worth pushing on. Suppose the $G$ rollouts are independent draws that each reproduce the candidate with probability $p$. Then the gate discards a sample with probability $p^G$, and a sample is kept with probability $v\,(1-p^G)$ where $v$ is the verification pass rate.
This is arithmetic, so it is **reasoned**. Two things follow. First, for $G=8$ a question the student solves 90% of the time is discarded 43% of the time, but one it solves 99% of the time is discarded 92% of the time. The gate is soft in the middle and a wall near certainty. Second, the real gate is harsher than the independent model, because rollouts of one model on one question are correlated: a student that is right once is likely to be right again. In the paper's ablation the gate adds +2.3 MMLU points on top of verification's +2.7 (reported, Table 5a). I take that as evidence the gate selects something, not as a measure of how much of the corpus it removes. The paper does not say.
## Why context and not target
The design choice that carries the paper is where the generated answer goes. In SFT it is the target. In GRPO it is the reward. In OPST it is the teacher's *context*, and the target is whatever the teacher then says.
The appendix gives three reasons this should matter for noisy synthetic data (reported):
1. The teacher can supervise steps the rationale never wrote. Tokens that move the rollout toward $a$ are upweighted, so the teacher fills in omitted reasoning.
2. The target lives in the model's own distribution, so the student does not copy the phrasing of the generation.
3. The model's prior limits a bad hint. A wrong $z$ shifts the teacher's distribution only as far as the model finds it plausible.
The third one is the testable claim, and the paper tests it. It disables both filters, replaces a fraction $p$ of the candidate answers with wrong ones, and compares OPST with SFT on MMLU (E2B, base 37.5):
| wrong answers | SFT | OPST |
|---|---|---|
| 0% | 38.8 | 42.7 |
| 25% | 38.1 | 42.1 |
| 50% | 35.4 | 41.4 |
| 75% | 33.1 | 40.8 |
(reported, Table 5c.) From 0% to 75% SFT drops 5.7 points and ends 4.4 below the base model. OPST drops 1.9 and stays 3.3 above it.
The other ablation on the same page isolates the hint itself (reported, Table 5b). No privileged information: 40.6. A placebo hint taken from a different problem on the same topic: 40.2. The answer alone: 44.6. The rationale alone: 45.8. Both: 47.7. So the gain tracks sample-specific information, and the rationale carries more of it than the answer. The roughly three points that remain with no hint at all are, the authors suggest, self-distillation toward the EMA teacher.
## What the paper reports
All of this is **reported**. I did not rerun any of it.
On Gemma 4 E2B without thinking, trained on self-generated data only, average over MMLU, GPQA and HellaSwag (Table 1):
| method | signal | average | gain |
|---|---|---|---|
| base | | 38.3 | |
| SFT on the generated answer | dense target | 40.0 | +1.7 |
| GRPO against the generated answer | sparse reward | 39.3 | +1.0 |
| OPST | dense context | 45.6 | +7.3 |
The same ordering holds on E4B (46.1 to 50.9, 47.3, 52.2) and with thinking enabled (E2B 58.4 to 60.3, 60.1, 63.0; E4B 71.3 to 72.2, 71.3, 76.0). On E2B with thinking the per-benchmark gains are uneven: MMLU +1.2, GPQA +6.5, GSM8K +8.2.
Against methods that use human data (Table 2, E2B without thinking): supervised OPSD on curated questions and solutions gets +8.3 on average. OPST gets +7.3, 88% of it, with no human question or answer. The two methods that remove answers but keep the human questions do worse: u-OPSD +6.3, and R-Zero, which does generate its own problems but trains with RL, +3.5. Both were re-implemented by the authors (marked ‡ in the paper).
Other results:
- **Transfer.** Seeding only MMLU and GSM8K topics improves MMLU by 11.3, more than MMLU topics alone (+10.2). Seeding all three in-domain benchmarks leaves MATH-500 4.9 points lower, so retention is mixed (Table 3).
- **Another family.** On Qwen3-4B-Base, a pretrained model with no instruction tuning, the same recipe lifts the four-benchmark average from 66.7 to 72.7, with GSM8K +13.3 (Table 8).
- **Vision.** With unlabeled OpenImages images and two kinds of generated privileged information, a scene caption and detected spatial relations, Gemma 4 E2B goes from 49.1 to 56.0 on VQAv2 (Table 4).
- **Hyperparameters.** No EMA ($\beta=1$): 38.1, essentially the base model. $\beta=0.05$: 47.7. $\beta=0.01$: 45.2. Rounds: 43.1 at three, 47.7 at seven, 46.2 at twelve and twenty. The EMA teacher beats the student weights by 1.8 points (Table 6).
- **Variance.** Three seeds on E2B, standard deviation at most 1.2 points (Table 7).
## How I read those numbers
Three observations, all **reasoned**.
*The topics are the benchmarks' own subject lists.* "Data-free" means no questions and no answers. It does not mean no knowledge of the test. The paper says topics are data-domain category labels where available, and the experiments seed from the domains of the four evaluation benchmarks. Self-training on "college_mathematics" and then scoring MMLU is not the same claim as improving on a task nobody told the model about. Table 3 helps: MMLU rose 5.1 points with no MMLU topics at all. But the paper's own limitations section says the topic descriptors are hand-chosen.
*Self-consistency is not correctness.* The authors say it directly: pseudo-verification measures self-consistency, so consistent errors still enter training, and OPST can only amplify knowledge the model already has. Both filters are agreement tests. A model that is confidently wrong in a stable way passes pseudo-verification and then, by construction, trains toward its own error.
*The comparisons are tight where it matters, thin elsewhere.* Three seeds and a 198-question GPQA Diamond set (one question is about 0.5 points, and the seed standard deviation is 1.0) mean that a GPQA gain of one point, like several of the baselines', is noise. OPST's +4.0 to +6.5 there is clear of it, and so are the large MMLU and GSM8K gains. The scale is two small Gemma models and one Qwen. Whether this holds at 70B is not tested.
## A toy to test the mechanism
I cannot rerun Gemma, so I rebuilt the loop where I can see every moving part. This tests the *mechanism*, whether context-not-target survives wrong labels, whether the EMA matters, what the filters keep. It does not test the paper's results, and the numbers below are not comparable to its.
The world. A question is a point $x\in[-1,1]^2$. The true answer is one of four classes, written as two binary tokens: a coarse bit $c$, then a fine bit $f$ given $c$. The boundaries are curved, so a model has to learn them. The topic is the half-plane $x_1<0$, which stands in for "the seeded domains". The other half-plane is never used to generate a question, which stands in for out-of-domain retention.
The model. One hidden layer of 48 tanh units, hand-written backprop in numpy, Adam. Its input is $x$, a one-hot *hint* slot, the step index and the previous token. The hint slot has a "no hint" value, so the same weights are both the student (no hint) and the teacher (hint = candidate answer). Pretraining uses 20 labelled points, with the hint set to nothing 40% of the time, to the truth 45% and to a random class 15%. That teaches the net to use hints and to weigh them against what it already believes. The pretrained model scores 80.0% on fresh in-domain points (**measured**, 30 seeds, 4,000 test points per region, standard deviation 6.6 across seeds).
The loop, as in the paper:
- **Generate.** The EMA teacher draws a topic question and answers it with its own sample at temperature 1. There is no rationale. That is the toy's biggest simplification and I come back to it.
- **Verify.** The candidate must equal the greedy answer at $x$ and at $x$ plus Gaussian noise of 0.05, my stand-in for a paraphrase.
- **Gate.** The student samples $G=8$ answers (my choice, the paper does not state one). All eight equal to the candidate means discard.
- **Train.** 300 questions per round, 20 student steps of 64 on-policy rollouts at a learning rate of 0.002. Student tokens are sampled; the teacher scores the *student's* prefix, so the second token's teacher distribution depends on what the student chose first. Loss is $D_{0.8}$ above, summed over both tokens. EMA $\beta=0.05$, 7 rounds, evaluate the teacher without a hint.
The core of one step, condensed from `opst_toy.py`:
```python
toks, (S0, S1), (sp1, sp2) = two_step(student, xb, NO_HINT, rng=rng) # on-policy rollout
T0 = feats(xb, ab, 0, None); T1 = feats(xb, ab, 1, toks[:, 0]) # teacher reads hint + student's prefix
q1, q2 = softmax(teacher.logits(T0)), softmax(teacher.logits(T1))
g = student.grad(S0, jsd_grad(sp1, q1) / bs) + student.grad(S1, jsd_grad(sp2, q2) / bs)
student.adam(g, lr)
teacher.p = {k: (1 - beta) * teacher.p[k] + beta * student.p[k] for k in teacher.p}
```
I derived the gradient of $D_\alpha$ with respect to the logits by hand and checked it against central finite differences: the maximum absolute difference is 4.4e-11 (**measured**). The files are at [`code/opst_toy.py`](/articles/on-policy-self-training/code/opst_toy.py), [`code/final.py`](/articles/on-policy-self-training/code/final.py) and [`code/trust.py`](/articles/on-policy-self-training/code/trust.py). The full sweep takes a few minutes on one CPU core, and `python3 final.py` reproduces every table below. I chose the base model's strength (20 labelled points, about 80%) and 7 rounds before running any method comparison, and I did not tune anything afterwards.
### Result 1: with only its own labels, the toy does not improve
When the candidate answer is the model's own sample, in-domain accuracy does not move. OPST ends at 79.9% against a base of 80.0%. SFT on the same candidates ends at 78.9% (**measured**).
I did not expect a gain, and I think the reason is informative (**reasoned**). The candidate is a sample from the model's own no-hint policy, so the hint contains nothing the model does not already believe. The gate then keeps exactly the questions where the student is uncertain, which are the questions near the class boundaries, where the model's own answer is least reliable. Training on those can sharpen a boundary but has no reason to move it to the right place.
The paper's setting has a gap that the toy lacks. The generator writes a *rationale* along with the answer. Its ablation shows the rationale alone is worth more than the answer alone (45.8 against 44.6), and the teacher that reads it can produce a longer, better-informed rollout than the student can without it. That is a real capability difference between teacher and student. My teacher and student see the same $x$, and the hint is a single class label drawn from the student's own distribution.
What the toy does show is the drift. Out-of-domain accuracy, on the half-plane that no generated question touched, falls from 78.8% to 74.2% for OPST and to 68.2% for SFT. OPST without the EMA ($\beta=1$) falls to 70.2% in that region and also loses in-domain, to 77.4%. Removing the two filters changes little: 79.5% in-domain and 75.5% out. A placebo hint, the candidates shuffled across questions, gives 78.5% and 71.1%.
### Result 2: give the toy a reliable author and SFT wins in-domain
To inject the missing gap by assumption, I let the "generator" write the candidate answer with knowledge the solver lacks: the candidate equals the truth with probability $q$ and a uniformly random wrong class otherwise. This is the paper's authoring-is-easier-than-solving argument made into a knob (**reasoned**: nothing in my toy justifies the value of $q$).
At $q=0.9$, OPST improves in-domain from 80.0% to 82.7%. SFT improves it to 85.2%. The paired in-domain difference, OPST minus SFT, is −2.5 points with a standard error of 0.6 across 30 seeds, and it is negative at every $q$ I ran from 0.5 to 1.0 (**measured**). Out-of-domain, OPST ends at 74.9% and SFT at 71.2%. Both are below the 78.8% they started at.
So in the toy the paper's headline ordering does not appear. SFT on good labels is faster in-domain, and both methods leak out-of-domain, OPST less. My reading (**reasoned**) is that the same property that makes OPST robust makes it slower: the teacher only trusts a hint partway, and the EMA lags the student, so each round moves the weights less than a direct cross-entropy step toward a correct label. When labels are mostly right, that conservatism costs speed. In the paper the SFT target is the generated text, rationale included, which is a different object from my one-token label, so I would not push this comparison further than that.
### Result 3: the mechanism holds
Here the toy agrees with the paper, and more sharply. I took candidates that are right 100% of the time, turned both filters off as the paper does, and replaced a fraction of them with wrong ones. In-domain accuracy after 7 rounds, 30 seeds (**measured**):
| wrong answers | SFT | OPST |
|---|---|---|
| 0% | 86.6 | 84.2 |
| 25% | 77.8 | 83.1 |
| 50% | 63.2 | 80.9 |
| 75% | 42.3 | 78.0 |
From 0% to 75% wrong, SFT loses 44.3 points. OPST loses 6.2. The shape matches the paper's Table 5c, with a much bigger SFT collapse, which is expected from a model small enough to be dragged all the way by its labels. One caution on the last row. A wrong label here is the truth plus a random offset, so at 75% each of the four classes is equally likely and the labels carry no signal. OPST at 78% is therefore a model that kept what it already knew, which is the right behaviour, not one that learned anything.
The cause is visible in the teacher. For each in-domain test point I gave the pretrained teacher a hint and measured how much probability it put on the hinted answer, binned by how confident the student already was (**measured**, 30 seeds):
When the student is unsure (confidence 0.38 to 0.50) and the hint disagrees, the teacher puts 37.5% on it. When the student is nearly certain (above 0.88), the same kind of wrong hint gets 2.5%. A hint that agrees with the student is trusted 72.0% to 99.4% of the time. SFT puts 100% on the hint in every case. That asymmetry is the whole mechanism. A wrong label that contradicts a confident student barely moves an on-policy self-distillation teacher, so the training signal for it is small, and the same label is a full-strength gradient for cross-entropy.
I also checked the EMA, on the $q=0.9$ configuration (**measured**):
| $\beta$ | in-domain | out-of-domain | in-domain sd |
|---|---|---|---|
| 1 (no EMA) | 78.7 | 72.8 | 11.3 |
| 0.2 | 80.7 | 72.6 | 6.8 |
| 0.05 | 82.7 | 74.9 | 4.7 |
| 0.01 | 81.8 | 77.8 | 5.1 |
No EMA loses both mean and stability: the standard deviation across seeds more than doubles. The in-domain optimum is at 0.05, the same value the paper picked, and a slower teacher keeps more out-of-domain accuracy. The paper's Table 6a has the same shape, with 38.1, 44.2, 47.7 and 45.2 at those four rates.
## What I did not run
- **Any LLM.** The toy is a 48-unit MLP over a 2-D plane with four answer classes. It has no language, no long rollouts, no rationale, no paraphrase and no problem authoring. Its "generator" samples questions uniformly from a half-plane.
- **The paper's pipeline, hyperparameters or benchmarks.** No code is released and the page does not state $G$, the learning rate or the hardware. Every Gemma, Qwen and VQA number in this article is reported.
- **The visual variant**, beyond describing it.
- **Larger models, or topics the benchmarks did not name.** Nobody has shown these either.
The toy's single-token hint is the weakest part, so Results 1 and 2 are the least transferable. Result 3 depends only on a teacher that has learned to weigh a hint against its own belief. That is a property of the pretraining mix I chose (45% correct hints, 15% wrong ones), and a real instruction-tuned LLM acquires its version of it differently.
## What I take from it
- **Put the generated answer in the teacher's context, not in the loss.** This is the transferable idea, and it is the one I could confirm. If you must train on synthetic labels you cannot check, a method whose target is the model's own conditional distribution degrades far more slowly as the labels get worse. The cost is speed when the labels are good.
- **Use an EMA teacher.** Both the paper (38.1 at $\beta=1$) and the toy (78.7%, with more than twice the seed spread) say the same thing.
- **Keep a hold-out the loop cannot touch.** In the toy, the unseeded region fell in every configuration, including the best ones. A paper-scale loop that only measures its seeded benchmarks would not see that.
- **Treat self-consistency as a ceiling.** Pseudo-verification and the gate both ask whether the model agrees with itself. If your problem has a checkable answer, a checker beats both. The authors' own list of next steps includes tool-grounded verification and label-free stopping criteria.
The reported gains are large enough to be worth a real replication. The thing I would watch for in one is not the headline MMLU number but the out-of-domain and held-out-topic columns, because the toy's one consistent failure was a slow leak there.