Reactive optimizers: an agent that rewrites updater.py while the model trains

An optimizer is usually decided before training starts: AdamW or Muon, a peak rate, a warmup, a cooldown, executed to the last step. If the loss spikes mid-run, the plan does not know.

Reactive optimizers, from Xingjian Bai, Jia-Hao Liang, Jayden Teoh and Vincent Sitzmann, puts a coding agent in that chair. The agent watches the run through a program it writes itself, analysis.py, and steers it through a second program it writes itself, updater.py, which turns gradients into weight updates at every step. Their package is called OPAL, for OPtimizer with an Agent in the Loop. The thread announcing it claims that "in every language and image setting we tried that has no tuned recipe, it matches or beats the best recipe one can one-shot, and it guards against loss spikes and overfitting by itself."

I read the blog post, the authors' nine-post thread and its replies, and the package (xingjian-bai/opal, MIT, commit 2e83eb0, version 0.3.7). I did not make a single hosted model call. Instead I ran OPAL's own machinery on a tiny CPU task with a hand-written Python function standing in for the agent, which is enough to test the plumbing and nothing more. Numbers below are labelled measured (I ran it or read it from a file), reported (the authors' figure, not re-run) or reasoned (my arithmetic on the other two). My runs were on a shared 4-core cloud VM, CPU only, two threads.

The loop

expand

The user-facing change is one line. You build your optimizer and scheduler as usual and wrap them:

opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
sched = torch.optim.lr_scheduler.CosineAnnealingLR(opt, T_max=steps)
opt = opal.wrap(opt, model, scheduler=sched, total_steps=steps,
                goal="ResNet-18 on CIFAR-10; judged by test accuracy after the last update",
                max_cost_usd=20)
for step in range(steps):
    loss = criterion(model(x), y)
    loss.backward()
    opt.step(loss=loss)   # the loss is passed so the agent can see it
    opt.zero_grad()
    sched.step()

From there, three clocks run at different speeds. This is what opal/wrapper.py, opal/schedule.py and the agent's instructions in opal/prompts/neutral.md say, all measured from the source:

1. Every update, the current updater.py gets the gradients and changes the parameters in place. The starting updater.py is generated: it calls your own optimizer and scheduler, unchanged. Until the agent edits it, the run is your run.
2. Every 50 updates (every), OPAL reads the training losses, looks for a non-finite one, and runs analysis.py on rank 0 with read-only access to the parameters, the gradients, the running updater and timing. Whatever it returns as {"report": …} is kept for the agent. If it returns {"wake": reason}, it asks for an unscheduled decision. The agent may make its analysis run more often by setting EVERY.
3. At decisions, the agent is shown a message and may edit both programs, or keep them. The default is 50 decisions per run, on the "front" schedule: a fifth of them spread over the first 5% of training, where the authors saw most code changes (in their SiT run, 49 of 125 decisions before update 50K changed a file, 6 of 310 after), and the rest spread over the remainder. A decision also fires on a non-finite loss.

The agent is told plainly what is fixed. The model, its initialisation, the data and their order, the loss, the batch size, the number of updates, the training loop and the evaluation all belong to the user. The optimizer is everything else: which rule updates which tensor, every hyperparameter per tensor, preconditioners, clipping, skipping updates, schedules. There is also a precision contract. Optimizer state must stay in the parameter's dtype or in float32, and a program that changes a parameter's dtype or device is reverted as if it had raised.

What the agent actually sees

I captured every message my stand-in agent was sent. The system prompt for my five-tensor MLP was 3,340 words (measured): the protocol, the editable scope, the interfaces, and a table of every parameter with its shape and group. Each decision message was 392 to 437 words (measured). It states when and why the decision was asked, whether the previous edit took effect, the last 50 per-update training losses, the latest analysis report, any traceback, the time per update next to the starting program's, and how many scheduled decisions are left. The workspace also holds read-only copies of the training script and model code, unless the user passes share_code=False.

The starting analysis.py reports, per parameter role, gradient and weight norms, the update-to-weight ratio and the gradient-momentum cosine, plus the slope and spread of the loss. The agents rewrote it to watch what they cared about: a SiT agent traced 17 of the 18 gradient spikes it logged to the timestep embedder (reported). Answering a question under the thread, Bai said the agents "use the average of cosine similarity of adjacent gradients as a metric of instability."

Safety rails, and where they stop

When the agent ends its turn, the wrapper diffs the workspace against the running program. A changed updater.py is built with previous=<old updater>, so momentum buffers can be carried over. Then it runs on trial for 20 updates. If it raises at its very first update, that update is undone exactly (parameters, gradients and optimizer state) and redone with the old program, provided a copy fits in snapshot_limit_gb, 2 GB by default. A failure later in the window falls back to the old program without undoing the failing update. Every version is kept under programs/.

That is the whole safety net, and the README is honest about its edge: the agent "cannot restart or roll back training, it can only fix its programs at the next decision." A bad but non-crashing learning rate is not caught by anything except the agent's next look.

The other edge is that the agent's code runs inside your training process. The rules say not to touch the data or the evaluation, but that is a rule, not a sandbox. When someone asked under the thread whether an agent could cheat by reaching for the data, Bai answered that "technically it could reach the data and do exactly what you describe, and we currently don't have a hard guardrail for that", adding that he had read every program version in the published runs and none touched the data loader or the test set (reported). With validation=N, OPAL also runs eval_integrity, a check that flags a program which moves the weights somewhere else for a dev evaluation and back afterwards. The agent is not told that check exists.

What I ran

OPAL makes an offline run easy: a backend can be "keep" (every decision keeps the programs) or any Python function that edits the workspace and returns a rationale. I installed it with CPU PyTorch 2.14.1 and ran three things.

The no-agent mode is bit-identical. The repo's MNIST example on its synthetic data printed the same parameter checksum, 9487201e1860, plain and wrapped with backend="keep". That run is only 40 updates and gets no decisions, so I wrote my own: a 32-256-256-1 MLP regressing a fixed random teacher network, AdamW at a deliberately timid 1e-4 with a cosine schedule, 3,000 updates of batch 128. Plain and keep with 8 decisions both ended at checksum 50c449e1669b and test MSE 0.00189 (measured). Eight trips through the decision machinery changed nothing.

A broken program is undone exactly. My stand-in agent wrote an updater.py whose step raises at its first decision (update 100), then a working one at its second (update 150): the user's AdamW, but with its own cosine schedule from a peak of 3e-3. The event log shows the broken version reverted with restored: true, and the next message to the agent carries the traceback and the line "the update was undone exactly and redone". To check "exactly", I ran the same script again with a first decision that keeps the programs. Both runs ended at checksum baef2ff6a6e7 (measured). Bit for bit, the failed program left no trace.

The plumbing works; the intelligence is mine. The scripted run ended at test MSE 0.00033 against 0.00189 for the timid recipe (measured). That says nothing about agents. I picked a bad starting rate on purpose, and my function raises it at a fixed decision. It only shows that an edit made mid-run takes over cleanly, with the optimizer state carried across.

Four of the repo's CPU test files passed, 24 tests (measured). The widget below is a TypeScript port of schedule.py and estimate.py; against the Python on 120 combinations of run length, decision count and schedule it gave zero mismatches (measured). Times per update on the shared VM swung between 22 and 58 ms across identical runs, so I report no overhead figure of my own.

What "matches the best recipe" means

The central claim is about settings where nobody has tuned a recipe. The authors build them from places where somebody has.

modded-nanogpt, Track 3. A 162M GPT on FineWeb, with dozens of leaderboard recipes tuned for exactly that setting. The authors derive four settings without a tuned recipe: the same model at a quarter of the batch (13,000 updates of 131,072 tokens), and the same model on 1 GB Goldfish sets of Chinese, Thai and Kyrgyz (3,000 updates). The baselines are five leaderboard records, adjusted the natural way: #46 and #30 (elaborate Muon recipes), #36 (the tuned Muon baseline), #21 (Shampoo) and #2 (AdamW). OPAL starts from #36. All of this is reported, from the blog's Figure 1.

I pulled the bar values out of the data embedded in that figure. The picture is consistent across the four settings, and it has two halves.

Against its own starting recipe, #36, OPAL gains a lot: 0.0323 in validation loss at a quarter batch, 0.0888 in Chinese, 0.0906 in Thai and 0.0834 in Kyrgyz (reasoned from reported values). That is the agent repairing a recipe that does not transfer.

Against the best recipe one could have picked, which is #46 in all four, the margins are 0.0029, 0.0035, 0.0035 and 0.0166 (reasoned). The OPAL README says the track assumes a run-to-run standard deviation of 0.0013 at the official setting. If that noise carries over to these settings, which is my assumption and not the authors', three of the four margins are between 2 and 3 standard deviations of a single run, and Kyrgyz is about 13. The Chinese and Thai OPAL values are means of three runs. The baselines appear to be single runs. "Matches" is the right word for three of them, and "beats" holds up for Kyrgyz.

On the official setting, where #46 is the tuned recipe, OPAL from #36 reached the 3.28 target at update 3,200, the same as #36 itself; #46 needed 2,725 (reported). The blog says the gains shrink "when a run is short and its recipe is fine-tuned", and this is that case. The README adds that one earlier run on OPAL 0.2.0 ended worse than #36, at 3.2878 against 3.2781.

Goldfish, Amharic. A 125M GPT trained from scratch on 1 GB of Amharic for 10 epochs (64,626 updates), scored in bits per sentence on FLORES-200. The official recipe ends at 83.48 and OPAL from it at 81.11. This is the most informative result in the post, because the authors also ran a sweep: nine more static recipes, the best of which (AdamW at 2e-3) ends at 80.89, ahead of OPAL. OPAL started from that swept recipe ends at 80.73 (all reported). So a learning-rate sweep beat a single reactive run from the default, and a reactive run on top of the sweep beat both. The agent is not a substitute for a sweep. It also works on top of one.

Image generation. LightningDiT-XL/1 on ImageNet 256 for 64 epochs: FID-50K 1.83 against 2.18 with guidance, 4.33 against 5.15 without. RAEv2-B, whose official recipe already uses a Muon variant: 3.19 against 3.39 (reported). The baseline here is the official recipe, retrained by the authors, not a sweep, which the post justifies by noting that diffusion papers rarely sweep optimizer settings.

expand

Seeds are the soft spot. The post says it outright: "Most experiments in this post are run once, a few three times." A commenter under the thread proposed the ablation that would settle the most interesting question: replay one run's sequence of edits on a new seed, and compare with an agent reacting afresh. That would separate "the agent found a better recipe" from "the agent responded to this particular run". Bai replied "Good point!" It has not been run.

Spikes and overfitting

Two controlled experiments test the "guards against loss spikes and overfitting" half of the claim, both on ResNet-18 and CIFAR-10.

In the spike test, the authors insert four stretches of random images and labels into the data. OPAL is not told. The first two stretches cost it 50 and 66 points of test accuracy, about what the static optimizers lose. In between, it added loss damping, replaced it with clipping, then wrote a gate that holds back batches whose loss sits far above recent clean ones. The third stretch then cost 12 points and the fourth none, while the clipped baselines and Stable-SPAM lost 28 to 76 points each (reported). This is the reactive part working as advertised: it learned the shape of the incident from two examples.

In the overfitting test, 10% of the training labels are wrong, and a held-out dev split is visible to the agent through opt.report_dev. The test set is not.

expand

AdamW run straight through ends at test loss 2.49 and 73.5% accuracy, and reproduces the wrong label for all 4,500 mislabeled training images. OPAL ends at 0.69 and 78.6%, giving the wrong label for 4% (reported). The fair comparison is not the straight-through run, though, because OPAL is using the dev set to pick its final weights. Early stopping on the dev set gives 72.9%. A moving average of AdamW's weights chosen on the dev set gives 77.5%, and that strategy was itself "adapted from OPAL" (reported). So the agent's edge over a careful human with the same dev set is 1.1 points of accuracy, from one run. What it does show is that it found the weight-averaging strategy on its own, around epoch 24, from the dev loss climbing.

What it costs

There are two costs: money for the agent, and wall-clock time while training waits for it.

opal/estimate.py carries the authors' own measurements, 567 decisions of Opus 5.5 at effort high (reported). The first decision reads the training code and writes the first program: median $1.08, range $0.33 to $1.62, and 251 s. Each later decision costs a median of $0.23 ($0.058 to $0.377) and takes 27 s. For the default 50 decisions, the typical estimate is $1.08 plus 49 times $0.23, or $12.35, with a range of $2.75 to $20.22 (reasoned, and matching opal.estimate.cost, measured). That agrees with Bai's reply under the thread: "$5 to $15 per run."

The published runs land across that range. LightningDiT-XL/1 cost $6.42 for about 100 decisions, about 6.4 cents each. RAEv2-B cost $13.30 for 50, about 27 cents each, and $9.71 for 43 when training did not wait (reported values, reasoned rates). Most later decisions are short "kept both programs" turns. The expensive ones are where the agent writes code.

Against the compute, the money is small. The LightningDiT run used about 74 H100-hours, which the authors price at about $510; the agent adds $6.42, about 1.3% (reasoned). The authors concede that counted in FLOPs it is impractical. Their bet is that for most researchers dollars and FLOPs are different budgets.

Time is the cost that bites. With training paused at each decision (the default), the published runs took 1.30x to 4.39x the static run's wall clock. With blocking=False, training continues on the old program and the edit lands later, a median of 65 updates late on Goldfish, and the overhead falls to 1.01x to 1.14x (reported). In the RAEv2-B pair, the non-waiting run scored 3.17 against 3.19 for the waiting one, a single run each. The README is blunt about short runs: its one-GPU example scripts took 27 to 31 minutes with the agent against 1 to 3 minutes without.

Where it sits

Three older ideas are close, and the differences are the point.

• Learned optimizers such as VeLO meta-train a network to output updates over thousands of tasks, then freeze it. OPAL trains nothing and keeps nothing between runs.
• Population-based training changes hyperparameters mid-run by running a population and copying the winners. OPAL runs one trajectory and replaces the population with reasoning: cheaper in GPUs, with no selection pressure to correct a bad call. The A-B-A tests the RAEv2 agent wrote, alternating 1.5x and 2x learning rates over a few hundred updates and comparing loss slopes, are a one-run imitation of what PBT does with many.
• Automated discovery, from Lion's symbolic search to Prime Intellect's experiment where frontier agents had days to beat a NanoGPT speedrun recipe, searches across runs for a better static optimizer. OPAL moves the search inside one run, so it does not leave you a recipe for the next.

The edits read like a careful engineer's. Agents moved hidden matrices from AdamW to Muon, and moved some back after no gain. One kept a single embedding's Adam moments in float32 after measuring entries about 15 in size, where adjacent bfloat16 values are 0.0625 apart. Others started planned cooldowns early when progress stalled. None of it is beyond a human expert; the point is that it happens at every decision of every run.

Two other pieces on this site sit at the edges. Dream-RSI is a loop where an agent rewrites the scheduler of a search, not of training, and the thread itself points at Dust as the kind of non-gradient optimizer a reactive agent could one day steer.

Where it breaks

• No hard sandbox. The agent's code runs in your process, with your data in reach.
• No rollback of training. The trial window catches exceptions, not bad ideas.
• Single runs. Three of four modded-nanogpt margins sit within a few standard deviations of single-run noise, and the replay ablation has not been run.
• Scale. FSDP, DeepSpeed, multi-node and optimizer compilation are not supported. Asked under the thread about losing fused kernels to an agent-written update rule, Bai said he did not know yet how to handle it.
• Reproducibility. The same seed can produce different choices, because the agent can. OPAL hashes its default prompt and starting programs, so at least the harness is pinned.

Verdict

The mechanism is simple and well built: two programs, a check every 50 updates, front-loaded decisions, a 20-update trial with an exact undo, and a no-agent mode I verified bit for bit on CPU.

The claims need their qualifiers. "Matches or beats the best recipe" means: large gains over the recipe it starts from, and margins over the best off-the-shelf recipe that are small in three of four language settings and clear in one. The Goldfish sweep is the honest result to remember: a sweep beat one reactive run, and a reactive run on top of the sweep beat both. For about $5 to $15 and the patience to let training wait, that is a reasonable thing to bolt onto a run of several hours on a recipe nobody has tuned. On a recipe somebody has tuned, expect a tie.