# Frontier RL is cheaper than you think: ship deltas, not mega-clusters

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/frontier-rl-cheaper
> date: 2026-07-08
> tags: explainer, reinforcement-learning, systems, inference-optimization
Fireworks' argument in [*Frontier RL Is Cheaper Than You Think*](https://fireworks.ai/blog/frontier-rl-is-cheaper-than-you-think) is narrow and load-bearing: the belief that reinforcement-learning post-training needs one giant co-located cluster rests on a single assumption — that **every policy update ships the full ~1 TB checkpoint to the rollout fleet**. It does not. Between adjacent RL checkpoints most weights do not change at all, so you can ship a compressed **delta** — a couple of percent of the model — and keep a rollout fleet fresh over ordinary cross-region links. That is the whole claim, and everything else follows from it.

<Callout type="warn">
This is a **vendor blog**. Fireworks sells the training and rollout-serving platform this argument recommends, so read the framing accordingly. The numbers below are **Fireworks-reported** from one sample setup, not an independent benchmark. What is *not* vendor-specific is the underlying physics — weight-update sparsity in RL — which a separate paper ([arXiv 2602.03839](https://arxiv.org/abs/2602.03839)) reports independently. I have kept the two apart throughout.
</Callout>

## RL has two jobs, not one

The mega-cluster instinct comes from pretraining, where the systems problem is keeping **one** huge synchronous job saturated. RL is a different shape. An RL run has two coupled jobs:

- The **trainer** runs forward, reward computation, backward, and the optimizer step. It wants dense, tightly-coupled hardware — the pretraining kind.
- The **rollout fleet** samples trajectories from the *current* policy — that is, it runs inference on the latest weights, across many parallel requests. It wants inference throughput, and it can live anywhere.

Pretraining only has the first job. RL has both, and the awkward part is the seam between them: how do you keep a large rollout fleet generating from a *fresh enough* policy without stalling on checkpoint transfers every step? That coupling is the whole systems problem, and it is why RL cost lives somewhere non-obvious. The same two-job structure shows up whenever RL trains agentic models — the trajectory-generation half is exactly the rollout fleet here (I wrote about the trajectory side in [Agents-A1](/articles/agents-a1)).

<Figure
  src="/articles/frontier-rl-cheaper/fig1.png"
  alt="Diagram of a cross-region RL weight-update loop. A policy trainer runs forward/backward plus an optimizer step and emits a base checkpoint every N=25 steps, with changed weights of about 2% per step. A weight-update handoff ships full checkpoints plus compact deltas to three rollout regions — US Ohio (+43 ms), US Virginia (+58 ms), EU Frankfurt (+145 ms) — each receiving a 2.0% compressed delta of weights. A 50-step sample window at the bottom shows one full checkpoint resetting the chain, then deltas of about 2% in between, for more than 98% less cross-region traffic."
  caption="The loop: one trainer emits a full checkpoint every N steps and a ~2% delta in between; every rollout region reconstructs the same checkpoint from a shared delta chain over ordinary links. >98% less cross-region traffic than shipping the full model each step (Fireworks AI, Fig 1)."
/>

## The 1 TB problem

A frontier checkpoint is around 1 TB. If every policy refresh really required shipping that whole tensor to the rollout fleet, the conclusion writes itself: keep trainer and inference on the same RDMA-class fabric, avoid long-distance transfers, treat remote capacity as second class. That is the mega-cluster story, and its side effect is economic — frontier RL looks like a market only a handful of companies with a co-located supercluster can enter.

The premise is the full-checkpoint transfer. Break it and the conclusion goes with it.

## The key insight: exploiting 98% sparsity

Between nearby RL checkpoints, most weights barely move. Fireworks reports that **more than 98% of weights in `bf16` remain bit-equivalent between consecutive checkpoints**, and the unchanged fraction is higher still at lower precision. Their explanation is mechanical: RL delivers a very sparse learning signal — a few bits of reward per rollout — so training runs a small learning rate, and most parameters shift so little in `fp32` that they never cross the threshold to change their 16-bit representation. The independent paper *Understanding and Exploiting Weight Update Sparsity for Communication-Efficient Distributed RL* ([arXiv 2602.03839](https://arxiv.org/abs/2602.03839)) reports the same phenomenon, often **around 99%** sparsity in practical RL settings.

If only ~2% of the bits change, you should move ~2% of the bits. The figure's illustrative per-tensor sample makes the sparsity concrete — the change is spread thinly, and even the busiest tensor moves under 1%:

<BenchBars
  title="Per-tensor change between adjacent checkpoints (%) — illustrative sample"
  unit="%"
  bars={[
    { label: "embed_tokens", value: 0.0 },
    { label: "attn.q_proj", value: 0.2 },
    { label: "attn.o_proj", value: 0.4 },
    { label: "mlp.gate_up", value: 0.7, highlight: true },
    { label: "mlp.down_proj", value: 0.2 },
    { label: "final_norm", value: 0.1 },
  ]}
/>

The mechanism is a periodic **full base checkpoint every N steps** to reset the chain, then a **compressed delta** in between. Only changed weight slices survive into the payload; each carries reconstruction metadata (`prev_snapshot_id`, `tensor_checksum`, `shape + dtype`) so every rollout cluster rebuilds the exact next checkpoint **losslessly** from shared object storage, then verifies the checksum before swapping.

<Figure
  src="/articles/frontier-rl-cheaper/fig2.png"
  alt="Three-panel diagram of delta-compressed weight updates. Panel 1, identify changed weights: a table of tensors (embed_tokens 0%, attn.q_proj 0.2%, attn.o_proj 0.4%, mlp.gate_up 0.7%, mlp.down_proj 0.2%, final_norm 0.1%) showing adjacent checkpoints differ in a few chunks, not everywhere, so unchanged chunks stay out of the transfer. Panel 2, package changed tensors: keep only the changed chunks, bit-pack and encode them into a delta payload, and attach reconstruction metadata (prev_snapshot_id, tensor_checksum, shape and dtype). Panel 3, reconstruct and swap: fetch the previous snapshot, decode the payload, apply the delta, verify the checksum, then swap weights in place."
  caption="Identify the changed weights, bit-pack only those into a compact payload with reconstruction metadata, then rebuild and checksum-verify the exact checkpoint on the rollout side before an in-place swap (Fireworks AI, Fig 2)."
/>

In Fireworks' sample setup a full checkpoint is **1024 GiB**, the average delta between adjacent checkpoints is **20.3 GiB — 1.98% of the model**, and over a 50-step window that cuts cross-region transfer volume by **about 94%** versus moving the full model every time. The arithmetic is worth writing down. With window $W$ steps, a full checkpoint every $N$ steps ($f=\lceil W/N\rceil$ fulls), delta fraction $\delta$, and $R$ regions, the total cross-region volume for a checkpoint of size $C$ is:

$$
V_\text{full} = W \cdot C \cdot R, \qquad V_\text{delta} = \bigl(f\,C + (W-f)\,\delta C\bigr)\cdot R
$$

so the fraction saved is $1 - \bigl[f + (W-f)\,\delta\bigr]/W$, independent of both $C$ and $R$. Plug in $W=50$, $N=25$, $\delta=0.0198$: $f=2$, and you move $[2 + 48\cdot0.0198]/50 \approx 5.9\%$ of the naive volume — the reported ~94% cut. Drag the delta size and the cadence and watch where the crossover sits:

<DeltaCost />

Two things the model makes obvious. The **percentage** saved does not depend on how many regions you feed — but the **absolute** bytes you stop moving scale with every region, which is the entire point of going distributed. And push the delta slider toward 100% and the two curves converge: at full-checkpoint-every-step you are back to the mega-cluster premise, where a co-located RDMA fabric is the only thing that can absorb the traffic.

## Async RL, and why the delta size decides it

Small deltas are necessary but not sufficient. The other half is **asynchronous RL** (also called Pipeline RL): deliberately let the rollout fleet run a little **off-policy** so that generation and training overlap instead of taking turns. Idle samplers are the expensive failure mode; a few steps of staleness is usually an acceptable price to keep them busy.

That trade only works if the handoff is fast. Delta-compressed updates keep it small: Fireworks reports distributing a new checkpoint across globally-distributed rollout clusters takes **a few minutes end-to-end**, and the actual in-GPU-memory **weight swap stays well under a minute** because download and decompression are pipelined ahead of the swap. The trainer side is pipelined too — every step uploads to shared object storage, each rank caches its previous upload and transmits only the diff, upload is sharded across training GPUs, download across inference replicas, and compression plus transfer plus signaling run in the background so training never blocks.

The payoff is where the wall-clock goes: less time waiting on checkpoint movement, more time generating rollouts on fresh weights. But the async win depends on the delta being small enough to hide behind a generation window. Drag the payload up and watch the "warm" fleet start to stall and fall off-policy:

<RolloutTimeline />

This is the honest coupling: async RL alone does not save you, and delta compression alone does not save you. It is the two together — a handoff small enough to overlap with generation — that turns a distributed fleet into usable capacity.

## A note on staleness

Running trainer and fleet asynchronously means the fleet always serves a policy a few steps behind the trainer. That gap is **staleness**, and it is a real tradeoff, not a free lunch. The systems layer does not remove it — the *algorithm* still has to tolerate off-policy data. What delta compression buys is a staleness that is **bounded and predictable**: policy movement becomes a routine background operation instead of a stop-the-world full-checkpoint transfer. If your RL algorithm cannot stomach any off-policy data, none of this applies.

## Multi-region rollout capacity

Here the systems point turns strategic. Most teams do not have one contiguous idle supercluster for rollouts; they have GPUs scattered across regions, clouds, and availability zones, and stitching them into one co-located sampler fleet is painful even when the aggregate count exists. Once weight updates are small, that fragmented capacity becomes usable: each rollout cluster independently pulls and reconstructs weights from the same shared delta chain, with **no direct connection back to the trainer**. Add, remove, or rebalance clusters while they all track the same stream of policy updates.

Fireworks cites this in production: they say they ran Cursor's **Composer 2** RL training this way, with Federico Cassano describing the run as ["distributed across 3 (sometimes 4) different clusters around the world"](https://x.com/ellev3n11/status/2034778708163404102). Treat that as a vendor-provided data point — a single external quote, not a controlled measurement — but it is at least a concrete one, and it is a coding-agent model, the same agentic-RL regime as [Agents-A1](/articles/agents-a1).

The approach is not new territory Fireworks invented alone: it names [AReaL](https://arxiv.org/abs/2505.24298) for async RL and rollout-training disaggregation, and engineering notes from [Kimi](https://moonshotai.github.io/checkpoint-engine/) and [MiniMax](https://www.minimax.io/news/forge-scalable-agent-rl-framework-and-algorithm) on RL parameter updates and async scheduling. The contribution is running the delta-compressed, multi-region version in production.

## When this argument stops working

The blog is unusually clear about its own boundaries, and the caveats matter:

- **Small models.** If trainer and rollout inference fit on one node or a compact cluster, bandwidth was never the bottleneck and the simpler co-located setup wins. The whole argument is about the ~1 TB regime.
- **Very frequent checkpoints.** If the trainer emits updates faster than the delta pipeline can distribute and apply one, staleness becomes the limiting factor and tighter co-location can make more sense again.
- **Entangled rollout stacks.** If your rollout workers don't cleanly separate inference from training, treating them as a standard inference fleet is a poor fit and the disaggregated design loses its appeal.

There is also an assumption baked into the headline sparsity number: it is the fraction of `bf16` weights that stay **bit-identical**. That is exactly the right metric for a lossless delta, but it is a property of the *representation*, not just the math — a run at higher precision, a larger learning rate, or a more aggressive RL objective could move more bits and shrink the win. The ~2% is a measured sample, not a guarantee.

## The take

Strip the vendor framing and the load-bearing claim is a clean systems observation: RL post-training updates are sparse enough that the weight-sync between trainer and rollout fleet — the thing everyone assumed forced co-location — is ~2% of what you feared. That is corroborated independently ([arXiv 2602.03839](https://arxiv.org/abs/2602.03839)), and the engineering that follows (delta compression + checksummed reconstruction + async overlap + sharded pipelined transfer) is the ordinary, correct way to cash it in. The reproducible headline — ~1 TB checkpoint, 20.3 GiB average delta, ~94% less cross-region traffic over 50 steps — comes from one Fireworks sample setup, and the Composer 2 case is a single external quote, so treat the specific figures as illustrative rather than benchmarked. But the shape of the argument survives that discount: if the weights barely change, the mega-cluster was never load-bearing for RL. That is a genuinely useful thing to know before you go shopping for a co-located supercluster.

---

*Built on Fireworks AI's [Frontier RL Is Cheaper Than You Think](https://fireworks.ai/blog/frontier-rl-is-cheaper-than-you-think) (published 2026-03-23). All quantitative figures are Fireworks-reported from a sample setup unless attributed to [arXiv 2602.03839](https://arxiv.org/abs/2602.03839); the `DeltaCost` and `RolloutTimeline` widgets are my own cost models of their argument (relative/illustrative units), and the two reproduced diagrams are from the source post for commentary. The per-tensor bar chart uses the figure's own illustrative sample values.*