DiffusionART: why DiffusionNFT needed EMA, and the 12x that follows from fixing it

mdjsonmcpcopy↗ai

2026-10-13 · 25 min · diffusion · reinforcement-learning · video-generation · flow-matching · post-training · grpo

Why read this

Notable· top 60%

DiffusionNFT's hidden consistency force and ART's covariance fix from the proofs and code; the 12.02x traced to its run, steps vs rollouts, and the release gap.

• Original analysis
• A lasting reference
• A new technique

·Image & video generation·Needs datacenter GPUs·Mixed licences·Research paper

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

2 of 3: A reference for a year or more

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 66 of 100, ranked 181 of 513 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

The post that sent me here said Kling AI's new diffusion RL method trains video models with "12x fewer GPU hours than DiffusionNFT and Flow-GRPO-Fast." My first reaction to any GPU-hour ratio in RL is suspicion. Two reward curves never stop at the same place, so a ratio between them depends on which point you pick, which run you pick, and what each method counts as a step.

So I read the paper (arXiv:2610.12572, Zhenglin Wan and colleagues at NUS HPC-AI Lab, Kling AI and HKUST), its LaTeX source, the training code at github.com/vanzll/DiffusionART, and the two checkpoints in the Hugging Face collection.

The 12.02x holds up as arithmetic. It just comes from a different run than the paper's main video experiment, and two of the four milestones behind it are easier to read once you know a DiffusionART "step" is half a batch. What I did not expect is that the most useful part of the paper is not the speedup at all. It is a diagnosis of why DiffusionNFT, the method it beats, needs an exponential moving average to stay alive. That diagnosis is clean, it comes with ablations, and it explains the fix.

Two ways to turn a score into an update

Online diffusion RL has the same problem every time. You generate a batch of images or videos, a reward model scores each finished sample, and that single number has to become a gradient for a network that was called sixteen or thirty times along the way. I walked through the landscape in the DiffusionOPSD article; the short version is that there are two families.

Reverse-process methods like Flow-GRPO and DanceGRPO treat each denoising step as a stochastic policy action. That needs an ODE-to-SDE conversion so a transition has a likelihood, then a PPO-style ratio on sampled transitions, weighted by a group-normalised advantage. It trains on states the model actually visited. Its weakness, in the paper's words, is credit assignment: each transition gets pushed along its own sampling noise, scaled by an advantage that the transition only partly caused. Flow-GRPO is also what the AUK speech-editing model used for its TTS stage, so it is not just an image technique.

Forward-process methods like DiffusionNFT skip likelihoods entirely. Take the clean samples, re-noise them with fresh noise, and regress the velocity with a reward-weighted positive branch and a negative branch. It is supervised flow matching with weights, which is why it is fast. It is also, per its own authors, unstable without two things: training both branches together and rolling out from an EMA copy of the policy whose decay has to be tuned per task.

DiffusionART keeps NFT's regression and drops the re-noising. That one change is the whole method, and the paper spends its first half explaining why it matters.

What NFT's loss is really asking for

DiffusionNFT maps each sample's raw reward to a goodness weight $r \in [0,1]$ by centering within the prompt's group and clipping. With $\beta > 0$ it builds $v_{\theta,\pm} = v_{\text{old}} \pm \beta(v_\theta - v_{\text{old}})$ and minimises

where $u_t = x_0 - \epsilon$ is the rectified-flow velocity target for a sample $x_0$ re-noised to $x_t$. The paper rewrites this, usefully, as a single regression (its Equation 1): with $a = (2r-1)/\beta$, the loss equals $\beta^2\,\mathbb{E}\lVert v_\theta - v_{\text{tgt}}\rVert^2$ plus a constant, where

A good sample pulls the velocity toward its own noising velocity; a bad one pushes away from it. At the on-policy point, every sample's force is proportional to $(2r-1)(u_t - v_{\text{old}})$.

That residual, $u_t - v_{\text{old}}$, is where the trouble hides.

The puzzle: each branch alone collapses

DiffusionNFT's motivating result is that the ideal velocity fields of the positive, original and negative endpoint distributions are collinear. If you move toward the positive field, you are moving away from the negative one. Either branch on its own should be enough.

It isn't. The paper reruns NFT without EMA and trains each branch separately. Both collapse to a GenEval score of zero within about ten steps, while the joint objective climbs to around 0.9.

expand

The explanation, Proposition 1, comes from separating two velocity fields that NFT quietly treats as one. There is $v^{\text{old}}_{\text{rvs}}$, the field the sampler actually uses to generate the endpoints. And there is $v^{\text{old}}_{\text{fwd}}$, the ideal velocity you would get by forward-noising those same endpoints and asking for the conditional mean of $u_t$. A perfectly trained flow model has the two equal. A real one does not. Call the gap $e = v^{\text{old}}_{\text{fwd}} - v^{\text{old}}_{\text{rvs}}$.

With $g$ the reward-improving direction (Lemma 1 shows it is a conditional covariance between the goodness weight and $u_t$) and $q$ the expected goodness at a given state, the branch forces at the on-policy point are

The positive branch drags the velocity toward the forward field. The negative branch shoves it away. Together they agree on $g$ and disagree on $e$, and what survives is $(2q-1)e$: a force set by how much the sampler disagrees with its own samples, with no reward in it.

Prompt-level reward centering makes $q$ average to one half across a prompt's samples. That cancels the term on average, which is why joint training works at all. It does not cancel it at each state, and it does not make $\sum_i (2r_i - 1)\,e_i$ vanish, because $e_i$ varies from state to state. States that happen to sit in high-reward territory get pulled toward the forward field; low-reward states get pushed off it. The paper calls this the forward-reverse consistency force.

I like this argument because it predicts the ablation instead of explaining it after the fact. Remove one branch and you remove the cancellation entirely; the force is then a full $q\,e$ or $(1-q)\,e$, unopposed, and the model is told to fix a mismatch it cannot see in its reward. The proof in Appendix B.2 is short algebra on Equation 1, and the decomposition is exact rather than a first-order approximation.

EMA was restoring, not going off-policy

The second half of the diagnosis is about EMA. DiffusionNFT samples from a lagged copy of the weights. The obvious story is that slow-moving rollouts help, the usual off-policy argument. The paper tests it and finds a different mechanism.

Proposition 2 splits the NFT loss with the drift $d = v_\theta - v_{\text{old}}$ and the residual $h = u_t - v_{\text{old}}$:

The linear term carries the guidance and the consistency force. The quadratic term pulls the policy back toward $v_{\text{old}}$. On-policy, $d = 0$, so the restoring term contributes no gradient, and the residual consistency force has nothing to push against. EMA keeps $v_{\text{old}}$ lagged behind, so $d$ is nonzero and restoration switches on.

Two ablations back this. Keep EMA rollouts but replace $v_{\text{old}}$ with $v_\theta$ inside the target, so the data is off-policy but there is no anchor: it still collapses. Drop EMA but split every rollout into two minibatches and update once on each, so the second update sees a nonzero drift: it survives, at the price of reaching an OCR score of 0.9 in 320 steps instead of 21, which is the 15.2x slowdown on the left of the figure below.

expand

Panel (c) is the most interesting number in the paper to me. Averaged over rollouts 0 to 17, the restoring term is 9.4 times smaller than the guidance in output space, yet its parameter gradient is 4.9 times larger. The authors' reading is that per-sample guidance points in many different directions across states and mostly cancels when the gradients are summed, while restoration points consistently back toward the anchor and survives. So in NFT, a weak restoring force ends up steering the update, and progress slows. They flag the full analysis as future work; I would read it as a strong hint, not a proof.

That observation also tells you what to build. If guidance cancels because each state carries its own noisy per-sample direction, give every state of a prompt the same direction.

The fix: aggregate first, then assign

DiffusionART tilts the rollout distribution toward reward, $p_\tau(x_0 \mid c) \propto p_{\text{old}}(x_0 \mid c)\,e^{\tau R}$, and asks which way that moves the mean endpoint. Lemma 2 is a standard exponential-family fact: at $\tau = 0$ the derivative of the tilted mean is the covariance between reward and endpoint. Under rectified flow, $u_t = x_0 - \epsilon$ with $\epsilon$ independent of $x_0$, so that covariance is also the reward-velocity covariance at every noise level. Estimate it from the prompt's $K$ samples and normalise (Equation 5):

Then, for every state $s_t$ the ODE sampler visited while generating that prompt's samples, freeze the target

and regress. The cached $v_{\text{old}}$ was recorded during the rollout, so nothing is re-noised and nothing is recomputed. The target never mentions the forward velocity, so there is no $e$ for anything to cancel. The abstract's phrase "removing the consistency force by construction" means exactly this.

expand

The code is as short as the equation. In src/visual_rl/algorithms/covariance.py, the per-prompt statistic:

# src/visual_rl/algorithms/covariance.py:71, 81-85
centered = r - means[groups]
numerator = x.new_zeros(group_count, x.shape[1]).index_add_(0, groups, centered[:, None] * x)
denominator = (counts - int(reduction == "unbiased")).clamp_min(1)
covariance = numerator / denominator[:, None]
covariance = torch.where((counts > 1)[:, None], covariance, 0.0)
rms = covariance.square().mean(1).sqrt()

It multiplies centered rewards by raw endpoints rather than by centered endpoints, which gives the same result because centered rewards sum to zero. It divides by $K-1$ rather than the paper's $K$, which the RMS normalisation then erases. And in art.py the loss is plain MSE toward a detached target:

# src/visual_rl/algorithms/art.py:91-94
target = (old - displacement).detach()
residual = target - v
desired_step = residual.detach()
unmasked = weight * residual.square().flatten(1).mean(1)

The minus sign is a convention, not a disagreement with the paper. The code's velocity points from data toward noise, the paper's from noise toward data. weight is the noise level $\sigma$, the paper's factor $t$ that emphasises high-noise states.

Playing with the widget makes one property obvious that the paper only states in passing. With $K = 2$, the centered rewards are $\pm(R_1 - R_2)/2$, so $G$ is just the winner's endpoint minus the loser's, rescaled to unit RMS. The size of the reward gap disappears in the normalisation. That means DiffusionART-Pairwise, which replaces scalar rewards with a 1 for the winner and 0 for the loser, produces exactly the same guidance as scalar DiffusionART when $K = 2$, except on ties. The pairwise $K = 2$ run in Table 1 is effectively a scalar $K = 2$ run that only had to ask which video was better. That makes the pairwise result less surprising, and it is still useful: a preference model is enough at that group size.

Compare this with DiffusionOPSD, which also turns the reward into a regression target but needs the reward's gradient to walk the clean-image estimate uphill. DiffusionART needs only reward values. Its direction comes from the spread of samples the model already produced, which is why it works with a black-box video reward model.

The second update and the mask

A single update toward $v_{\text{old}} + G$ learns fast and then decays. Figure 6(b) in the paper shows exactly that on OCR, labelled "ART: 1 update." So DiffusionART reuses the cheap trick from the NFT ablation: split the rollout into two disjoint minibatches and take one step on each, both toward the same frozen target. Before the second step, the policy has already moved by $d(s_t)$, so the gradient is proportional to $G(c) - d(s_t)$: the guidance plus a restoring term that grows with each state's drift. There is no EMA schedule to tune. In NFT the same trick cost 15.2x, because restoration overwhelmed scattered per-sample guidance. Here the guidance is shared across a prompt's states, so it survives aggregation and stays dominant.

The last piece is a PPO-style brake. ODE transitions have no likelihood, so the paper borrows Flow-GRPO's reference SDE step to define Gaussian transitions for the current and old velocity, and uses their per-dimension KL as a drift score (Equation 7). A state is dropped from the second update when that score passes $\delta = 10^{-5}$ and the move is outward, meaning the drift and the descent direction have a positive inner product. Corrections that pull back toward the anchor always pass:

# src/visual_rl/algorithms/art.py:134-136
masked = (active & corrective_mask_enabled & (energy >= threshold) & (alignment > 0)).detach()
keep = (anchor | (active & ~masked)).detach()
policy = torch.where(keep, unmasked, torch.zeros_like(unmasked))

In the video recipe, the reference SDE noise level is 0.8 and the threshold is the paper's 1e-5 (configs/examples/hunyuan_art.yaml). The paper is candid that the mask is a safety net, not the engine. On SD3.5-M and FLUX, training without it stays effective and can be slightly faster (Appendix A.6). On HunyuanVideo, the unmasked run drops around step 200 and recovers, and over the first 300 steps the maximum first-update gradient norm is 1.96 with the mask against 3.33 without.

Put together, one iteration is: sample $K$ videos per prompt with the deterministic ODE solver, caching states and predictions; compute $G(c)$ per prompt; freeze targets; take two masked MSE steps. No transition likelihoods, no re-noising, no EMA, no differentiable reward.

Where the 12.02x comes from

The post's claim comes from this figure.

expand

The abstract's sentence is careful, and once you know what it refers to it checks out. The run is a separate one with visual quality and motion quality weighted equally and no text-alignment reward. The main text-to-video experiment uses text alignment, visual quality and motion quality weights of 7:2:1. All three scores come from VideoReward on 53-frame, 480x480 HunyuanVideo samples. "Late-stage reward" is each baseline's level at the end of its curve, the open circles. Appendix A.5 gives the mean training reward over each run's final 50 steps: 13.018 for DiffusionART, 10.105 for DiffusionNFT and 8.510 for Flow-GRPO-Fast. 13.018 over 10.105 is 1.288, and over 8.510 it is 1.530. Those are the 28.8% and 53.0% exactly.

The 12.02x and 11.72x themselves are not in any table. Neither crossing time is given as a number in the text, so I could only check them against the plot. Reading the GPU-hour panel, DiffusionART crosses NFT's final level at roughly 125 GPU-hours and NFT gets there at roughly 1,450; Fast's level is crossed at roughly 90 against roughly 1,050. Both come out near 11.6 to 11.7x, which is as close as reading pixels gets you. I believe the labels, but I can't reproduce them to two decimals.

Three things about this plot are worth knowing before you quote it.

First, it is training reward on the reward model being optimised. The held-out evaluation for this 1:1 run is not tabulated. The main run's evaluation is (below), and it tells the same story, but the headline ratios are training-curve ratios.

Second, the GPU-hour axis is wall-clock times GPU count, and the repository's pre-release README, README.review.md, which is still in the tree, adds to the plot caption: "GPU hours include evaluation and waiting time." The public README dropped that clause. Nothing in the release says how much of each run's hours went to evaluation, so treat the axis as wall-clock cost, not pure training compute.

Third, "training steps" means optimizer updates, and the methods do not take the same number of updates per batch of generated videos. The repo's docs/paper-video.md says it plainly: "ART/Fast/Dance make two disjoint updates per rollout (200 rollouts for U400); NFT/ReFL make one (400 rollouts)." Every Hunyuan rollout is 32 prompts times 8 videos, 256 videos, whatever the method. So DiffusionART at step 20 has generated 10 rollouts' worth of video, and DiffusionNFT at step 50 has generated 50.

That explains a reply under the post that noticed the oddity: against Flow-GRPO-Fast the plot says 22.1x fewer steps but 11.72x fewer GPU hours, and against DiffusionNFT it flips, 5.2x on steps and 12.02x on hours. Put the paper's own step and GPU-hour pairs on one table and divide.

Per rollout, DiffusionART and DiffusionNFT cost about the same: 4.4 to 5.3 GPU-hours for ART across the three milestones, 5.0 to 5.1 for NFT. Both train on nine denoising positions per video, so that is what you'd expect. Against NFT, then, the GPU-hour ratio is roughly the rollout ratio. At visual quality 6, that is 10 rollouts against 50 (the paper's 2.5x step ratio becomes 5x in rollouts, and 5.7x in GPU-hours). The step axis understates DiffusionART's advantage over NFT by about 2x.

Flow-GRPO-Fast goes the other way. It takes two updates per rollout like ART, but trains on two of the first six positions per video instead of nine, so each rollout costs 2.35 to 2.55 GPU-hours, about half of ART's. Its step ratio overstates the compute gap by nearly 2x. The 11.72x is the honest number and the 22.1x is not.

Neither correction rescues the baselines. At every milestone DiffusionART needs a fraction of the compute. The cleanest line in the paper is the 7:2:1 run reaching visual quality 5.5 in 63.4 GPU-hours against DiffusionNFT's 1,401.4: 11.4x fewer steps and 22.1x fewer GPU hours. The same 22.1 appears on Figure 1 for a different comparison, so be careful which one you quote.

One more provenance note. In the release's config inventory, the Flow-GRPO-Fast long run behind Figure 1 is marked unresolved: configs/paper/video/cover_hunyuan_fast_11_long.yaml is a "bounded 400-update reference," because "neither supplied inventory nor inspected source gives its paper-plotted endpoint." The plotted run went past 1,000 steps. So that curve cannot be re-run from the release as published.

expand

The main run, evaluated

The headline experiment is full-parameter fine-tuning of the CFG-distilled HunyuanVideo-13B, learning rate 1e-5, no KL penalty, 400 updates, 16 sampling steps for rollouts, with text alignment, visual quality and motion quality weighted 7:2:1. Evaluation uses VideoReward at 4 FPS on held-out prompts with a 30-step sampler (Table 3):

The 31.4% and 57.4% gains over "the strongest baseline" are against DiffusionNFT (EMA): 8.970 over 6.827 and 4.224 over 2.683, both check. Text alignment ends level, 2.111 against 2.096. Two rows stand out to me. Both GRPO variants finish with lower visual quality than the model they started from. With 70% of the reward on text alignment, they spent their budget there. And on-policy NFT drives text alignment to −4.480. The first half of the paper predicts exactly this collapse, and here it shows up in a production-scale model.

expand

Image-to-video on the step-distilled Wan2.1-I2V-14B, with four sampling steps, is the most lopsided result. DiffusionART reaches motion quality 6 in 45.8 GPU-hours against Flow-GRPO-Fast's 220.3, and ends at 11.196 VQ and 10.476 MQ, against Fast's 5.771 and 4.710 (Table 4). NFT plateaus early; the authors attribute that to the residual consistency force. On images the gaps are smaller, which is useful calibration. On SD3.5-M DiffusionART matches tuned DiffusionNFT (EMA) without per-task tuning, and reaches Flow-GRPO's step-500 reward in 10.0x, 5.2x and 9.6x fewer steps on GenEval, OCR and PickScore. On FLUX.1-dev with NFT's SD3.5 hyperparameters carried over unchanged, GenEval lands at 0.9684 against 0.8490 for NFT and 0.9183 for Flow-DPPO (Table 2).

The group-size table is worth reading with a little care (the text calls it "Table 8"; it is Table 1). Pairwise $K = 2$ ends at VQ 12.235 in 260.8 GPU-hours, and pairwise $K = 8$ at 12.092 in 883.3, which is where the 3.4x saving comes from. But pairwise $K = 4$ lands lower than both, at 10.883, and scalar $K = 8$ beats every pairwise run at 13.637. With one run per cell, I read "insensitive to $K$" as "$K = 2$ is good enough here," not as a trend.

What I could not find anywhere: repeated seeds, error bars, or a human preference study. Every video number is VideoReward scoring HunyuanVideo or Wan outputs. Evaluation uses held-out prompts and a different sampler, which helps. It is still the same reward model family that was optimised, so a reader worried about reward hacking has the qualitative grids in Appendix D and nothing else.

What is actually on Hugging Face

The collection has two models. vanzl/DiffusionART-SD35M-MultiReward is a rank-32 PEFT LoRA for SD3.5 Medium, from step 300 of the four-reward run, under Stability's community licence. Its release.json records GPU inference verified on an H800.

vanzl/DiffusionART-HunyuanVideo is full transformer weights. I summed the twelve shard headers: 1,264 tensors, 12,821,012,544 parameters, all F32, 51.3 GB. The card asks for roughly 100 GiB of GPU memory and disk for its verified inference path, and the weights are under the Tencent Hunyuan Community License.

Its release.json is unusually honest, and it says this is not the model in Table 3. The source run is a different one (tk1w4go1), with 7:2:1 weights, a 100-update target, and a reference_mse_beta of 0.1 toward the pretrained velocity. The paper's runs use no KL regularisation, so that is an extra regulariser. The run's recorded terminal state is crashed_after_optimizer_step_40, and the released checkpoint is optimizer step 30 (rollout 15). Its own evaluation on 100 validation prompts reads VQ 6.51, MQ 2.68, TA 1.68. A solid step up from the pretrained 5.195, 1.943 and 1.118, but well short of the 400-update 8.970, 4.224 and 2.111. The same file sets independent_gpu_inference_verified to false. If you download it to see the paper's results, you will see an early, regularised checkpoint. Weighting the reward toward visual quality early and adding a pull toward the base model is a reasonable thing to ship. It just isn't what the tables measured.

The code is in better shape than most paper releases. It has one framework with DiffusionART and six baselines across SD3.5-M, FLUX.1-dev, HunyuanVideo and Wan2.1, pinned environments, about a hundred test files, and per-run provenance that admits what is missing. docs/paper-video.md describes its YAMLs as "training-semantic recipes, not bitwise trajectory reproductions," and lists which runtime commits could not be recovered. Two practical caveats. The video recipes assume 4 nodes of 8 GPUs, and THIRD_PARTY_NOTICES.md says "a project-wide license has not yet been chosen," so the code has no licence of its own yet, beyond the MIT and Apache-2.0 notices for the Flow-GRPO, DiffusionNFT and DanceGRPO code it derives from.

What I'd take from it

The durable contribution is Section 3. "NFT needs EMA" was folklore with a tuning table attached. This paper gives a mechanism: a self-consistency force that only cancels on average, and a restoring term that EMA happens to switch on. It backs both with ablations. Anyone running forward-process diffusion RL should read those two propositions even if they never touch DiffusionART.

The method is small enough to port. If you already have an NFT-style trainer that caches rollout states, the change is: replace the re-noised per-sample target with $v_{\text{old}} + G(c)$ on the visited states, split the batch into two optimizer steps, and add the outward-drift mask. The paper and the 1:1 run make a strong case that this is both faster and steadier than NFT and the GRPO variants on large video models, and on images it at least removes the per-task EMA tuning.

The 12.02x is real. I would quote it with its context: one VQ:MQ 1:1 run, training reward, GPU hours that include evaluation and waiting, and baselines that were run once. Against Flow-GRPO-Fast, quote GPU hours, never steps. And if you want the model from the tables, it is not on Hugging Face yet.

How I checked

I read the arXiv HTML and the LaTeX source (main.tex, Appendix A and B), rendered the figures from the source PDFs, and recomputed every ratio the text gives: 273/24, 1401.4/63.4, 50/20, 427/20, 250.2/44.2, 502.5/44.2, 33/19, 117/19, 165.6/42.9, 149.4/42.9, 320/21, 883.3/260.8, the final-50-step means behind 28.8% and 53.0%, and Table 3's 31.4% and 57.4%. All match to the stated precision. The 12.02x and 11.72x have no underlying numbers in the text, so I read the crossings off Figure 1 and got about 11.6 to 11.7x for both. Per-rollout costs use the updates-per-rollout counts from docs/paper-video.md in the repo at commit 016610f, which I cloned and read without running: covariance.py, art.py, nft.py, training/objectives.py, and the paper configs in configs/paper/video/. The "evaluation and waiting time" clause is from README.review.md in the same commit. Model facts are from each Hugging Face repo's release.json and card, plus the safetensors headers of all twelve HunyuanVideo shards, read by HTTP range request. I did not train or run anything, so the curves themselves are the paper's. The DiffusionART-Pairwise equivalence at $K = 2$ is my own algebra on art_pairwise_covariance, not a claim the paper makes.