Chimera: unbundling RoPE into a diffusion Transformer that extrapolates 6× on video

Visual generation is hitting the same wall language models hit a few years back: the tokens keep multiplying. A high-resolution image is thousands of tokens, a video clip is tens of thousands, and once you want text, image, and video sharing one context, full attention's quadratic cost stops being a rounding error and starts being the budget. Language models solved their version of this with linear and hybrid attention. The catch, as Chimera — Adobe Research's new hybrid visual diffusion Transformer — points out, is that those solutions don't transfer directly: a diffusion backbone has to preserve spatiotemporal locality and support genuinely bidirectional interaction across modalities, neither of which a causal language model has to worry about.

Chimera's answer is a single-stream backbone that processes text, image, and video tokens together, mixing them with Kimi Delta Attention (KDA) for cheap $O(N)$ state tracking, periodic Multi-head Latent Attention (MLA) for exact global interaction, and modality-aware short convolutions for local structure — with no positional embeddings anywhere in the stack. The paper backs this with HeteroP, a hyperparameter-transfer scheme built for a backbone that is not one uniform shape, and fits genuine Chinchilla-style scaling laws on top of it. The headline results: a real zero-shot 6× video-length extrapolation (5-second training clips generalizing to 30 seconds with only 6.5% FID degradation, versus 50%+ for two full-attention baselines), and a compute-efficiency claim over a matched Wan2.1 baseline that the abstract prints as 7.3× but the paper's own arithmetic, two pages later, computes as 6.8×. Both numbers are worth seeing, and the mechanism behind the extrapolation result is the more interesting story.

expand

One stream, three mechanisms, no positions

Text tokens (from a frozen T5-style encoder) and visual tokens (from a frozen Wan2.1 VAE, patchified) are concatenated into a single sequence and pushed through the same stack. Visual tokens are flattened in temporal-major raster order — position $(i,j,k)$ in a $(T, H, W)$ grid maps to sequence index $m = iHW + jW + k$, with a single image just a one-frame video — so there is one token order for every modality, not a per-modality scheme bolted on afterward.

Inside a block, the attention sublayer is either KDA or MLA on a fixed 3:1 KDA-to-MLA schedule: three linear layers, then one global layer, repeating. The first and last blocks use a dense SwiGLU feed-forward; every other block routes through sparse MoE — 56 experts, top-8 active, no shared experts, balanced by an auxiliary-loss-free bias added only at top-K selection (not to the mixture weights). With that bias, batch-level MaxVio — the metric tracking how far the busiest expert's load sits above the average — settles near 0.5; strip the bias out and it blows past 5, close to the theoretical collapse bound of 6. Every sublayer — attention and FFN alike — is wrapped in Identity Hyper-Connections (iHC): the residual stream is duplicated into $M{=}4$ parallel copies with token-dependent read/write gates, a simplified version of hyper-connections that fixes the residual-mixing matrix to the identity instead of learning a doubly-stochastic one via Sinkhorn iterations — cheaper, at the cost of losing learned cross-stream mixing.

None of KDA, MLA, or NoPE is new by itself — they're the same three ideas Kimi K3 uses to run a 1M-token language model, adopted here essentially as-is and pointed at diffusion instead of next-token prediction. KDA keeps a fixed-size recurrent state per head instead of a growing KV cache:

with $\alpha_t \in (0,1)^{d_k}$ a per-channel forget gate and $\beta_t$ a scalar write strength — the exact recurrence KDA has a half-life walks through: each channel forgets a fixed fraction of its state per step, so it has a half-life $n_{1/2} = \ln(0.5)/\ln(\alpha)$ measured in tokens. That article works the math out for language tokens; Chimera is the same forget-gate law now doing memory management for pixels and frames instead of words. MLA restores exact bidirectional interaction on top, compressing keys and values through a low-rank projection, with one twist: the direct key path is left unrotated — no RoPE at all, on either mechanism. That's the part worth slowing down on.

What RoPE was actually doing

Every visual diffusion Transformer before this one, and most language models, lean on RoPE to inject position. Chimera's authors run a mechanistic audit of what that rotation is actually buying, on Qwen3-4B first and then on the visual diffusion models FLUX.2 and Wan2.2, and decompose each attention logit into its per-frequency-pair cosine contributions (the pieces sum to the real logit to within $3 \times 10^{-13}$, so this isn't an approximation of the mechanism — it's an exact accounting of it).

The clearest case is layer 0, head 1 in Qwen3-4B, a strong "previous-token" head: 98.5% of its queries attend most strongly to the immediately preceding token. Trace why, and it's one channel pair — the fastest-rotating one, turning roughly one radian per token — whose cosine happens to peak exactly at offset 1. The slow, high-amplitude pairs sitting alongside it contribute a flat, content-driven background that doesn't move the peak at all. Search across every head for the best "attend exactly $n$ tokens back" detector and the same pattern holds: the strongest head hits a 0.56 fraction at $n{=}2$, but only 0.07 at $n{=}10$ — RoPE-based position selection is a short-range tool, not a general one, in both language and visual models alike.

From that audit the paper pulls out three things RoPE is doing at once, inside the same rotated channels:

1. Position selection — a canonical previous-token/induction-head trick, and (per the numbers above) reliable only at short offsets.
2. Recency decay — an implicit average property of the summed rotations, not an explicit mechanism, and one a trained head can learn to bypass.
3. Layout encoding — the channel partition across positional axes (time, height, width, text index) is set by hand at design time; every additional axis divides the available channels further and never adapts.

Chimera's move is to stop asking one set of rotated channels to do all three jobs and give each its own dedicated module instead:

Token order for KDA comes for free from the recurrence itself — a scan is inherently order-aware, no rotation required. Position selection moves to the modality-aware short convolution: a depthwise kernel mixing tokens at explicit index offsets, which is a cheap, parameter-light way to do exactly the short-range job the audit found RoPE was actually good at. Recency decay moves to KDA's own forget gate $\alpha_t$ — explicit and content-adaptive, rather than an emergent side effect. And layout encoding falls out of the convolution's native shape: causal 1D along the token index for text, causal-in-time 3D over the $(T,H,W)$ grid for video — each modality's structure is encoded by the operator itself, with no channels spent partitioning anything. Text and visual tokens each get their own kernel here, implemented as one fused Triton pass that measures 2.2–2.3× faster forward, 1.5–1.8× faster forward-plus-backward, and up to 4× less peak activation memory than the naive gather-convolve-scatter version.

With all three biases reassigned, MLA is left to do only content matching — the paper's framing is that MLA without RoPE is the limiting case where every channel has zero rotary frequency, so its logits depend purely on content. KDA's queries and keys carry no positional phase either. Nothing in the stack is tied to how long the training sequences were, which is the actual, mechanistic reason extrapolation works — not a property tacked on after the fact, but the direct consequence of where each inductive bias now lives. It's the same bet Kimi K3 makes for a 1M-token language model: no RoPE means nothing to rescale when the context grows past training length. Chimera is that bet, replayed in a diffusion Transformer over image and video tokens instead of a causal LM over text.

HeteroP: a scaling ratio per tensor, not per model

Fitting a Chinchilla-style law needs a family of models at different sizes, each trained with hyperparameters as good as they'd be at full scale — otherwise you're not comparing model sizes, you're comparing tuning quality. The standard fix is µP-style hyperparameter transfer: tune a small proxy model, then derive the large model's learning rate, init, and weight decay from a single width ratio. Chimera's backbone breaks that assumption, because it isn't one uniform width. Widening the model changes the KDA head width, the MLA compression rank, the MoE expert width, the router width, and the timestep-conditioning MLP width all differently — a single global ratio, tuned for the backbone, is the wrong ratio for the rest.

HeteroP's fix is to stop pretending there's one ratio. For each parameter group $W$, it computes its own width ratio from that group's own functional fan-in, plus one shared depth ratio from the block-count ratio:

Concretely: hidden weights get init variance and learning rate scaled by $m_W^{-1}$ and weight decay scaled by $m_W$ (keeping the LR-times-decay product invariant); attention and FFN residual branches get an additional $m_L^{-1}$ output scaling, a depth correction borrowed from CompleteP; input adapters, norms, and the readout keep their base LR and init (standard µP convention), with the readout's forward pass separately rescaled by $m_W^{-1}$. The proxy model is width 512, depth 4 (22M activated, 59M total parameters); the largest is width 2048, depth 32.

The validation is direct: under HeteroP, the optimal base learning rate sits at about $10^{-3}$ across a 56× range in activated parameters (20M to 1.12B) and an 8× range in depth (4 to 32 layers). Under standard parameterization — one global ratio for everything — the optimum drifts sixfold, from $10^{-4}$ to $6\times 10^{-4}$, and several of the high-learning-rate runs at large scale diverge outright. That drift isn't just an inconvenience for the scaling-law fit, it actively biases it: trained without HeteroP, the same image model family gives a fitted exponent of $N_{opt}\propto C^{0.588}$ (envelope) or $C^{0.581}$ (isoFLOP) — inflated by 0.08–0.10 over HeteroP's 0.505/0.481 — which the paper's own extrapolation shows prescribes a compute-optimal model roughly 2× oversized (and correspondingly undertrained) three orders of magnitude of compute out from where it was fit. Get the transfer wrong, and the law tells you to build the wrong-shaped model at scale.

What the scaling law says

With HeteroP holding hyperparameter quality constant across scale, the paper fits $\hat L(N,D) = E + AN^{-a} + BD^{-b}$ — activated parameters $N$, visual-latent-token count $D$ — using three independent estimators (a training-loss envelope, an isoFLOP profile, and a direct parametric fit) for image and video pretraining separately. All three estimators agree, and they disagree with each other by modality:

The parametric fits — $\hat L_{image} = 0.126 + 5.28N^{-0.315} + 33.8D^{-0.336}$ ($R^2{=}0.993$) and $\hat L_{video} = 0.124 + 8.07N^{-0.330} + 145.2D^{-0.394}$ — land on nearly identical irreducible-loss terms (0.126 vs 0.124), consistent with both modalities sharing the same denoiser and VAE. The paper adds an axis prior scaling-law work doesn't have: the compute-optimal image-to-video data ratio. It drifts from roughly 4:1 to 3:1 as compute grows from $10^{18}$ to $10^{19}$ FLOPs (image gets relatively cheaper to learn from per token as budget grows), while the video-loss-optimal ratio stays pinned at 1:1 — the video-heaviest mixture the authors actually tested, so that half of the result is a boundary effect, not a discovered optimum, and the paper is upfront that it didn't search past it.

The number that doesn't quite add up

Guided by those laws, the paper trains an 11B-total / 2B-activated Chimera and compares it against matched 2B full-attention baselines — Wan2.1 and Z-Image — all four models trained in-house to the same $5\times10^{20}$-FLOP budget on identical data. At a shared training loss of 0.149, Wan2.1 needs $4.29\times10^{20}$ FLOPs and Z-Image needs $3.75\times10^{20}$; Chimera-dense (no MoE, no iHC, no HeteroP) reaches it in $2.55\times10^{20}$ — a clean 1.7×. The complete configuration (MoE + iHC + HeteroP) reaches it in $6.27\times10^{19}$ FLOPs.

expand

Do the division on the two numbers printed a page earlier and you get $4.29\times10^{20} / 6.27\times10^{19} = 6.84$ — which is exactly what Section 5.5's own sentence says: "a 6.8× compute-efficiency gain over Wan." The abstract, the introduction, and the plotted label in Figure 12 above all instead say 7.3×, from the same pair of FLOPs figures.

Worth separating, too: the component ablation (dense → +MoE → +iHC → +HeteroP) reports a 4.1× cumulative gain at loss 0.149 — MoE alone gets to 1.5×, +iHC to 1.7×, +HeteroP the rest of the way. That 4.1× is measured relative to Chimera-dense, not to Wan2.1, so it isn't a third candidate for the headline number — it's answering a different question (how much of the complete system's win comes from which piece), and it's consistent with either the 6.8× or 7.3× reading of the Wan-relative number, since $1.7 \times 4.1 \approx 7.0$, which lands between the two and settles nothing on its own.

Zero-shot length extrapolation: NoPE's actual payoff

This is where the RoPE audit cashes out. Chimera is trained only on 5-second, 81-frame clips, then asked — with no length-specific fine-tuning at all — to generate 30 seconds, 6× its training length. Every metric below is computed only on the final 5 seconds of each generated clip, isolating the extrapolated region, over 512 generated vs. 512 reference videos at matched prompts, seeds, resolution, and fps:

expand

The numbers: Chimera's FID goes from 77.1 at 5 seconds to 82.1 at 30 — a 6.5% degradation. FVD moves from 685.8 to 829.5, up 20.9%. Wan2.1-T2V-1.3B's FID degrades 50.5% over the same stretch, HunyuanVideo-1.5's 53.6% — both well past the point where a video model's later seconds are visibly falling apart. Chimera also posts the lowest absolute FID and FVD of the three at 30 seconds, not merely the smallest percentage move — it isn't winning by having started worse and degrading less, it's ahead the whole way.

This is the sibling result to SANA-Video 2.0, the other linear-attention video approach covered here, and the two make an interesting contrast. SANA-Video 2.0 keeps the same 3:1 linear-to-global attention idea and Block Attention Residuals for cross-depth flow, but keeps RoPE and optimizes for raw single-GPU latency at a fixed, modest clip length. Chimera keeps RoPE out entirely and stakes the design on exactly the axis SANA-Video 2.0 doesn't test: generalizing far past the lengths it was trained on. Different bets, same underlying conviction that softmax attention over every token pair was never the part of video generation worth paying full price for.

What $O(N)$ buys in memory and latency

A softmax KV cache costs $O(N \cdot H \cdot d_h)$ — grows with sequence length. KDA's recurrent state costs $O(H \cdot d_h^2)$ — fixed, independent of $N$. Measured directly: a matched KDA/MLA and MHA/MLA backbone, both around 2B activated parameters at the same 3:1 ratio, batch size 1, BF16, 512 text tokens plus 18×28 visual tokens per frame, on one NVIDIA A100-SXM4-80GB —

— the linear variant supports 1.68× longer sequences before it runs out of memory, and runs 2.14× faster at the 255k-token point both backbones can reach. The paper is careful about what this comparison actually shows: FlashAttention removes the quadratic attention workspace, but not the quadratic arithmetic — so this isn't a straw-man comparison against an un-optimized baseline, it's the honest gap that remains after the standard fix.

Benchmarks, and what 600 H100-days buys

Trained for only about 600 H100-days, Chimera is competitive on text-to-image quality with models that cost far more to build:

On GenEval, Chimera lands at 0.82 overall — tied with Z-Image-Turbo, matching FLUX.1-dev, beaten only by Seedream 3.0's 0.84 — and it beats both FLUX.1-dev and Z-Image-Turbo on DPG-Bench specifically. The paper also quotes Z-Image-Turbo's own reported training budget, about 12.4K H100-days, as roughly 20× Chimera's — worth reading as a cross-lab comparison rather than a controlled one: different codebases, different clusters, and a number each lab measured on its own infrastructure, not a shared benchmark. The GenEval/DPG-Bench baseline rows themselves are the field's normal practice, too — each competitor's own published number, not re-run under Chimera's exact sampling protocol. Worth knowing before quoting either comparison as settled.

Honest limits

The paper is unusually direct about what it hasn't shown yet:

Set against that, the scaling-law and compute-efficiency measurements themselves are unusually rigorous for the genre: Wan2.1 and Z-Image aren't cited from their own papers here, they're re-implemented and trained in-house at matched 2B scale on identical data, specifically so the efficiency comparison isn't citing someone else's number under someone else's conditions.

The take

The RoPE audit is the part of this paper worth remembering past the benchmark tables. It's not "we removed positional embeddings and it worked" — it's a demonstration, with an exact per-frequency accounting, that RoPE was quietly doing three separate jobs through the same rotated channels, that one of those jobs (position selection) only works at short range anyway, and that giving each job its own dedicated, non-attention mechanism is what actually buys extrapolation — not a side effect of going linear, a direct consequence of where position now lives in the model. HeteroP is the less flashy but equally load-bearing half: none of the scaling-law numbers mean anything if the hyperparameters drift as you change scale, and a heterogeneous backbone needs a heterogeneous transfer scheme to keep them from drifting.

The 6.8-versus-7.3 gap doesn't undercut any of that — it's a rounding-sized discrepancy in one headline multiplier, not in the mechanism. But it's exactly the kind of thing worth checking yourself before repeating a number, which is the whole reason to read the arithmetic instead of just the abstract.

Source: Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers (Ge, Jiang, Wang et al., Adobe Research, 2026), read from the arXiv HTML render. Figures 1–3 here are the paper's Figures 2, 12, and 16b, reproduced for commentary; all benchmark and scaling-law numbers are the paper's own, self-measured against in-house-trained baselines except where marked as cited. The RoPE re-assignment, HeteroP drift, and length-extrapolation diagrams are mine — the first is a schematic of the paper's own finding, the second reproduces the real numbers from its Figure 7 ablation on an illustrative loss curve, the third traces the actual tested points from its Figure 16b. Related: Kimi K3 for where KDA, MLA, and NoPE come from; KDA has a half-life for the forget-gate math this piece leans on; and SANA-Video 2.0 for the other linear-attention video architecture on this site.