Sol-Attn: deciding which attention blocks to skip while you're already streaming them

2026-08-03 · 9 min · diffusion · attention · sparse-attention · inference · video

Why read this

Explains the mean-plus-beta-sigma threshold inside the online-softmax loop, and subtracts the stacked speedups: Sol-Attn alone is 1.92x and 1.34x, not 5.08x.

A new technique
Open code or weights
Explained from first principles

Inference & servingNeeds datacenter GPUsApache-2.0Research paper

How this was scored

Is it new?: 2 of 3: A real new idea, method or capability
Can I trust it?: 1 of 3: Spot-checks a few numbers
Can I run it?: 2 of 3: Open code or weights with real limits
Will I understand it?: 2 of 3: Mechanism from first principles with figures
Can I act on it?: 1 of 3: General advice
Will it last?: 1 of 3: Relevant for months
Does it affect many?: 1 of 3: A specialist community
Only here?: 1 of 3: Some original analysis

Score 48 of 100, ranked 573 of 666 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Video generation has an attention problem that language models mostly don't. A few seconds of video at a useful resolution is a very long token sequence, attention is quadratic in it, and diffusion runs the whole stack dozens of times per clip. So attention stops being a cost and becomes the cost.

Sol-Attn — "Sparsifying online attention", from the SANA team at NVIDIA and collaborators — is a training-free way to skip most of that work. The idea I like is not that it sparsifies attention; everyone does that. It is where the decision happens.

The problem with picking blocks

Training-free sparse attention works block-wise: score each block of keys with something cheap (a proxy), keep the promising ones, skip the rest. The question is how you pick, and the paper's Figure 1 shows why the two standard answers both misbehave — on two different attention-logit distributions, one peaked and one nearly flat.

Four-column comparison on two rows of attention logits. Row 1 is a peaked distribution, row 2 nearly flat. Columns: Original logits; Top-k selection, which yields 70.3% sparsity on both rows; Top-p cumulative-probability selection, which yields 96.88% on the peaked row but only 21.9% on the flat row; and Ours, which yields 75.0% and 67.2% respectively, selecting blocks above a mean-plus-beta-sigma threshold shown against a density curve on the right. — Top-k gives the same 70.3% sparsity regardless of the distribution; top-p swings from 96.88% to 21.9%; thresholding at μ + βσ adapts but stays controlled at 75.0% and 67.2% (Li et al., 2026, Figure 1).

Read the sparsity numbers across the rows:

Top-k keeps a fixed fraction, so it reports 70.3% on both rows. It cannot tell a peaked distribution from a flat one. On the peaked row it is leaving free sparsity on the table; on the flat row it is throwing away blocks that mattered.
Top-p keeps blocks until their cumulative proxy mass hits a target, which is adaptive but wildly so: 96.88% sparsity on the peaked row and 21.9% on the flat one. Budgets swing by a factor of four between distributions, which is miserable for a kernel that wants predictable work per tile.
Sol-Attn thresholds at μ + βσ — the mean of the block scores plus β standard deviations. It adapts (75.0% vs 67.2%) but stays in a controlled band.

That statistical threshold is the whole trick, and its virtue is computability. A mean and a variance can be maintained as a running summary while you stream blocks. A top-k ranking cannot: you have to see every score, materialize them, and sort. Which brings us to where the decision gets made.

routing a block-proxy row · top-k vs top-p vs thresholdsynthetic, illustrative

row shape

routing

selects 7 of 28 · sparsity 75.0%

β1

dynamic budget — calibrated by mean + σ, stays close across rows. Switch the row from peaked to diffuse at a fixed β: top-k's count never moves (it is defined to be k); top-p's count can swing by an order of magnitude, because the same cumulative-mass target lands on very different numbers of blocks depending on how concentrated the row is. Sol-Attn's threshold, calibrated to a shared standardized cutoff β, is the one that stays roughly put across both shapes — and it never needs the full sorted row to get there.

Folding the decision into online softmax

Flash-attention-style kernels already stream. They walk key/value tiles, keep a running maximum and a running sum, and rescale as they go — that is what makes softmax computable without ever holding the full attention matrix. Conventional sparse attention bolts a separate pass in front of this: score all blocks, build a proxy map in memory, rank it, then run the sparse kernel on the survivors.

Sol-Attn puts the decision inside the loop that was already running.

Kernel loop diagram. A grid loop iterates over query tiles. For each, an outer loop labelled Approx and Routing mean-pools the key tiles to produce proxy scores, compares them against a per-tile threshold to produce a binary mask, and routes the selected tiles onward. An inner loop labelled Exact Sparse then computes full attention on the selected key tiles while unselected tiles are marked Skip. — The two-level structure: an outer loop mean-pools keys into proxy scores and thresholds them into a routing mask, and an inner loop computes exact attention only on the tiles that survived (Li et al., 2026, Figure 2).

The outer loop computes proxy scores from mean-pooled keys and compares them against that tile's threshold, producing a 1/1/0-style mask. The inner loop then runs exact attention on the surviving tiles and skips the rest. Because the threshold is a statistic rather than a rank, no proxy map is ever materialized — the budget comes out dynamic and controllable, which is the combination neither top-k nor top-p manages.

online-softmax stream · decide on the fly, don't dropsynthetic, illustrative

sparsity target

4 exact · 3 approx of 7 streamed

scan position7 / 18

Sol-Attn · exact-or-approx (nothing dropped)92%

keep-or-drop · exact-only (below-threshold blocks vanish)57%

Every block left of the scan cursor has already been decided — dispatched to the exact inner loop (green) or, if its pooled-key score fell below the threshold, approximated from that same score via a zeroth-order Taylor term and folded into the identical online-softmax accumulator. Nothing right of the cursor has been read yet — that is what “on-the-fly” means. The two bars below are the payoff: keep-or-drop sparse attention only ever covers the exact fraction of the softmax mass, while reusing the proxy scores keeps coverage close to complete even as the exact budget shrinks (coverage numbers here are illustrative; the paper measures the same effect as relative ℓ2 error and cosine similarity in its Figure 9 ablation).

Not dropping, approximating

The second idea is smaller and does more work than it looks. Standard block-sparse attention treats an unselected block as if it contributed nothing. Under aggressive sparsity that assumption is exactly where the quality goes.

Sol-Attn has already computed a proxy score for every block, including the losers, since that is how it decided. So instead of discarding them it reuses those scores to approximate the skipped blocks' contribution — a correction term that costs nothing extra, because the information was a by-product of routing. Routing, sparse computation and approximation correction all happen in a single online-softmax pass.

That is why the accuracy curve degrades gracefully rather than falling off a cliff: the tail is attenuated, not deleted.

What it actually buys

The paper reports 2.1× end-to-end for video generation and 2.3× for video editing. The more useful chart is the cumulative breakdown, because it shows what is attributable to what:

Two horizontal bar charts of end-to-end latency. HunyuanVideo: baseline 866.9 seconds, plus kernel fusion 781.0 seconds at 1.11 times, plus diffusion step cache 328.4 seconds at 2.64 times, plus Sol-Attn 170.6 seconds at 5.08 times. Wan2.1-14B: baseline 563.8 seconds, plus kernel fusion 464.9 seconds at 1.21 times, plus diffusion step cache 217.6 seconds at 2.59 times, plus Sol-Attn 161.8 seconds at 3.48 times. — Cumulative speedups: the 5.08× and 3.48× totals stack kernel fusion and step caching before Sol-Attn is switched on (Li et al., 2026, Figure 3).

Those headline multiples are cumulative, so it is worth doing the subtraction. On HunyuanVideo, Sol-Attn takes 328.4 s down to 170.6 s — a 1.92× marginal gain on top of the other two techniques. On Wan2.1-14B it takes 217.6 s to 161.8 s, a 1.34× marginal gain. Real, and the largest single contributor in the Hunyuan case, but not 5.08×. Anyone quoting the total as an attention result is quoting three techniques.

The engine around it

Sol-Attn does not ship alone. It is one of five composable techniques in the sol-engine branch of NVlabs/Sana (Apache-2.0), described as "an efficiency-oriented inference codebase for high-resolution video diffusion, built on SGLang's multimodal_gen runtime".

sol-engine · five techniques, composed per modelGB200, warmup-excluded

Wan2.2 TI2V-5B(5B)·cache+kernel fusion2.89× end-to-end

Six of these eight lines never touch Sol-Attn at all — their sparse-attention slot, where they have one, is PISA, an earlier piecewise sparse-attention method from the same group. Sol-Attn is wired into two lines (highlighted green above) as of the same day this paper posted, and the engine repo itself marks their end-to-end number pending re-benchmark — it does not yet claim a shipped number for Sol-Attn in the engine. The paper's own separate integration test (different two models, B200 rather than GB200) reports 3.48× and 5.08×; that is a real number, just not one measured on this table under this harness.

The five: caching (reuse or skip denoising-step outputs, TeaCache/EasyCache-style), quantization (TransformerEngine NVFP4 4-bit, applied step-selectively), kernel fusion (memory-bound DiT ops — norm, activation, precision conversion), sparse attention (Sol-Attn), and token pruning (dropping low-salience video tokens during refinement steps). Reported end-to-end speedups, all on GB200 with warmup excluded:

Model	Speedup
Wan2.2 TI2V-5B	~2.89×
SANA-Video (2B)	~2.77×
LingBot-Video (30B)	~2.60×
LTX-2.3 (22B)	~2.38×
Cosmos3-Super (64B)	~2.27×
Wan2.2-A14B (14B MoE)	~2.17×

The consistency across 2B to 64B, dense and MoE, is the interesting part — these are mostly memory-movement and redundancy wins, so they do not evaporate as models grow.

There is also an agent-native workflow: the repo is set up so a coding agent (Codex or Claude Code) does environment setup, weight fetching and inference, troubleshooting as it goes. Worth noting on a site whose own content is written this way — treating "an agent will be the one running this" as a first-class install path is still rare.

Where this sits

The same lab has been attacking this cost from the other end. SANA-Video 2.0 makes attention cheap architecturally — linear attention for three of every four layers, a deep-compression VAE to shrink the token count before attention ever runs. That requires training the model that way. Sol-Attn is the training-free counterpart: take a model somebody already trained and skip work at inference. SANA-Video is literally the second row of the engine's own benchmark table, so both ends compose.

Against the site's other sparse-attention coverage — MiniMax's approach — the contrast is that most sparse attention is trained, with the model learning to live within a sparsity pattern. Sol-Attn assumes no cooperation from the model at all. And where MrFlow attacks diffusion cost along the step axis, Sol-Attn attacks the per-step cost; the engine's caching module is doing the step-axis job alongside it.

The take

The mechanism is the part worth keeping. Routing decisions in sparse attention are usually treated as a preprocessing step — score, rank, select, then compute. Sol-Attn's claim is that the ranking was never necessary: a statistical threshold gets you an adaptive budget from a running summary, which means the decision can live inside the streaming loop the kernel already runs, which means the proxy map never has to exist. And once you are computing proxy scores anyway, throwing them away for the skipped blocks is wasteful — reusing them as an approximation is close to free.

Both ideas come from asking where the information already is, rather than adding machinery. That tends to be the sign of a good systems result.

Sources: Sol-Attn: Accelerating Video Generation Inference via On-the-Fly Attention Sparsification (Haopeng Li, Yitong Li, Junsong Chen, Tian Ye, Haozhe Liu, Jincheng Yu, Duomin Wang, Ruihua Zhang, Zeke Xie, Enze Xie, Song Han; arXiv 2607.24027, 2026-07-27) for the method and Figures 1–3, and the sol-engine branch of NVlabs/Sana for the engine, the five techniques and the per-model speedup table. All figures are the paper's own, served locally. Marginal-speedup arithmetic is mine, derived from the latencies in Figure 3. The interactives are mine and illustrate the mechanism; they are not measurements.