SANA-WM: a minute of camera-controlled world, and where its memory actually lives

mdjsonmcpcopy↗ai

2026-10-15 · 21 min · world-models · video-generation · linear-attention · diffusion · nvidia

Why read this

High· top 30%

Counts SANA-WM's 2.6B from the weights, shows the 5 softmax layers still dominate cost at 60 s, and pins down what each single-GPU claim includes.

• Original analysis
• Widely used
• A lasting reference

·Image & video generation·Needs a workstation GPU·Mixed licences·Practitioner model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

2 of 3: A reference for a year or more

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 70 of 100, ranked 111 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

A reshare doing the rounds puts it simply: NVIDIA has a 2.6B open-source world model that takes an image, a text prompt and a camera trajectory and turns them into a controllable world, "all on a single GPU", RTX 5090 or H100. That is a fair summary of the abstract of SANA-WM (Zhu, Liu, Zhao et al., NVIDIA, May 2026), which promises 720p, one-minute videos with 6-DoF camera control, trained in 15 days on 64 H100s.

I have written about this family twice. Sana is the image model, where I rebuilt the FLOP table from the configs and found the deep-compression autoencoder doing more of the work than the linear attention. SANA-Video 2.0 is the video model that gave up on pure linear attention for a 3:1 linear-to-softmax hybrid. SANA-WM is the third step: the same 3:1 shape, but the linear layers are now a gated delta rule that runs one frame at a time, and the model has to follow a camera for a full minute instead of filling five seconds.

I cloned NVlabs/Sana at commit 0354b3d (2026-10-11), read the SANA-WM model, camera and streaming code, read the paper (arXiv v1), and range-read the safetensors headers of the three released Hugging Face repos without downloading any weights. Every number below is labelled: measured (I computed it from a file), reported (the authors' figure, not re-run) or reasoned (my arithmetic on the other two). I did not run the model; the full bidirectional release is 102.5 GB of files at its current revision (measured, from the Hub listing; the card below shows the repo's 145.79 GB of storage, which probably also counts superseded files such as the single 41 GB refiner.safetensors the model card's own table still names).

expand

What it is, in one table

Why a minute is hard: count the tokens

Start where the cost starts. The LTX-2 VAE squeezes 8 raw frames into one latent frame in time and 32 pixels into one cell in each spatial direction; the DiT patch size is 1 (the x_embedder weight is [2240, 128, 1, 1, 1], measured). At the fixed 704 × 1280 output, one latent frame is 22 × 40 = 880 tokens. A 60-second clip is 961 raw frames: the first latent frame holds frame 0 on its own, and each of the next 120 holds eight. So the transformer sees 121 × 880 = 106,480 tokens at once (reasoned, from the config).

Full softmax attention over that is the familiar quadratic. Per layer, forming $QK^\top$ and multiplying by $V$ costs about $4N^2d$ floating-point operations, with $N$ the token count and $d = 2240$. At $N = 106{,}480$ that is $1.0 \times 10^{14}$ per layer per forward pass (reasoned). The weight matrices, by contrast, cost $2PN$ for $P$ parameters: $5.7 \times 10^{14}$ for the whole 2.66B network (reasoned). Twenty softmax layers would spend about 3.6 times the rest of the network on attention alone.

Memory is the sharper edge for a model that rolls out autoregressively. A softmax layer that wants to recall the whole past has to keep its keys and values: $2Nd$ numbers, 0.95 GB per layer in bf16 at 60 seconds (reasoned). The paper's own scaling plot shows an all-softmax variant running out of an 80 GB H100 at 40 seconds (reported, Figure 7b).

The linear part: a gated delta rule, one frame per step

SANA-Video's linear attention keeps a running sum of key-value outer products, a $D \times D$ matrix per head, and reads it with the current queries. Its weakness at a minute is that nothing ever leaves: "stale features accumulate with the same effective weight as more recent ones", and the paper reports drift and unstable training as the state grows. The delta-rule family is the standard fix, and the state-size question is the standard worry about it.

Gated DeltaNet decays the state and writes only what it failed to predict:

Here $S_i$ is the state, $\gamma_i \in (0,1]$ a forget gate and $\beta_i \in [0,1]$ a write strength. The bracket is the error between the value the state should return for this key and the value it does.

SANA-WM's change is the step size. A language model scans one token per step; SANA-WM scans one latent frame per step, all 880 tokens at once:

That makes a minute 121 recurrent steps, not 106,480, and the state stays $112 \times 112$ per head: 20 heads × 12,544 numbers, about 1 MB per layer in fp32 (reasoned). It also creates a numerical problem the paper handles neatly. $M_t$ is applied 121 times, so it must not expand anything, and the subtracted term now sums 880 keys instead of one. With the keys RMS-normalised and scaled by $1/\sqrt{D \cdot S}$ (with $S = 880$ tokens), its trace is at most 1 and $\lVert M_t \rVert \le \gamma_t$. The paper's ablation shows why it matters: the usual $1/\sqrt{D}$ scaling hit NaNs at step 16, no scaling at step 1, and only the extra $1/\sqrt{S}$ trained (reported, Figure 6).

The bidirectional model runs this scan forwards and backwards and sums the two; the chunk-causal one keeps the forward scan global and resets the backward one at each chunk boundary. The parameter count of a GDN block and a softmax block is the same to within 0.1M (130.8M against 130.7M, measured): the softmax blocks keep the same QKV and output projections and simply skip the recurrence and the key convolution. The decay and write-gate tensors are still in the softmax blocks' checkpoint, unused.

expand

Where the softmax anchors sit, and what they still cost

Linear state cannot do exact recall. A $112 \times 112$ matrix is a lossy summary, so SANA-WM keeps exact attention in every fourth block, at indices 3, 7, 11, 15 and 19 (measured, softmax_every_n: 4 in _build_camctrl_cls_list). In the bidirectional model those five layers attend over all 106,480 tokens with no mask (measured: _forward_softmax_attn with frame_causal=False). The paper calls them the layers that "anchor long-term spatial consistency".

That is the design, and it is the right shape. What it is not is linear. Five quadratic layers are a quarter of twenty quadratic layers, and the quadratic term still grows. The widget runs the arithmetic on the released shapes:

The numbers it prints, all reasoned from the shapes above:

Per forward pass at a minute, the hybrid is about 2.4 times cheaper than all-softmax and about 1.9 times dearer than all-GDN. Nearly half its compute is in the five softmax layers. The point where those five layers cost as much as every weight in the network is about 119,000 tokens, roughly 67 seconds of video (reasoned). That is a model sized, deliberately or not, to stop at about a minute. These are FLOP counts, not timings: FlashAttention and the fused Triton GDN kernels run at different efficiencies, and the paper measures rather than models. Its Figure 7b puts the hybrid's single-step latency on an H100 between the cumulative-linear and all-softmax curves at every length, with all-softmax going out of memory at 40 seconds (reported).

The memory story is where the release splits in two. In the chunk-causal rollout, a softmax layer that keeps the whole past holds about 0.95 GB of keys and values per layer at 60 seconds (reasoned), which is why five of them fit and twenty did not. The streaming release goes further. Its stage-1 softmax layers keep the first chunk as an attention sink and a sliding window of past chunks, default --num_cached_blocks 2 with the sink token on (measured, inference_sana_wm_streaming.py and sana_streaming_cache.py). Reading the window logic, a chunk sees chunk 0, the chunk before it and itself: 4 + 3 + 3 latent frames, under 80 MB per softmax layer, flat in length (reasoned).

Here is what that means. In the deployed streaming model, exact attention reaches the opening frames and the last three seconds or so of video. Everything in between survives only as the GDN state. That is a legitimate trade, and the paper says so in its own words: the sink and window keep "softmax memory and per-chunk latency constant with respect to rollout length". But it is the opposite of the headline picture of softmax anchors holding the whole minute together. In the version that runs in real time, they do not.

expand

Camera control at two rates

A world model is only a world model if it goes where you point it. SANA-WM takes a camera-to-world matrix per raw frame (an (F, 4, 4) array) or a keyboard string such as w-100,dw-60 that the script integrates into one. The difficulty is the same VAE that made the minute affordable: one latent token summarises eight raw frames, and over eight frames the camera can turn several degrees.

So the camera enters twice.

• Coarse, at the latent rate: UCPE. For each token, the pose of its latent frame and the intrinsics give a world-space ray. The model builds a small coordinate frame around that ray and applies it, as a 4 × 4 transform, to a group of channels in the camera branch's queries, keys and values; the remaining channels get ordinary rotary embedding. Attention then compares tokens in each other's ray frames. The camera branch has its own Q, K, V and output projections in every block (four 2240 × 2240 matrices, measured), shares the main branch's GDN gates, and is zero-initialised.
• Fine, at the raw rate: Plücker mixing. For each of the eight raw frames inside a latent frame, every cell gets its Plücker ray: the direction $d$ and the moment $o \times d$, six numbers. Eight frames × 6 = 48 channels, passed through a zero-initialised patch embedder (plucker_embedder.proj.weight [2240, 48, 1, 1, 1], measured) and added after every block's attention through a per-block zero-initialised projection. The raymap is computed at the latent resolution, 22 × 40, not at pixel resolution (measured, _build_chunk_plucker).

The two halves of a Plücker ray separate the two kinds of motion. A pure rotation repaints the directions across the whole grid and leaves the moments alone if the camera sits at the origin; a pure translation leaves every direction unchanged and moves only the moments. Pick a path and scrub through the minute:

Everything the camera needs costs 502M parameters: the four camera projections and the Plücker projection in each of 20 blocks, plus the embedders (measured). That is 19% of the 2.66B, and the reason the checkpoint is named sana_wm_1600m_720p while holding 2.66B parameters: the backbone is the 1600M-class Sana configuration, 20 blocks at width 2240, and the camera branch, the output gates and the temporal convolutions in the feed-forward layers sit on top (reasoned from the tensor names).

The paper's ablation, on its held-out OmniWorld split with 5-second clips (reported, Table 4):

Input-level Plücker on its own barely moves rotation error. The work is done inside attention, and the raw-rate Plücker branch adds the last 14% of translation accuracy. Astronex-World uses PRoPE; on this one table, UCPE plus Plücker beats it on every camera metric and loses no FVD.

What keeps a minute consistent

There is no memory module. SANA-WM has no point cloud, no retrieved frames, no 3D state; the paper's own limitations section says it "lacks explicit 3D scene memory". Consistency comes from three things: the GDN state, the softmax layers (globally in the bidirectional model, windowed in streaming), and the camera conditioning, which tells every token where in the world it is looking. A second stage then cleans up.

The second stage is not small. It is an LTX-2 video model with a LoRA, trained to map a noised stage-1 latent to a high-fidelity one in three Euler steps, with a clean reference slice of the first frame prepended as fixed keys and values. The paper calls it 17B; the transformer in the release has 18,876,174,592 parameters, of which 5,714,096,256 sit in tensors with audio in their names (measured). LTX-2 is an audio-video model; SANA-WM uses the video half.

On the authors' own 60-second benchmark (80 scenes, simple and hard revisit trajectories), the refiner earns its keep (reported, Tables 2 and 8):

Camera following is the clear win: the lowest rotation error in the table on both splits. Visual quality with the refiner matches LingBot-World, which runs at 480p on eight GPUs. Revisit memory, the test of whether a place looks the same when you come back to it, is middling for everyone: 12 to 15 dB. WorldCrafter, which adds an explicit learned memory, scores 18.016 dB on its own benchmark and includes SANA-WM among its baselines. The benchmarks differ, so that is a direction, not a ranking.

expand

The 2.6B and the single GPU, claim by claim

The parameter count holds. The stage-1 checkpoint is 872 tensors and 2,663,575,828 parameters, stored in fp32 except the 691,200-number caption-padding embedding in bf16, which is why the file is 10.65 GB (measured). "2.6B" describes stage 1. The rows of the main table that match LingBot-World on quality are labelled "2.6B+17B".

The single-GPU claims come in four layers, and each is true of something different:

1. Stage 1 on one H100 (reported). Bidirectional stage 1 peaks at 49.2 GB and the autoregressive one at 51.1 GB (Tables 2 and 9). Both fit an 80 GB card with room to spare.
2. Stage 1 plus refiner on one H100 (reported). 74.7 GB peak, 22.0 videos an hour on eight H100s, one clip per GPU. The release's files add up to 102.5 GB, including 48.8 GB of text encoder for the refiner (measured), so the pipeline necessarily loads in stages; the docs offer --offload_vae and --offload_refiner for tighter cards.
3. "60 s of 720p in 34 s" on an RTX 5090 (reported). Figure 7a is explicit about what this is: a 4-step distilled, chunk-causal stage 1 with the attention sink and local window, quantised to NVFP4, and 34 s of DiT time. With the VAE it is 42 s. There is no refiner in that path. The same figure shows the distilled model without the sink running out of memory on the 5090.
4. Real-time streaming with the refiner (reported, partly). The streaming docs give 47.3 GB peak in bf16 and 29.4 GB with both stages in NVFP4, "Runs on a 32 GB RTX 5090". But the real-time multiples in that table are measured on GB200 and B200; there is no 5090 throughput figure. The H100 bf16 row says 1.09× real time, while the paragraph above it says "~0.93× realtime on a single H100". One of the two is stale.

So the reshare's "single GPU, RTX 5090 or H100" is true, and the two GPUs run different products. The H100 runs the model in the paper's tables. The 5090 runs a distilled, windowed, 4-bit stage 1, whose softmax layers see the start and the last few seconds, and whose quality the paper does not put in a table.

Against the other world models on this site

A different design shipped the same week: Matrix-Game 4.0 gives the world's rules to a game engine and asks a 5B video model only to repaint a low-poly frame, which is how it reaches 60 fps on one card. SANA-WM keeps the whole world inside the model, so it can go anywhere a camera can, and has to remember it.

SANA-WM's distinctive bet is architectural rather than about memory: make the backbone cheap enough to train natively on minute-long clips, rather than distilling a long model out of a short teacher. The paper's argument for doing it that way is that "short-horizon teachers provide limited supervision for minute-scale scene persistence". Its revisit numbers say native training alone does not buy persistence; WorldCrafter's say a memory might. For a cheaper way to stretch an existing model, NVIDIA's own LongLive-Plug distils long-context behaviour as a LoRA instead.

Licence: three sets of terms

• Code in NVlabs/Sana is Apache-2.0 (measured, LICENSE).
• Stage-1 weights: all three Hugging Face repos are tagged apache-2.0 (measured). The model card adds that "the bundled LTX-2 refiner and VAE inherit the LTX-2 upstream license".
• LTX-2's licence is the LTX-2 Community License Agreement, under which entities with annual revenue of at least $10,000,000 "are required to obtain a paid commercial" licence (measured, Lightricks' LICENSE). The refiner and both VAEs come under it, and the VAE is on every output path.
• The refiner's text encoder is a Gemma 3 12B, under Google's Gemma terms; the stage-1 text encoder is Gemma-2-2B.
• The example training dataset derived from Sekai "is redistributed for non-commercial research use only" (measured, docs). The paper itself is CC BY-NC-SA 4.0.

The frontmatter says mixed because it is. A research group can use all of it. A company above the revenue line cannot route around the LTX-2 terms by skipping the refiner: --no_refiner (which the docs call "lower-quality debugging") still decodes through the LTX-2 VAE that stage 1 was adapted to. The Apache-2.0 part it can use freely is the DiT, which produces latents nobody can look at without that decoder.

What I would check with a GPU

• Run the streaming model on a 5090 and time it. The only 5090 number in the release is the DiT time of a model that is not the one the docs say fits.
• Compare streaming against bidirectional on the revisit metric. If my reading of the window is right, a loop that returns after twenty seconds has to be recalled through a $112 \times 112$ state, and revisit PSNR should drop in streaming. The paper reports stage 1 autoregressive and bidirectional (Table 9) but not the distilled streaming model.
• Time a bidirectional forward at 30 and 60 seconds. My FLOP model says the five softmax layers go from 31% to 47% of the work; a profile would say whether the kernels agree.

The take: SANA-WM is a well-built efficient world model whose most interesting idea is unglamorous. A gated delta rule that steps one frame at a time, with a key scale chosen so a minute of steps cannot blow up, is what lets a 2.6B model train on minute-long clips at all. Treat the softmax anchors as expensive, and in the streaming release as short-sighted. Read "one GPU" as one of four different configurations.