WorldCrafter: skipping the reconstruction, not the 3D

mdjsonmcpcopy↗ai

2026-09-26 · 23 min · world-models · video-generation · 3d · diffusion · open-weights · explainer

Walk a video world model around a courtyard and back to where you started, and you usually arrive somewhere else. The pavilion has a different roof. The pond moved. Nothing is wrong with any single frame; the model simply has no record of the first visit that survives the trip.

The standard fix is to give it one in 3D: estimate depth for past frames, lift them into a point cloud, and warp that cloud into the view you are about to render. WorldCrafter, from ARC Lab at Tencent with Peking University, asks the obvious question. What if you skip the reconstruction entirely?

Having read the paper and the released code, my answer is that it skips the reconstruction but not the 3D. There is no depth map, no point cloud and no warp at inference. But 3D reasoning enters twice. Once in how past frames are chosen, which is a plain geometric frustum test on camera poses. And once in what they are fed through: a memory encoder initialised from a model trained to do novel-view synthesis. What reaches the video generator is shaped, literally, like four latent frames for where the camera is about to be. In effect, a learned view renderer stands where the explicit one used to.

expand

Why a revisit breaks

WorldCrafter is a chunked autoregressive video model. It generates nine latent frames (33 video frames) at a time, conditioned on a short window of what came before, then slides forward. In the paper's notation, each chunk's latent $\mathbf{z}_t$ flows under a velocity predicted from the memory $\mathbf{M}$, the recent frames $\mathbf{z}^{\mathrm{r}}$, the target camera trajectory $\mathbf{C}$ and the text $\mathbf{y}$:

Drop $\mathbf{M}$ and you have an ordinary streaming generator, of the kind ABot-World-0 runs on one desktop GPU. Its recent window is enough to keep motion continuous. It is not enough to remember a room you left a minute ago, because that room is no longer in the window.

The paper sorts the existing fixes into three families, and each has a known cost:

• Context memory puts old frames back into attention. Full history is expensive, so methods retrieve a few frames by camera overlap. A token budget that holds four frames holds four frames.
• Spatial memory reconstructs: depth, then a 3D representation, then a render or warp into the target view, concatenated with the noise. It aligns revisits well and, in the paper's words, "can overconstrain scene dynamics". It is also only as good as its depth.
• Implicit memory compresses history into a learned state. The recent geometry-aware variants borrow features from VGGT, which the authors argue keeps geometry and loses the appearance you need to redraw what you saw.

WorldCrafter is in the third family, with a different donor. It takes its encoder from LagerNVS, a model trained to synthesise novel views, on the argument that reconstructing an unseen view forces a representation to keep geometry and appearance. That is the whole bet.

The window, counted

Before the mechanism, the budget it has to live in. WorldCrafter's DiT is initialised from Helios-base, and its config is the familiar Wan-shaped one: 40 layers, 40 heads of 128, a 5,120-wide residual stream. Helios's window held a nine-frame noisy chunk plus a FramePack-style clean history: a compressed 16-frame segment, a 2-frame segment, the latest frame and an attention-sink frame. WorldCrafter deletes the 16-frame segment and puts memory tokens in its place, "equivalent in number to the tokens of 4 uncompressed history frames."

Counted from the shipped files, that is this:

A 384×640 frame is a 48×80 latent; at 1×2×2 patches that is 960 tokens per latent frame. The window is 3,840 memory tokens, 240 tokens of mid history (two frames at 2×4×4 patches), 1,920 for the sink and the latest frame, and 8,640 for the chunk being denoised: 14,640 in all, with memory at 26%. Those counts are my arithmetic from transformer/config.json and the safetensors shapes of patch_memory, patch_mid and patch_short, not figures the paper states.

Two details in the code matter. The memory arrives shaped exactly like four latent frames, [16, 4, 48, 80], and enters through its own patch embedding that is initialised from the clean-history one, so the DiT first sees memory as if it were four more clean frames. And only the noisy chunk gets the camera branch; memory and history run at timestep zero and carry no camera injection.

The ablation that matters most keeps this slot fixed and changes only what fills it. Put four retrieved raw latent frames there and revisit LPIPS is 0.497. Put the memory encoder's output there and it is 0.255. Same tokens, same training schedule, half the error. Everything below is about what the encoder does with its 3,840 tokens that four raw frames cannot.

Step one: pick the past by coverage, not similarity

The memory encoder takes a fixed nine latent frames. One is always the latest frame. The other eight come from the entire history, which is never thrown away; only the encoder's input is bounded.

The usual way to pick is to rank past frames by how much their field of view overlaps the target and take the top few. WorldCrafter's select_trajectory_fov_history does something more careful. It takes the four poses the next chunk will render (latent slots 2, 4, 6 and 8), samples each one's viewing frustum as a 10×10×10 grid of points from 0.1 m to 30 m, and marks which grid points each past frame's frustum contains. Then it picks greedily:

# worldcrafter/repencoder/trajectory_fov.py (trimmed; bookkeeping removed)
covered = anchor_fov.clone()                    # the latest frame is always in
for slot in range(budget):                      # budget = 8
    resulting = covered.unsqueeze(0) | candidate_fov
    full_fairness = torch.sort(resulting.float().mean(dim=2), dim=1).values
    connected = (candidate_fov & covered.unsqueeze(0)).sum(dim=(1, 2)) > 0
    rows = torch.cat((full_fairness, frontier_ratio[:, -1:], frontier_fairness,
                      new_fov_total[:, None], new_frontier_total[:, None]), dim=1)
    eligible = available & connected
    if not eligible.any():
        eligible = available                    # "disconnected fill"
    chosen = _pick_best(rows, available=eligible, stable_values=candidates)
    covered = resulting[chosen]

The key is lexicographic, and its first four entries are the per-target coverage sorted ascending. So each pick is the frame that most raises the worst-covered upcoming view, then the next worst, and so on. After that come "frontier" terms: the part of each target view that neither the latest frame nor an earlier target in the same chunk can see, which is the part memory has to supply. Candidates must share covered points with what is already chosen unless none do, and ties go to the more recent frame.

I ported that function to a top-down toy so you can watch it choose:

On the last chunk of the toy walk, the camera turns back toward the pavilion from where it started. Similarity ranking spends all eight slots on the frames of the last two chunks. They overlap the upcoming views the most, and they are near-copies of each other. Max-coverage takes none of them beyond the latest frame, which is always in. It reaches back to the first two chunks, which faced the pavilion, and fills the rest from four other parts of the walk. In the toy, the union goes from 67% to 100% of the sample points. Those percentages are my toy's, not the model's. The measured version is the paper's ablation, below: swapping max-coverage for similarity ranking moves revisit LPIPS from 0.255 to 0.296.

One thing the code does not do is check occlusion. A point counts as seen if it falls inside a past frame's frustum, even with a wall in between. And the frustum is a fixed 100° by 71.13° from 0.1 m to 30 m, which matches the default camera but not every scene.

Step two: an encoder that learned 3D from view synthesis

The nine chosen latents go into a memory encoder the code calls RepEncoder. Its architecture and weights come from the LagerNVS encoder, with one surgical change: the image-processing front end is gone. A new InputLayer takes each 16×48×80 latent, un-standardises it, resizes it to 44×74 and runs a stride-2 convolution to a 22×37 grid of 1,024-wide tokens. That grid enters DINO at its third block: the patch embedding and the first two blocks are dropped, and the remaining 22 are kept. The paper's Figure 2 still labels the box "DINO layers", which is consistent with that: the shallow ones are replaced, the deep ones stay.

After DINO comes a VGGT-style backbone that alternates per-frame and cross-frame attention, with each frame's camera pose (expressed relative to the latest frame) entering as a camera token. The output is a scene representation of 814 tokens per source frame, 768 wide: 7,326 tokens for the nine frames, holding geometry and appearance without ever becoming a point cloud.

expand

The parameter census, summed from the safetensors header of repencoder/model.safetensors:

That total matches the parameter_count the Fast package records in repencoder_frozen.json. 97,690,304 of it are unmerged rank-64 LoRA factors, which suggests the encoder was adapted with low-rank updates rather than fully fine-tuned. The paper does not say which; the file layout does. For scale, the DiT it feeds is 14,291,783,744 parameters.

Step three: read the memory out where you are going

The representation has 7,326 tokens and the DiT has room for 3,840. Something has to choose what survives. The paper compares two ways.

Pose-free readout maps the whole representation to 3,840 tokens and leaves the DiT's attention to find what is relevant. Pose-guided readout asks the representation a question: what would the scene look like from these four poses? WorldCrafter ships the second.

In code, the four query poses are the same slots 2, 4, 6 and 8 of the upcoming chunk. Each becomes a six-channel ray map at 384×640, cut into 8×8 patches: 48×80 = 3,840 query tokens per view. Nine bidirectional blocks cross-attend queries and scene tokens in both directions, a final block updates the queries, and a 3×3×3 convolution maps the result to 16 channels:

# worldcrafter/repencoder/model.py, forward_conditioned
input_features = self.input_layer(source_latents)      # [B, 9, 814, 1024]
dino_features  = self.dino_tail(input_features)
scene          = self.vggt(dino_features, source_camera_tokens)   # [B, 9, 814, 768]
target_features = self.repfeature(scene, target_rays)  # 4 target views
memory4        = self.output_layer(target_features)    # [B, 16, 4, 48, 80]

The readout is initialised from LagerNVS's shallow decoder layers, the part of LagerNVS that decodes a target view. So the memory is four latent-frame-shaped tensors, one per upcoming pose, produced by a network that began life as a view renderer. The paper is careful to say this conditions the DiT "without reconstructing target-view images", and nothing in the released code decodes memory4 or supervises it as an image. But its shape and lineage say what it is for.

The ablation says it matters. With the same inputs and the same token budget, pose-free readout gives revisit LPIPS 0.333 and rotation error 18.307; pose-guided gives 0.255 and 13.536. The authors attribute this to "a more effective allocation of the fixed memory budget to target-relevant information." A readout that knows where you are going can spend its tokens on what you will see.

Camera control, and getting to 16 fps

Camera control is a separate branch. The target trajectory gives a camera-to-world pose and intrinsics per frame. Following PRoPE, relative camera geometry enters self-attention as a positional transform, implemented as the parallel camera-attention branch from UCPE: its own query, key and value projections, added to the original attention through a zero-initialised projection. The Base package's camera_adapter.pth holds 400 BF16 tensors, ten per block across the 40 blocks, 1,050,777,600 bytes; at two bytes each that is about 525M parameters. That figure is my division, not a stated count.

Keyboard control compiles to that trajectory. The inference guide's actions are forward1, yaw_left30, pitch_up15 and so on, one action per 33-frame chunk, plus reverseN, which retraces the preceding N chunks. That is the revisit primitive, as a command.

Training is four stages, all on data whose camera poses come from Depth Anything 3 and whose captions come from Qwen2.5-VL:

So "no depth" is true at inference and not in training: a depth model produced the pose labels the whole system learned from.

The real-time model, WorldCrafter-fast, is distilled with distribution matching over a coarse-to-fine pyramid: 3 spatial resolutions, 2 denoising steps each. It is two models. A high-noise one, distilled after the MIND synthetic data was added, runs every step but the last and keeps the subject-following that data taught. A low-noise one, distilled before it, runs the last step and keeps textures from smearing. The paper reports 16 fps on a 4-GPU machine and does not name the GPU. The repo's interactive demo is validated on a single H200 and carries the note "The interactive demo is currently being debugged."

What the numbers say

All the results are on a benchmark the authors built: 145 images (83 dynamic, object-centric scenes and 62 static ones), each with 5 metric camera trajectories, so 725 videos per method. The trajectories run 528 to 1,648 frames and include closed loops that return to earlier views. Memory is scored by comparing a revisit frame with its matched first-visit frame. Camera control is scored by recovering the trajectory from the generated video with VGGT-Ω and aligning it to the commanded one. Eight recent camera-controllable world models are compared.

The paper's headline is "47.6% improvement relative to the strongest baseline". That is LPIPS: 0.487 for Lyra 2.0 against 0.255. PSNR goes from 14.050 to 18.016 dB, and MEt3R, a multi-view consistency score, from 0.334 to 0.166. On camera error the base model is first on all three metrics, though the margins there are smaller: TransErr 1.475 against Lyra's 1.538, CamMC 1.546 against 1.624.

The strongest baseline on memory is Lyra 2.0, and Lyra 2.0 reconstructs. It builds spatial memory from Depth Anything 3 depth and warps it. The other depth-based spatial memories, Alaya-EVOKE and Matrix-Game 3.5, sit mid-table. So the result is not "3D memory loses". It is that a learned memory with a view-synthesis lineage beat the best explicit one by a wide margin, on this benchmark.

And the distilled model is more consistent than the base: LPIPS 0.186 against 0.255, MEt3R 0.129 against 0.166. It is also worse on camera control, third on every camera metric. The paper does not explain the inversion. One reading, mine and untested: a model that moves the camera less than it is told, and keeps the scene stiller, has less to reconcile when it comes back.

On general video quality (VBench), WorldCrafter posts the highest overall score, by half a point:

The column I would not skip is dynamic degree. The two WorldCrafter models have the two lowest scores in the table; every baseline is between 97.087 and 100.000. The gap is small, and it is the direction you would expect from a memory that pulls each chunk toward what was already there.

expand

Some context the numbers need. The benchmark is the authors' own, and neither it nor the evaluation code is released. Every video is resized to 640×384, which is WorldCrafter's native resolution, before scoring. Baselines run as their authors ship them: full-step models for SANA-WM (with its refiner), Lyra 2.0 and HY-WorldPlay, distilled models for the rest. That is a fair choice, but it means the table compares products, not architectures.

What each piece buys

The ablations run on the base model before distillation, one change at a time, at the same memory-token budget:

Every piece earns its place, and the encoder earns the most. Swapping it for raw frames costs more than any other single change, and it costs camera accuracy as well as memory: rotation error nearly doubles, from 13.536 to 26.522. Freezing the encoder in the final stage costs much less, which says the encoder is most of the way there after its warm-up on latents, and joint training closes the rest.

expand

The cost panel is the other half of the argument for skipping the reconstruction. For depth-based spatial memory the authors measure 0.409 s of Depth Anything 3 depth and alignment plus 0.937 s of warping, 1.346 s per chunk. The memory encoder takes 0.049 s and the readout 0.013 s: 0.062 s, a 21.7× reduction. A chunk is 33 frames, just over two seconds of video at 16 fps. By my arithmetic on the paper's timings, depth-and-warp would use about two thirds of that before any denoising ran, and the memory encoder about 3%. The paper does not say what hardware timed either.

What ships

Code and weights are public. The paper does not state a parameter count; everything below is from the files.

The repository, at cff3e1c (24 September), has the diffusers pipeline, the memory encoder, the retrieval, the UCPE branch, an inference CLI and the keyboard demo, plus seven image-to-video and three text-to-video examples.

The sizes follow from shipping F32: 59.5 GB for Base, 147.2 GB for Fast. The model card's sampling budget is five high-noise steps and one low-noise step per chunk for image-to-video.

The licence is the constraint that matters most. WorldCrafter's own terms permit use "only for academic purposes" and forbid "any commercial or production purposes under any circumstances". The components it builds on are non-commercial anyway: VGGT-derived code under CC-BY-NC-4.0 and the LagerNVS-derived encoder under the FAIR Noncommercial Research License. Of the third-party components the licence file lists, only Helios-Base is Apache-2.0.

Not released: training code, the curated Open-Sora-Plan subsets, the 725-video benchmark and its evaluation scripts, and any ablation checkpoint. The headline tables can be read but not reproduced.

Where it breaks

The paper names two limits. Consistency "can still break down along particularly complex or extended trajectories". And re-encoding history at every chunk costs latency. The code confirms the second: the encoder runs from scratch on nine frames per chunk, with no state carried between chunks. The authors' proposed fix is a streaming encoder that folds each new chunk into the memory.

Three more, from reading the code, all reasoned rather than measured:

• The memory is indexed by the poses you asked for, not the ones you got. Retrieval and the encoder's camera tokens both use the commanded trajectory. Wherever the generator drifts from the command, the memory is filed slightly wrong. The camera table measures how much it drifts; nothing in the paper measures what that drift does to recall.
• The frustum test has no occlusion. A frame on the far side of a wall counts as seeing what the wall hides. In open scenes that is harmless. In a building with rooms, retrieval can pick frames that saw nothing useful.
• The memory is shaped like latents but never checked as one. memory4 is [16, 4, 48, 80] in the VAE's standardised latent space, and it came from a view renderer. Decoding it through the VAE would show whether the model learned to render a rough target view or something the DiT reads that no person could. It is the first experiment I would run with the released weights.

For a streaming world model, "3D memory" never had to mean a reconstruction. It has to mean something that can answer "what should I see from here?" quickly. WorldCrafter answers with a learned view synthesiser in latent space, chosen and queried with plain camera geometry. On the authors' benchmark, that was better and 21.7× cheaper than the depth-and-warp alternative. See also the argument for keeping world state separate from the renderer in XGEN-JING and DAO, and TencentARC's other bet on native geometry in GAE.