# WorldCrafter: skipping the reconstruction, not the 3D
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/worldcrafter
> date: 2026-09-26
> tags: world-models, video-generation, 3d, diffusion, open-weights, explainer
Walk a video world model around a courtyard and back to where you started, and
you usually arrive somewhere else. The pavilion has a different roof. The pond
moved. Nothing is wrong with any single frame; the model simply has no record
of the first visit that survives the trip.
The standard fix is to give it one in 3D: estimate depth for past frames, lift
them into a point cloud, and warp that cloud into the view you are about to
render. [WorldCrafter](https://arxiv.org/abs/2609.24984), from ARC Lab at
Tencent with Peking University, asks the obvious question. What if you skip
the reconstruction entirely?
Having read the paper and the released code, my answer is that it skips the
reconstruction but not the 3D. There is no depth map, no point cloud and no
warp at inference. But 3D reasoning enters twice. Once in how past frames are
chosen, which is a plain geometric frustum test on camera poses. And once in
what they are fed through: a memory encoder initialised from a model trained
to do novel-view synthesis. What reaches the video generator is shaped,
literally, like four latent frames for where the camera is about to be. In
effect, a learned view renderer stands where the explicit one used to.
## Why a revisit breaks
WorldCrafter is a chunked autoregressive video model. It generates nine latent
frames (33 video frames) at a time, conditioned on a short window of what came
before, then slides forward. In the paper's notation, each chunk's latent
$\mathbf{z}_t$ flows under a velocity predicted from the memory $\mathbf{M}$,
the recent frames $\mathbf{z}^{\mathrm{r}}$, the target camera trajectory
$\mathbf{C}$ and the text $\mathbf{y}$:
$$
\frac{\mathrm{d}\mathbf{z}_{t}}{\mathrm{d}t}=\mathbf{v}_{\theta}\!\left(\mathbf{z}_{t},t\mid\mathbf{M},\mathbf{z}^{\mathrm{r}},\mathbf{C},\mathbf{y}\right)
$$
Drop $\mathbf{M}$ and you have an ordinary streaming generator, of the kind
[ABot-World-0](/articles/abot-world) runs on one desktop GPU. Its recent window
is enough to keep motion continuous. It is not enough to remember a room you
left a minute ago, because that room is no longer in the window.
The paper sorts the existing fixes into three families, and each has a known
cost:
- **Context memory** puts old frames back into attention. Full history is
expensive, so methods retrieve a few frames by camera overlap. A token budget
that holds four frames holds four frames.
- **Spatial memory** reconstructs: depth, then a 3D representation, then a
render or warp into the target view, concatenated with the noise. It aligns
revisits well and, in the paper's words, "can overconstrain scene dynamics".
It is also only as good as its depth.
- **Implicit memory** compresses history into a learned state. The recent
geometry-aware variants borrow features from VGGT, which the authors argue
keeps geometry and loses the appearance you need to redraw what you saw.
WorldCrafter is in the third family, with a different donor. It takes its
encoder from LagerNVS, a model trained to synthesise novel views, on the
argument that reconstructing an unseen view forces a representation to keep
geometry *and* appearance. That is the whole bet.
## The window, counted
Before the mechanism, the budget it has to live in. WorldCrafter's DiT is
initialised from Helios-base, and its config is the familiar Wan-shaped one:
40 layers, 40 heads of 128, a 5,120-wide residual stream. Helios's window held
a nine-frame noisy chunk plus a FramePack-style clean history: a compressed
16-frame segment, a 2-frame segment, the latest frame and an attention-sink
frame. WorldCrafter deletes the 16-frame segment and puts memory tokens in its
place, "equivalent in number to the tokens of 4 uncompressed history frames."
Counted from the shipped files, that is this:
A 384×640 frame is a 48×80 latent; at 1×2×2 patches that is 960 tokens per
latent frame. The window is 3,840 memory tokens, 240 tokens of mid history
(two frames at 2×4×4 patches), 1,920 for the sink and the latest frame, and
8,640 for the chunk being denoised: 14,640 in all, with memory at 26%. Those
counts are my arithmetic from `transformer/config.json` and the safetensors
shapes of `patch_memory`, `patch_mid` and `patch_short`, not figures the paper
states.
Two details in the code matter. The memory arrives shaped exactly like four
latent frames, `[16, 4, 48, 80]`, and enters through its own patch embedding
that is initialised from the clean-history one, so the DiT first sees memory as
if it were four more clean frames. And only the noisy chunk gets the camera
branch; memory and history run at timestep zero and carry no camera injection.
The ablation that matters most keeps this slot fixed and changes only what
fills it. Put four retrieved raw latent frames there and revisit LPIPS is
0.497. Put the memory encoder's output there and it is 0.255. Same tokens,
same training schedule, half the error. Everything below is about what the
encoder does with its 3,840 tokens that four raw frames cannot.
## Step one: pick the past by coverage, not similarity
The memory encoder takes a fixed nine latent frames. One is always the latest
frame. The other eight come from the entire history, which is never thrown
away; only the encoder's input is bounded.
The usual way to pick is to rank past frames by how much their field of view
overlaps the target and take the top few. WorldCrafter's
`select_trajectory_fov_history` does something more careful. It takes the four
poses the next chunk will render (latent slots 2, 4, 6 and 8), samples each
one's viewing frustum as a 10×10×10 grid of points from 0.1 m to 30 m, and
marks which grid points each past frame's frustum contains. Then it picks
greedily:
```python
# worldcrafter/repencoder/trajectory_fov.py (trimmed; bookkeeping removed)
covered = anchor_fov.clone() # the latest frame is always in
for slot in range(budget): # budget = 8
resulting = covered.unsqueeze(0) | candidate_fov
full_fairness = torch.sort(resulting.float().mean(dim=2), dim=1).values
connected = (candidate_fov & covered.unsqueeze(0)).sum(dim=(1, 2)) > 0
rows = torch.cat((full_fairness, frontier_ratio[:, -1:], frontier_fairness,
new_fov_total[:, None], new_frontier_total[:, None]), dim=1)
eligible = available & connected
if not eligible.any():
eligible = available # "disconnected fill"
chosen = _pick_best(rows, available=eligible, stable_values=candidates)
covered = resulting[chosen]
```
The key is lexicographic, and its first four entries are the per-target
coverage sorted ascending. So each pick is the frame that most raises the
*worst-covered* upcoming view, then the next worst, and so on. After that come
"frontier" terms: the part of each target view that neither the latest frame
nor an earlier target in the same chunk can see, which is the part memory has
to supply. Candidates must share covered points with what is already chosen
unless none do, and ties go to the more recent frame.
I ported that function to a top-down toy so you can watch it choose:
On the last chunk of the toy walk, the camera turns back toward the pavilion
from where it started. Similarity ranking spends all eight slots on the frames
of the last two chunks. They overlap the upcoming views the most, and they are
near-copies of each other. Max-coverage takes none of them beyond the latest
frame, which is always in. It reaches back to the first two chunks, which faced
the pavilion, and fills the rest from four other parts of the walk. In the toy,
the union goes from 67% to 100% of the sample points. Those percentages are
my toy's, not the model's. The measured version is the paper's ablation, below:
swapping max-coverage for similarity ranking moves revisit LPIPS from 0.255 to
0.296.
One thing the code does not do is check occlusion. A point counts as seen if
it falls inside a past frame's frustum, even with a wall in between. And the
frustum is a fixed 100° by 71.13° from 0.1 m to 30 m, which matches the
default camera but not every scene.
## Step two: an encoder that learned 3D from view synthesis
The nine chosen latents go into a memory encoder the code calls `RepEncoder`.
Its architecture and weights come from the LagerNVS encoder, with one surgical
change: the image-processing front end is gone. A new `InputLayer` takes each
16×48×80 latent, un-standardises it, resizes it to 44×74 and runs a stride-2
convolution to a 22×37 grid of 1,024-wide tokens. That grid enters DINO at its
third block: the patch embedding and the first two blocks are dropped, and the
remaining 22 are kept. The paper's Figure 2 still labels the box "DINO
layers", which is consistent with that: the shallow ones are replaced, the
deep ones stay.
After DINO comes a VGGT-style backbone that alternates per-frame and
cross-frame attention, with each frame's camera pose (expressed relative to the
latest frame) entering as a camera token. The output is a scene representation
of 814 tokens per source frame, 768 wide: 7,326 tokens for the nine frames,
holding geometry and appearance without ever becoming a point cloud.
The parameter census, summed from the safetensors header of
`repencoder/model.safetensors`:
| part | what it is | parameters |
|---|---|---|
| `input_layer` | latent patch embedding, new | 148,480 |
| `dino_tail` | 22 of DINO's 24 blocks, 1,024 wide | 300,237,824 |
| `vggt` | frame and global attention with camera tokens | 658,085,056 |
| `repfeature` | pose-guided readout, 768 wide | 182,325,376 |
| `output_layer` | 3×3×3 conv to 16 latent channels | 331,792 |
| **total** | | **1,141,128,528** |
That total matches the `parameter_count` the Fast package records in
`repencoder_frozen.json`. 97,690,304 of it are unmerged rank-64 LoRA factors,
which suggests the encoder was adapted with low-rank updates rather than fully
fine-tuned. The paper does not say which; the file layout does. For scale,
the DiT it feeds is 14,291,783,744 parameters.
## Step three: read the memory out where you are going
The representation has 7,326 tokens and the DiT has room for 3,840. Something
has to choose what survives. The paper compares two ways.
**Pose-free readout** maps the whole representation to 3,840 tokens and leaves
the DiT's attention to find what is relevant. **Pose-guided readout** asks the
representation a question: what would the scene look like from these four
poses? WorldCrafter ships the second.
In code, the four query poses are the same slots 2, 4, 6 and 8 of the
upcoming chunk. Each becomes a six-channel ray map at 384×640, cut into 8×8
patches: 48×80 = 3,840 query tokens per view. Nine bidirectional blocks
cross-attend queries and scene tokens in both directions, a final block updates
the queries, and a 3×3×3 convolution maps the result to 16 channels:
```python
# worldcrafter/repencoder/model.py, forward_conditioned
input_features = self.input_layer(source_latents) # [B, 9, 814, 1024]
dino_features = self.dino_tail(input_features)
scene = self.vggt(dino_features, source_camera_tokens) # [B, 9, 814, 768]
target_features = self.repfeature(scene, target_rays) # 4 target views
memory4 = self.output_layer(target_features) # [B, 16, 4, 48, 80]
```
The readout is initialised from LagerNVS's shallow decoder layers, the part of
LagerNVS that decodes a target view. So the memory is four
latent-frame-shaped tensors, one per upcoming pose, produced by a network that
began life as a view renderer. The paper is careful to say this conditions the
DiT "without reconstructing target-view images", and nothing in the released
code decodes `memory4` or supervises it as an image. But its shape and lineage
say what it is for.
The ablation says it matters. With the same inputs and the same token budget,
pose-free readout gives revisit LPIPS 0.333 and rotation error 18.307;
pose-guided gives 0.255 and 13.536. The authors attribute this to "a more
effective allocation of the fixed memory budget to target-relevant
information." A readout that knows where you are going can spend its tokens on
what you will see.
## Camera control, and getting to 16 fps
Camera control is a separate branch. The target trajectory gives a
camera-to-world pose and intrinsics per frame. Following PRoPE, relative camera
geometry enters self-attention as a positional transform, implemented as the
parallel camera-attention branch from UCPE: its own query, key and value
projections, added to the original attention through a zero-initialised
projection. The Base package's `camera_adapter.pth` holds 400 BF16 tensors,
ten per block across the 40 blocks, 1,050,777,600 bytes; at two bytes each
that is about 525M parameters. That figure is my division, not a stated count.
Keyboard control compiles to that trajectory. The inference guide's actions
are `forward1`, `yaw_left30`, `pitch_up15` and so on, one action per 33-frame
chunk, plus `reverseN`, which retraces the preceding N chunks. That is the
revisit primitive, as a command.
Training is four stages, all on data whose camera poses come from Depth
Anything 3 and whose captions come from Qwen2.5-VL:
| stage | trains | data | iterations | GPUs, batch |
|---|---|---|---|---|
| 1 | DiT, adapted to the new window | 760,000 Open-Sora-Plan videos | 5,000 | 32, 32 |
| 2 | UCPE camera branch, DiT frozen | 40,000 filtered OSP + 6,000 DL3DV | not stated | 32, 128 |
| 3 | memory encoder, on VAE latents | DL3DV + filtered OSP | 5,000 | 16, 16 |
| 4 | readout, encoder, DiT and camera jointly | DL3DV + filtered OSP, then MIND synthetic | 8,000 + 1,000 | 32, 32 |
So "no depth" is true at inference and not in training: a depth model produced
the pose labels the whole system learned from.
The real-time model, WorldCrafter-fast, is distilled with distribution
matching over a coarse-to-fine pyramid: 3 spatial resolutions, 2 denoising
steps each. It is two models. A high-noise one, distilled after the MIND
synthetic data was added, runs every step but the last and keeps the
subject-following that data taught. A low-noise one, distilled before it, runs
the last step and keeps textures from smearing. The paper reports 16 fps on a
4-GPU machine and does not name the GPU. The repo's interactive demo is validated on a single H200 and
carries the note "The interactive demo is currently being debugged."
## What the numbers say
All the results are on a benchmark the authors built: 145 images (83 dynamic,
object-centric scenes and 62 static ones), each with 5 metric camera
trajectories, so 725 videos per method. The trajectories run 528 to 1,648
frames and include closed loops that return to earlier views. Memory is scored
by comparing a revisit frame with its matched first-visit frame. Camera
control is scored by recovering the trajectory from the generated video with
VGGT-Ω and aligning it to the commanded one. Eight recent camera-controllable
world models are compared.
The paper's headline is "47.6% improvement relative to the strongest
baseline". That is LPIPS: 0.487 for Lyra 2.0 against 0.255. PSNR goes from
14.050 to 18.016 dB, and MEt3R, a multi-view consistency score, from 0.334 to
0.166. On camera error the base model is first on all three metrics, though the
margins there are smaller: TransErr 1.475 against Lyra's 1.538, CamMC 1.546
against 1.624.
The strongest baseline on memory is Lyra 2.0, and Lyra 2.0 reconstructs. It
builds spatial memory from Depth Anything 3 depth and warps it. The other
depth-based spatial memories, Alaya-EVOKE and Matrix-Game 3.5, sit mid-table.
So the result is not "3D memory loses". It is that a learned memory with a
view-synthesis lineage beat the best explicit one by a wide margin, on this
benchmark.
And the distilled model is *more* consistent than the base: LPIPS 0.186 against
0.255, MEt3R 0.129 against 0.166. It is also worse on camera control, third on
every camera metric. The paper does not explain the inversion. One reading,
mine and untested: a model that moves the camera less than it is told, and
keeps the scene stiller, has less to reconcile when it comes back.
On general video quality (VBench), WorldCrafter posts the highest overall
score, by half a point:
| | subject consistency | background consistency | dynamic degree | overall |
|---|---|---|---|---|
| WorldCrafter | 82.695 | 90.589 | 96.893 | 81.910 |
| WorldCrafter-fast | 81.365 | 89.559 | 96.505 | 80.285 |
| Alaya-EVOKE | 80.522 | 88.994 | 99.029 | 81.406 |
| SANA-WM | 79.413 | 89.625 | 98.835 | 80.841 |
The column I would not skip is dynamic degree. The two WorldCrafter models have
the two lowest scores in the table; every baseline is between 97.087 and
100.000. The gap is small, and it is the direction you would expect from a
memory that pulls each chunk toward what was already there.
Some context the numbers need. The benchmark is the authors' own, and neither
it nor the evaluation code is released. Every video is resized to 640×384,
which is WorldCrafter's native resolution, before scoring. Baselines run as
their authors ship them: full-step models for SANA-WM (with its refiner),
Lyra 2.0 and HY-WorldPlay, distilled models for the rest. That is a fair
choice, but it means the table compares products, not architectures.
## What each piece buys
The ablations run on the base model before distillation, one change at a time,
at the same memory-token budget:
| variant | MEt3R | LPIPS | PSNR | RotErr | CamMC |
|---|---|---|---|---|---|
| context memory, 4 raw frames | 0.382 | 0.497 | 13.907 | 26.522 | 2.150 |
| frozen memory encoder | 0.227 | 0.305 | 16.873 | 15.428 | 1.886 |
| pose-free readout | 0.251 | 0.333 | 16.486 | 18.307 | 1.828 |
| similarity-based retrieval | 0.213 | 0.296 | 17.125 | 14.657 | 1.664 |
| **WorldCrafter** | **0.166** | **0.255** | **18.016** | **13.536** | **1.546** |
Every piece earns its place, and the encoder earns the most. Swapping it for
raw frames costs more than any other single change, and it costs camera
accuracy as well as memory: rotation error nearly doubles, from 13.536 to
26.522. Freezing the encoder in the final stage costs much less, which says
the encoder is most of the way there after its warm-up on latents, and joint
training closes the rest.
The cost panel is the other half of the argument for skipping the
reconstruction. For depth-based spatial memory the authors measure 0.409 s of
Depth Anything 3 depth and alignment plus 0.937 s of warping, 1.346 s per
chunk. The memory encoder takes 0.049 s and the readout 0.013 s: 0.062 s, a
21.7× reduction. A chunk is 33 frames, just over two seconds of video at
16 fps. By my arithmetic on the paper's timings, depth-and-warp would use about
two thirds of that before any denoising ran, and the memory encoder about 3%.
The paper does not say what hardware timed either.
## What ships
Code and weights are public. The paper does not state a parameter count;
everything below is from the files.
The [repository](https://github.com/TencentARC/WorldCrafter), at `cff3e1c`
(24 September), has the diffusers pipeline, the memory encoder, the retrieval,
the UCPE branch, an inference CLI and the keyboard demo, plus seven
image-to-video and three text-to-video examples.
The sizes follow from shipping F32: 59.5 GB for Base, 147.2 GB for Fast. The
model card's sampling budget is five high-noise steps and one low-noise step
per chunk for image-to-video.
The licence is the constraint that matters most. WorldCrafter's own terms
permit use "only for academic purposes" and forbid "any commercial or
production purposes under any circumstances". The components it builds on are
non-commercial anyway: VGGT-derived code under CC-BY-NC-4.0 and the
LagerNVS-derived encoder under the FAIR Noncommercial Research License. Of the
third-party components the licence file lists, only Helios-Base is
Apache-2.0.
Not released: training code, the curated Open-Sora-Plan subsets, the 725-video
benchmark and its evaluation scripts, and any ablation checkpoint. The
headline tables can be read but not reproduced.
## Where it breaks
The paper names two limits. Consistency "can still break down along
particularly complex or extended trajectories". And re-encoding history at
every chunk costs latency. The code confirms the second: the encoder runs from
scratch on nine frames per chunk, with no state carried between chunks. The
authors' proposed fix is a streaming encoder that folds each new chunk into the
memory.
Three more, from reading the code, all reasoned rather than measured:
- **The memory is indexed by the poses you asked for, not the ones you got.**
Retrieval and the encoder's camera tokens both use the commanded trajectory.
Wherever the generator drifts from the command, the memory is filed slightly
wrong. The camera table measures how much it drifts; nothing in the paper
measures what that drift does to recall.
- **The frustum test has no occlusion.** A frame on the far side of a wall
counts as seeing what the wall hides. In open scenes that is harmless. In a
building with rooms, retrieval can pick frames that saw nothing useful.
- **The memory is shaped like latents but never checked as one.** `memory4` is
`[16, 4, 48, 80]` in the VAE's standardised latent space, and it came from a
view renderer. Decoding it through the VAE would show whether the model
learned to render a rough target view or something the DiT reads that no
person could. It is the first experiment I would run with the released
weights.
For a streaming world model, "3D memory" never had to mean a
reconstruction. It has to mean something that can answer
"what should I see from here?" quickly. WorldCrafter answers with a learned
view synthesiser in latent space, chosen and queried with plain camera
geometry. On the authors' benchmark, that was better and 21.7× cheaper than
the depth-and-warp alternative. See also the argument for keeping world state
separate from the renderer in [XGEN-JING and DAO](/articles/xgen-jing-dao), and
TencentARC's other bet on native geometry in
[GAE](/articles/gae-geometry-native).
Read from `repencoder/manifest.json` (`output_shape [16, 4, 48, 80]`,
`target_slots [2, 4, 6, 8]`), the `patch_memory`, `patch_mid` and `patch_short`
kernel shapes in the Base safetensors headers, and the window split in
`diffusers/pipeline.py`. The token total is my arithmetic. A pipeline path that
feeds the memory at another shape or patch size would overturn it.
From `select_trajectory_fov_history` in `trajectory_fov.py` at `cff3e1c`: the
lexicographic key leads with per-target coverage sorted ascending, and
visibility is frustum containment only. A later commit that adds a depth or
visibility test, or a different retrieval path used at inference, changes it.
Summed from safetensors headers fetched by HTTP range request, and matched
against `parameter_count` in `adapter_high_noise/repencoder_frozen.json`. The
header sum is 32 higher because it includes the input layer's latent mean and
standard deviation buffers. Loading both modules and summing `p.numel()` would
settle it.
The benchmark, trajectories and evaluation code are unreleased, so every
number in the results section is reported, not measured. A third-party run on
a public revisit benchmark, with Lyra 2.0 on the same trajectories, would
confirm or shrink the 47.6% LPIPS margin.
From Figures 4 and 5: fast wins all four memory metrics and is third on all
three camera metrics. My guess, that under-following the camera makes revisits
easier to match, is untested. A per-video correlation between camera error and
revisit LPIPS would test it.