Dream-Cubed: diffusion directly on Minecraft's block IDs, where inpainting comes free

2026-08-03 · 12 min · diffusion · generative-models · minecraft · world-models · 3d

Most generative work on 3D game worlds goes through a pixel renderer or a learned latent space before it touches anything the game itself understands. Dream-Cubed skips both. It trains a diffusion model directly on Minecraft's own vocabulary — integer block IDs, the same representation the game engine uses — and gets a specific, useful property for free as a result: place a block by hand anywhere in a chunk, and the model will build around it exactly, with no special training and no extra inference-time machinery. The paper is a three-month-old preprint from a team at NYU and Sakana AI, posted with no peer review and, as of this writing, no citations. It also lands in a niche that's suddenly crowded — the paper names two concurrent Minecraft/voxel generators of its own accord — and its own limitations section is unusually candid about how little its evaluation actually proves. All of that is worth knowing before the mechanism, which is genuinely worth understanding.

Blocks as tokens, not pixels

Each training sample is a $32 \times 32 \times 32$ tensor of integer block IDs — dirt, water, stone, whatever the game placed there — over a vocabulary of 117 block types in the core procedurally-generated set, extended to 177 once six professionally human- authored maps are folded in. The dataset totals 1,667,781 procedural chunks plus 358,762 human-authored ones: 2,026,543 chunks, tens of billions of tokens at $32^3 = 32{,}768$ voxels per chunk, spanning fifteen biomes from ocean to village to cave.

One backbone serves both diffusion families the paper compares: a 280M-parameter 3D Diffusion Transformer, 25 blocks, hidden dimension 768, 8 attention heads. A 3D convolution patchifies each chunk into non-overlapping voxel patches; fixed 3D sine-cosine position embeddings and a biome-label embedding condition every block through AdaLN modulation, the same conditioning mechanism DiT uses for timestep and class in image diffusion.

Two diffusion families, one backbone

Masked discrete diffusion (MD4). Add a [MASK] token to the block vocabulary. The forward process independently masks each voxel with probability $p_{\text{mask}}(t) = \sin(\pi t / 2)$ for $t \sim \mathrm{Uniform}(0,1)$ — at $t{=}0$ nothing is masked, at $t{=}1$ everything is. The network sees the corrupted chunk and predicts the original block ID at every masked position, trained with cross-entropy on masked positions only. Sampling starts from an all-[MASK] chunk and iteratively unmasks positions over a fixed number of steps.

Continuous diffusion (DDPM), in an embedding space. Every block name — "dirt", "sand" — is embedded once via OpenAI's text-embedding-3-small, giving a frozen 16- dimensional lookup table with a semantic prior baked in for free. A standard cosine noise schedule runs $x_t = a_t x_0 + b_t \varepsilon$ over 1000 steps, trained with v- prediction. At the end of sampling, the continuous output is decoded back to discrete block IDs by nearest-neighbor lookup against the embedding table.

Both are trained on the identical backbone, the identical data, the identical compute budget — the paper's stated goal is a controlled, apples-to-apples comparison of the two diffusion formulations, not a fight either one is rigged to win.

Why inpainting is free under one formulation and not the other

This is the mechanism worth sitting with. MD4's forward process decides, independently, per voxel, whether that voxel gets masked. A voxel the user has placed by hand simply isn't a candidate for that decision — it's excluded from the start, at every step, all the way through sampling. There is no moment where the model has to reconcile "what I was trained to expect here" with "what's actually here," because the fixed voxel was never part of the corruption process to begin with.

masked diffusion inpainting — fixed blocks vs. resolving blocks

denoising step4 / 10

user-fixed blocks 8 — never maskedgenerated blocks resolved 21/52

The path cells (thick outline) were never part of the corruption process — MD4 masks each voxel independently, and a user-fixed voxel simply is never sampled for masking, at any step. So it holds its true block from step 0 through step 10, no re-noising, no clamping, no extra machinery. Drag the step slider: the dotted cells are still [MASK] and resolve nearest the fixed path first, but the path itself never so much as flickers.

A continuous DDPM cannot get the same thing for the same reason. Its forward process adds Gaussian noise to every position, at every step, following one global schedule — clean voxels aren't a special case the network was ever trained to see. If you clamp a user- placed block to its clean embedding partway through sampling — the natural thing to try — the model is still conditioned on that position carrying noise amplitude $b(t)$ at timestep $t$ , and it sees zero instead. That's a real mismatch between what training taught the network to expect and what inference is handing it, not a cosmetic one.

holding one voxel fixed, two ways

process position t0.55

P(mask) at the fixed voxel

0.00

P(mask), an ordinary free voxel

0.76

schedule mismatch at the fixed voxel: 0.00 — nothing to reconcile

MD4 masks each voxel independently, so a user-fixed voxel just has its masking probability locked to 0, for every t — it was never a candidate for corruption. An ordinary free voxel still follows the real schedule, p_mask(t) = sin(πt/2), climbing toward 1 as t → 1. There is no conflict between the two, at any point in the process — that is the entire reason inpainting falls out for free.

The paper is upfront that it doesn't solve this: closing that gap needs extra machinery — RePaint-style repeated re-noising and re-sampling — which it flags in an appendix as an unresolved comparison point, not a capability it demonstrates for the DDPM side. Exact conditioning is what falls out of the masked formulation for nothing; it's what the continuous one would have to be re-engineered to approximate.

Five columns of paired images: each pair shows a small hand-authored block pattern (a ring, a zigzag path, a lake outline, an arch with floating platforms, a volcano cone with lava and debris) next to a full rendered Minecraft chunk that incorporates it — a moat around a hill, a winding path through grass, a lake surrounded by mountains, a waterfall through ruins, and a gray stone volcano with an orange crater. Three rows show variations of each pattern type. — User-authored block patterns (left of each pair) held fixed while the model fills in a coherent chunk around them — rings become moats, zigzags become paths, a small lava-and-debris seed becomes a volcano (Merino et al., 2026, Figure 6).

If you've read the piece on iLLaDA on this site, the mechanism will look familiar: masked discrete diffusion over text tokens is the same "absorbing-state" idea MD4 applies here to voxels — a masking probability schedule, a network trained to fill in exactly the masked positions, bidirectional context by construction. iLLaDA's own masking ratio is closer to a straight linear schedule ( $t$ itself, roughly); Dream-Cubed's MD4 uses the $\sin(\pi t/2)$ reparameterization from Shi et al.'s original MD4 paper — a detail, not a different mechanism. What changes here is the alphabet the diffusion runs over: block IDs instead of vocabulary tokens, arranged on a 3D grid instead of a 1D sequence. Same masking idea, different token space — see also the masked-diffusion-lm entry in the architecture map for where this sits relative to the wider non-autoregressive-LM family.

Outpainting is the same trick, tiled

Generating a world larger than one $32^3$ chunk uses a sliding window: partition the larger canvas into overlapping cells, generate them in sequence, and for every cell after the first, treat the already-generated overlap with its neighbors as more fixed context — recursively the same "these voxels are excluded from masking" trick, just applied at world scale instead of one seeded pattern.

Three isometric Minecraft world renders side by side, each roughly a 5 by 5 grid of stitched chunks. Left, labeled Unconditional World: a mixed landscape of hills, a village, forest, and several small ponds. Middle, labeled Biome-conditioned world: a world split between a desert on the left and a snowy taiga forest on the right, with a lake at the boundary. Right, labeled Block-conditioned world: a landscape built around a user-placed ring of water encircling a small grass island, with a volcano rising in the background. — 5×5-chunk outpainted worlds: unconditional, biome-conditioned, and block-conditioned (the ring-and-island on the right is the same free conditioning trick, at world scale) (Merino et al., 2026, Figure 7).

The cost of this is real and disclosed: a single 5×5 outpainted world takes over an hour of H100 inference time, generated cell by cell, sequentially — the paper calls inference speed "a practical barrier to all envisioned applications," not a solved problem.

What the numbers actually say

MD4 and DDPM land in a statistical tie on the paper's own metric. Adjusted FID (generated minus a reference FID from held-out chunks) averages 59.26 for MD4 at patch size 2 versus 59.29 for DDPM at the same patch size — indistinguishable overall, with MD4 winning 9 of 15 biomes and DDPM winning 6.

Patch size is where the two formulations actually separate. MD4 holds up at patch sizes 2 (4,096 tokens per chunk) and 4 (512 tokens), with visible artifacts only at patch 8; DDPM works at patch 2 but fails outright at patch 4 under the identical configuration. That's the one place in the paper where discrete and continuous diffusion give clearly different answers, and it favors the discrete side.

Naive frequency matching doesn't work for rare, structured content. Three data mixtures were compared: a balanced split, natural biome frequency, and a village-boosted split. Natural frequency wins on average FID — but ocean chunks, over-represented 5.3× relative to balanced, and forest, at 1.7×, improve, while village and cave, both rare and structurally complex, get worse. Boosting village samples specifically recovers the village-biome losses. The honest reading: matching real-world frequency is not automatically the right training mixture once some categories are both rare and hard.

The human study is small and its own authors say so: 19 Minecraft-experienced participants (all from the authors' own institution), roughly 1,000 two-alternative forced-choice trials, free pan/zoom/rotate:

Human preference — win rate over real chunks (%)

MD4 (patch 2)

67.1%

MD4 (patch 4)

57.1%

DDPM (patch 2)

55.2%

020406080

Both MD4 configurations beat real chunks at statistical significance (patch 2: p less than 0.001, patch 4: p = 0.042) — a result the authors attribute candidly to classifier-free guidance pushing generated samples toward a "prototypical" idealized biome, more uniform than messy real terrain, rather than claiming their model has somehow out-built reality. MD4 (patch 2) and DDPM (patch 2) tie head-to-head at 49.4%, consistent with the FID result above.

Min FID gap between two models	Agreement with human preference	n	p
0	54.3%	512	0.029
5	61.7%	227	less than 0.001
10	62.9%	159	less than 0.001
15	66.1%	109	less than 0.001

Agreement between FID and human raters rises with the size of the FID gap being compared, but tops out at 66.1% even at the largest gap tested — a coin flip with a thumb on the scale, offered by the authors themselves as evidence that FID is only weakly informative here, and only at large gaps.

warning

The evaluation the whole paper hangs its numbers on has real, self-acknowledged holes. FID is a render-based metric: it cannot see building interiors, cave systems, or whether a door is actually reachable — none of the things that make a Minecraft structure functional rather than merely picturesque. It's computed from only 1,500 rendered images per model, and costs roughly 60 GPU-hours to run — expensive enough that the authors say it can't be used for model selection during training, only for a final after-the-fact score. And the human study, small as it is, evaluated only biome- conditioned generation. It never tested the inpainting or outpainting capability the paper actually leads with — the figures above are demonstrations, not measured results. The dataset itself is drawn from Minecraft version 1.12.2 (2017), for tooling compatibility, so newer blocks and biomes aren't represented at all.

Training cost is disclosed cleanly: 4×H100 GPUs, classifier-free guidance with 20% label dropout during training and a guidance scale of 4.0 at inference, patch-2 models run 20 epochs and patch-4 models 160 (matched for equal token exposure), roughly 192 GPU-hours total across every model in the paper. Inference is the bottleneck end of the system: about 2.5 minutes per chunk at patch 2, 25 seconds at patch 4.

A crowded moment, honestly disclosed

Dream-Cubed cites its own competition directly rather than presenting itself as singular: Scaffold Diffusion (a NeurIPS 2025 workshop paper that conditions on an input occupancy scaffold instead of generating from nothing) and PERSIST (arXiv 2603.03482, roughly a month earlier, which uses a 3D DiT with rectified flow matching but as one component inside a video-generation system, not a standalone generator) are both named as concurrent work on the same general problem, in the same few months of 2026. Solaris (arXiv 2602.22208) is cited as another concurrent voxel/world-modeling effort in the same window. None of WorldGAN, Scaffold Diffusion, or XCube — the paper's narrative comparison points — are actually benchmarked against Dream-Cubed's FID or human-preference numbers on shared ground; the positioning against prior work is qualitative throughout, and there is no table anywhere in the paper showing Dream-Cubed beating a previously published Minecraft or voxel generator on a metric both were scored on. For a reader wanting a settled state-of-the-art claim, that table doesn't exist yet.

Code, data, and all pretrained models are released (github.com/SakanaAI/DreamCubed), which is worth crediting on its own — a preprint this new, this openly scored against its own limitations, and this fully released, is a reasonable way to publish work you don't yet have citations to back up.

The take

The technical point is narrow and real: masked discrete diffusion turns user-block- conditioning, inpainting, and outpainting into a structural guarantee — unmasked voxels were never part of the corruption process, so they can't drift — while the equivalent constraint on a continuous DDPM has to be bolted on after the fact, and the paper is explicit that it doesn't fully solve that side. Working directly in block-ID space, skipping pixels and learned latents entirely, is what makes that guarantee possible in the first place. Everything past that point is evidence you should discount appropriately: FID and DDPM come out statistically tied on the paper's own numbers, the human study that exists didn't test the paper's headline capability, and the field around this exact problem got crowded within the same few months this was written. Read Dream-Cubed for the mechanism and the pictures it produces — both hold up on inspection — and treat the quantitative claims as a first data point from one preprint, not a result that's been through the wringer yet.

Built on Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes (Tim Merino, Sam Earle, Ryunosuke Iwai, Julian Togelius, Edoardo Cetin; NYU / Sakana AI, preprint, April 2026) and its code and data release. Figures are the paper's own Figures 6 and 7, reproduced for commentary. Tables and numbers are the authors' except where marked as this site's own illustrative simplification (the block-grid and schedule-mismatch demos use hand-picked, not trained, values); interactive diagrams are mine.