Matrix-Game 4.0: a 5B video model re-skins a low-poly game at 60 fps

mdjsonmcpcopy↗ai

2026-10-15 · 18 min · world-models · video-generation · realtime · kv-cache · quantization · inference

Why read this

Notable· top 60%

Reads Matrix-Game 4.0's runtime and headers: the engine owns the game, the model only repaints; 60 fps, 83.4 fps and 112 ms reconciled from one 48 ms chunk.

• Original analysis
• Runs on a consumer GPU
• Widely used

·Image & video generation·Apache-2.0·Practitioner model

How this was scored

Is it new?

2 of 3: A real new idea, method or capability

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

1 of 3: Relevant for months

Does it affect many?

2 of 3: A widely used model, tool or lab release

Only here?

2 of 3: A teardown or measurement few others did

Score 66 of 100, ranked 193 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

Skywork's announcement for Matrix-Game 4.0 packs a lot into one post: "the world's first open-source video model game that runs in real time on a single consumer GPU", "480p at 60 FPS", "112ms end-to-end response", "up to 83.4 FPS generation", a single RTX 5090, and a loop that reads "Player input → Game engine updates the world → Video model renders the next frame → Player interacts again".

60 and 83.4 are both frame rates and 112 ms is a latency, and I wanted to know which pipeline produces all three at once. So I read the code in SkyworkAI/Matrix-Game (the Matrix-Game-4/ directory, at the commit of release day), the configs on Skywork/Matrix-Game-4.0, and the safetensors headers of the three scene checkpoints, which I range-read without downloading the 10 GB files. There is no paper yet; the project page says "soon". Every number below is labelled measured (I read or computed it from a file), reported (Skywork's figure, not re-run) or reasoned (my arithmetic on the other two). I did not run the model.

The game is code, the model is a renderer

The most important design decision is in the first line of docs/architecture.md:

native game -> clean RGB + frame metadata -> causal renderer -> bounded display
publisher -> WebRTC -> presented-frame HUD/audio

The game is a native C++ program (game/src/ for the two shooters, game/racing/src/ for the racer) with Jolt physics, collision, NPC encounters, a map, a HUD and sound. The shooters step physics at 120 Hz, and on every captured frame the game renders the world with OpenGL in flat low-poly colours: buildings as tinted boxes, characters as blocks, roads as grey planes. That frame is the model's input. The model card says it directly: "An executable game determines gameplay, physics and input, while the model re-renders clean low-poly RGB." The Hugging Face pipeline tag is video-to-video.

So the player's keystrokes never reach the network (measured: no action tensor exists anywhere in the inference path; CastleStream.push takes RGB frames and nothing else). You press W, the engine moves the character, the next low-poly frame shows the character further along, and that is the only way the model learns you pressed anything. The HUD, crosshair, minimap and audio are drawn on top of the model's output by the browser, matched to the frame being presented, and "UI pixels must never enter the model" (docs/offline.md).

expand

This is the same split NVIDIA used in StreamRender-H3, three days earlier, where a three.js racer sends a 14-colour semantic map and a 33B model paints it on eight GB200s. Matrix-Game sends plain shaded RGB rather than class colours, and it runs a model about a sixth the size on one card. It is also the opposite of the action-conditioned world models this site has covered, like ABot-World-0 and Astronex-World, where the network is told "move forward" and must invent what comes next. Here nothing has to be invented about where things are. Collisions, hit detection and the score live in C++, and a hallucinated wall cannot stop you, because "Neural output does not change collision or gameplay" (model card).

The project page puts it well: in pixel-only world models, "once something leaves the view it is gone, and game rules have to be re-inferred from every frame." The engine owns the state instead. What is left for the network is hard but well-posed: make this proxy look like a AAA game, consistently, four frames at a time.

expand

What the model actually sees

Each scene ships as its own checkpoint (fps-castle/, fps-egypt/, driving_game/), and each is a fine-tune of Wan2.2-Fun-5B-Control, a diffusion transformer that takes a control video alongside its noise. I compared the headers:

(measured, from the safetensors headers; the parameter count matches model.json and the model card.) Every shape is identical. Matrix-Game dropped the base model's reference-image convolution and kept the rest, so its "reference frames" are not the base model's image channel; they are KV entries, below.

The 148 input channels are the interesting part. A comment in inference/wrappers.py spells out the layout: 48 noisy latent channels, 48 control channels, 48 inpainting channels that are always zero, and 4 mask channels that are always zero. The control channels are the low-poly frames passed through the codec. Wan2.2's VAE compresses 16× spatially and 4× in time, so a 480×864 frame becomes a 30×54 grid of 48-channel latents, and four RGB frames become one latent frame. After the (1, 2, 2) patch embedding that is 15×27 = 405 tokens per latent frame (measured, FRAME_TOKENS = 15 * 27 in rtx5090/stream.py).

Text comes in by cross-attention from a umT5-XXL encoder: a fixed scene prompt for the shooters ("a third-person shooter game in an ancient Egyptian desert city with a great pyramid…"), and for racing optionally a caption the engine writes per chunk from what is in view. Those captions come from a catalog the engine ships (racing/assets/caption-catalog.json), not from a language model.

One chunk at a time

The model is causal over latent frames. Each step generates one latent, which decodes to four RGB frames (one, for the very first chunk). The attention window is 21 latent frames including a "sink" at position zero (measured, kv_window_latents: 21 in every inference.json). For the shooters the sink is a reference image of the scene, encoded once and written into the cache but never shown; for racing it is shown as frame zero. Every new chunk attends to the sink, the 20 most recent chunks, and itself: 405 queries against 8,505 keys (reasoned: 21 × 405).

rtx5090/stream.py lays out what happens per chunk, and its docstring is a good summary:

Per chunk: encode the control frames, denoise, decode, hand the frames out, then commit the clean latent to the KV history.

In code, with the rtx5090 preset:

1. Encode. The four low-poly frames go through TAEHV, a small streaming autoencoder that lives in Wan2.2's latent space (base/taew2_2.pth, 22.9 MB measured), instead of the 2.8 GB Wan VAE the quality preset uses. Result: one 48×30×54 control latent.
2. Denoise. Start from Gaussian noise, run the DiT once at timestep 1000 with the control latent in channels 48 to 95, and take the predicted clean latent. CFG is 1, so there is no second unconditional pass. One step only works because the model has been distilled for it; the project page says "distilled to one or two denoising steps", and a comment in wrappers.py refers to a DMD deployment setting. The training code is not released, so I cannot say more than that.
3. Decode. TAEHV turns the latent into four 480×864 RGB frames, which are packed to uint8 and copied to pinned host memory.
4. Commit. Run the DiT a second time on the clean latent at timestep 0, with the control channels zeroed (drop_control=True), purely to write this chunk's keys and values into the cache. On a full window it stops after the last layer has written its K/V and skips the output head.

Two things about step 4 matter for speed. First, it means every chunk costs two DiT passes, not one. Second, the RTX 5090 path queues it after the frame download starts, and "Nothing in a chunk waits for the GPU except the frame download." The frames leave while the GPU is still busy writing history. The clean pass costs throughput and no latency.

Why a second pass at all? Later chunks should condition on the frame the model actually produced, seen as clean input, not on the half-noisy activations of step 2. That is the Self Forcing family's recipe against drift. Zeroing the control channels means the history carries appearance, not the proxy.

The rest of the RTX 5090 runtime is the systems work that makes a 5B model fit the budget:

• FP8 linears. All ten linear layers in each of the 30 blocks are converted to e4m3 at load time, one scale per weight matrix and one per activation call, computed on the GPU so nothing synchronizes with the host. That is 4,907,335,680 of the 5,001,016,512 parameters, 98.1% (reasoned from the header shapes). Embeddings and the head stay BF16.
• A ring KV window. The cache is a fixed buffer of 21 latent frames; a new chunk overwrites the oldest. In BF16 that buffer is 30 layers × K and V × 8,505 tokens × 3,072 × 2 bytes, about 3.14 GB (reasoned). It never grows, which is what makes a long session possible on a 32 GB card. See attention and the KV cache for why this is the memory that usually runs out.
• CUDA graphs. The whole block stack is captured once during a 23-chunk warm-up and replayed, so the CPU launches one graph per pass instead of hundreds of kernels.
• Optional SageAttention. The rtx5090-sage preset keeps quantized keys and values per range of the ring and requantizes only the range that changed.

There is one more constraint worth knowing before you plan a long session. Positions are not recycled: "A session has a finite RoPE budget (8192 generated latent chunks by default)." That is 1 + 4 × 8,191 = 32,765 frames, about 9.1 minutes at 60 fps or 13.7 minutes at 40 fps (reasoned). Then you start a new session.

Three speeds, one pipeline

Now the three numbers. The project page gives the clue the post leaves out: its stat strip reads 83 FPS "1-step denoising, RTX 5090", 112 ms "key press to first reacting frame", and 48 ms "to denoise one chunk (1 latent frame, 4 frames)" (reported).

83.4 fps is the 48 ms, restated. Four frames in 48 ms is 83.3 fps; 83.4 fps is four frames in 47.96 ms (reasoned). It is model throughput: frames produced per second of model time, the same thing the server logs as model_fps, which excludes waiting for the game.

60 fps is the capture ceiling. The server takes --fps from 1 to 60, and for the shooters it must divide the 120 Hz physics rate (measured, server/app.py). At 60 fps the engine produces a chunk every 66.7 ms. If the model needs 48 ms of GPU per chunk, it is busy 72% of real time and keeps up with room to spare (reasoned). If it cannot keep up, nothing is dropped: "When production is blocked, the game slows relative to wall time instead of running a catch-up burst" (docs/architecture.md). So 60 is what you get when 83.4 is available, and 83.4 is headroom, not something you see on screen.

112 ms is a latency, and the chunk sets most of it. A key press first has to land in an engine frame. That frame then has to wait for the rest of its chunk: a frame at the start of a chunk waits three more frame periods, 50 ms at 60 fps, before the GPU can start. Then encode, denoise and decode. The output frames are presented one period apart, so each frame is delayed by the same total: three frame periods of filling, plus the time to first frames out, plus up to one more frame period for the press to land. With 48 ms as the time to frames out that is 98 to 115 ms, and 112 sits inside it (reasoned). I cannot tell from the page whether the 48 ms includes the clean pass, or whether the 112 ms includes WebRTC and the browser; neither is said.

The widget below runs that schedule. Pick the capture rate and the number of steps, set how long a DiT pass and the codec take, and move the key press around inside a chunk.

The defaults split 48 ms as 6 ms of codec and two DiT passes of 21 ms each, which is my assumption; Skywork publishes only the total. Under that split, a press reaches the screen 77 to 94 ms later before any network (reasoned), comfortably under 112 ms, which would leave about 20 to 35 ms for video encoding, transport and the browser. Toggle the clean pass to run before hand-out, as the reference path without early publishing does, and the range becomes 98 to 115 ms. Switch to two denoising steps and the GPU needs 69 ms per chunk, more than the 66.7 ms the engine allows at 60 fps, so the game slows slightly. That is why the README tells quality users to "reduce --fps to a sustainable measured capture rate".

Two lessons fall out of it that apply to any chunked streaming generator. Throughput is four frames divided by everything the GPU does, including work done after the frames have left. Latency is mostly the chunk itself: at 60 fps, three frame periods of filling is 50 ms before the model has started, and no amount of kernel work removes it. A chunk of one latent frame (four RGB frames) is already the smallest this VAE allows, so the next latency win would have to come from a higher capture rate or a codec with less temporal compression.

Is 48 ms plausible on one card? By my count each DiT pass is about 4.86 TFLOP at this window (3.52 in the linear layers, 1.27 in self-attention over 8,505 keys, a little in cross-attention), so two passes per chunk is about 9.7 TFLOP (reasoned). Doing that in 48 ms means roughly 200 TFLOP/s sustained, mostly through FP8 tensor cores. That is aggressive for a consumer card and well beyond what the same model reaches in BF16 on an A100, which the README measures at about 38 to 39 fps for the shooters, or roughly 104 ms per chunk (reported, reasoned). I could not test it, and the release does not include the measurement.

What the repository claims, and what it doesn't

The README is more careful than the post. Its only performance table is for one A100-SXM4-80GB at 864×480 with the realtime preset: ~38-39 fps for Castle and Ancient Egypt, ~38.5 fps for racing with a fixed prompt and ~28.9 fps with dynamic captions (reported). The RTX 5090 section, added on release day, gives setup commands and no numbers. --fps 40 is "a target capture cadence, not a throughput or latency guarantee." The model card goes further: "There is no guaranteed frame rate, consumer-GPU memory fit, or exact scene reconstruction. This model card does not claim a performance benchmark." The racing docs say none of model_fps, online_fps or browser FPS "alone is an action-to-photon latency measurement."

So 60 fps, 83.4 fps and 112 ms on an RTX 5090 are reported on the project page and in the post, and nowhere in the code or the model card (measured: none of those numbers appears in the repository). They are consistent with each other and with the runtime's schedule, as above. That is weaker than measured.

The project page also describes more than the release contains:

• Spatial memory by camera pose. The overview figure shows "3 frames' K/V by pose" retrieved into the attention context. The rollout code has a kvbank memory mode with memory_max_frames=3, but the streaming sessions hard-code memory_mode="none", and the architecture doc is explicit: "No automatic cache reset, SDEdit initialization, retrieved memory or dynamic prompt generation in the released gameplay configuration" (measured). Turn around and walk back, and what you see is whatever the 20-chunk window and the low-poly proxy imply, not a retrieved memory.
• The coding agent. "Coding Agent turns ideas into playable game logic" describes a workflow, not a component. The README invites you to "customize the source with a coding agent", meaning your own: edit terrain, NPC behaviour and rules in C++, rebuild, play. Nothing in the release calls a language model.
• Dynamic prompts exist for racing only, from the engine's caption catalog, and cost throughput: 38.5 against 28.9 fps on the A100 (reported).

The model card lists the limits: the checkpoints are "specialized to the supplied low-poly scenes and camera profiles", and unseen geometry, fast camera motion, occlusion and long sessions "can cause appearance drift or geometry errors". "The same world model can support entirely different gameplay — by changing the code, not retraining the model" holds for new rules on the same map; it does not hold for a new map, which needs a new checkpoint.

Is it the first?

"The world's first open-source video model game that runs in real time on a single consumer GPU" depends on what "video model game" means.

If it means any interactive generated world, no. ABot-World-0 (Apache-2.0, August) streams an action-conditioned 5B world from a single RTX 5090 at up to 16 fps, NVIDIA's SANA-WM generates a minute of camera-controlled world from 2.6B parameters (not in real time on one consumer card), and Matrix-Game's own 2.0 and 3.0 were real-time interactive world models before this. If it means a game whose rules run in an engine and whose every frame is rendered by a video model, then the closest prior work on this site is StreamRender-H3, which needs eight GB200s for 24 fps, and the Streaming Code World Model, which needs 5.1 s per 0.708 s chunk on an H200. Against those, Matrix-Game 4.0 at 60 fps on one consumer card, with code, weights and the three games released, is a real step. I have only checked the systems this site covers, so read "first" as "first I know of", and note that the consumer-GPU part is reported, not measured.

Licence

The code is Apache-2.0 (Matrix-Game-4/LICENSE; the repository root is MIT for earlier versions). Scene weights, sink images and game assets are Apache-2.0, the shared Wan2.2 and umT5 files keep their upstream Apache-2.0 terms, TAEHV is MIT, and some racing audio is CC BY 3.0. Training code and data are not included.

What I take from it

The interesting decision here is a division of labour, not a model. Put everything a rule can decide in an engine, give the network only the part that rules are bad at, which is how things look, and the network's job becomes small enough for one card at 60 fps. The cost is that it is a renderer per scene, not a world model you can drop into any game, and the model can be no more correct than the low-poly proxy it is given.

The speed claims hang together. 83.4 fps is the 48 ms chunk, 60 fps is the capture ceiling, and 112 ms is mostly the time to fill a four-frame chunk plus the model. They are still Skywork's numbers on Skywork's machine, published on a project page and not in the repository, which reports an A100 at about 38 fps. If you have a 5090, the runtime logs model_fps and per-stage timings with --profile-chunks. Those, plus a camera pointed at the screen while you press a key, would settle it.

Built on Skywork AI's Matrix-Game 4.0 project page, the SkyworkAI/Matrix-Game repository (Matrix-Game-4/, Apache-2.0) and the Skywork/Matrix-Game-4.0 model card and configs. Both figures are reproduced from the project page for commentary. The frame-budget timeline is an original visualization of the release's chunk schedule; its stage split is an assumption, not a measurement.