# Matrix-Game 4.0: a 5B video model re-skins a low-poly game at 60 fps

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/matrix-game-4
> date: 2026-10-15
> tags: world-models, video-generation, realtime, kv-cache, quantization, inference

Skywork's [announcement](https://x.com/Skywork_ai/status/2110615377088745817) for Matrix-Game 4.0 packs a lot into one post: "the world's first open-source video model game that runs in real time on a single consumer GPU", "480p at 60 FPS", "112ms end-to-end response", "up to 83.4 FPS generation", a single RTX 5090, and a loop that reads "Player input → Game engine updates the world → Video model renders the next frame → Player interacts again".

60 and 83.4 are both frame rates and 112 ms is a latency, and I wanted to know which pipeline produces all three at once. So I read the code in [SkyworkAI/Matrix-Game](https://github.com/SkyworkAI/Matrix-Game) (the `Matrix-Game-4/` directory, at the commit of release day), the configs on [Skywork/Matrix-Game-4.0](https://huggingface.co/Skywork/Matrix-Game-4.0), and the safetensors headers of the three scene checkpoints, which I range-read without downloading the 10 GB files. There is no paper yet; the [project page](https://matrix-game-v4-0.github.io/page/) says "soon". Every number below is labelled **measured** (I read or computed it from a file), **reported** (Skywork's figure, not re-run) or **reasoned** (my arithmetic on the other two). I did not run the model.

<ModelCard repo="Skywork/Matrix-Game-4.0" />

<RepoCard repo="SkyworkAI/Matrix-Game" />

## The game is code, the model is a renderer

The most important design decision is in the first line of `docs/architecture.md`:

```text
native game -> clean RGB + frame metadata -> causal renderer -> bounded display
publisher -> WebRTC -> presented-frame HUD/audio
```

The game is a native C++ program (`game/src/` for the two shooters, `game/racing/src/` for the racer) with Jolt physics, collision, NPC encounters, a map, a HUD and sound. The shooters step physics at 120 Hz, and on every captured frame the game renders the world with OpenGL in flat low-poly colours: buildings as tinted boxes, characters as blocks, roads as grey planes. That frame is the model's input. The model card says it directly: "An executable game determines gameplay, physics and input, while the model re-renders clean low-poly RGB." The Hugging Face pipeline tag is `video-to-video`.

So the player's keystrokes never reach the network (measured: no action tensor exists anywhere in the inference path; `CastleStream.push` takes RGB frames and nothing else). You press W, the engine moves the character, the next low-poly frame shows the character further along, and that is the only way the model learns you pressed anything. The HUD, crosshair, minimap and audio are drawn on top of the model's output by the browser, matched to the frame being presented, and "UI pixels must never enter the model" (`docs/offline.md`).

<Figure
  src="https://media.thesatyajit.com/ai/articles/matrix-game-4/fig2.jpg?v=18cb360ceb4a3067"
  alt="Side-by-side frame from a racing session. Left: the generated frame, a photoreal city street with brick buildings, a silver sedan seen from behind on wet asphalt, yellow crosswalk stripes, a pursuit HUD at top left, an input overlay at bottom left and a minimap at bottom right. Right: the engine's low-poly proxy of the same moment, with flat pink, brown and yellow boxes for buildings, a blue box for the car and a grey road."
  caption="One frame of the 54-second racing session: the generated frame with the game's HUD on the left, the engine's low-poly proxy on the right. Only the right half, without the HUD, is the model's input (Matrix-Game 4.0 project page, long-play racing clip)."
/>

This is the same split NVIDIA used in [StreamRender-H3](/articles/streamrender-h3), three days earlier, where a three.js racer sends a 14-colour semantic map and a 33B model paints it on eight GB200s. Matrix-Game sends plain shaded RGB rather than class colours, and it runs a model about a sixth the size on one card. It is also the opposite of the action-conditioned world models this site has covered, like [ABot-World-0](/articles/abot-world) and [Astronex-World](/articles/astronex-world), where the network is told "move forward" and must invent what comes next. Here nothing has to be invented about where things are. Collisions, hit detection and the score live in C++, and a hallucinated wall cannot stop you, because "Neural output does not change collision or gameplay" (model card).

The project page puts it well: in pixel-only world models, "once something leaves the view it is gone, and game rules have to be re-inferred from every frame." The engine owns the state instead. What is left for the network is hard but well-posed: make this proxy look like a AAA game, consistently, four frames at a time.

<Figure
  src="https://media.thesatyajit.com/ai/articles/matrix-game-4/fig1.png?v=e834577754072e17"
  alt="System diagram in four columns: Input, Harness, Interface, Rendering. An instruction and the player's WASD and shoot inputs enter a harness made of a Coding Agent, which writes new prompts and new mechanisms, and a rule-based Game Engine. The harness emits semantic prompts, reference frames and a dense low-poly proxy every frame, which feed a Causal DiT generating 4 frames per chunk in 1 to 2 steps, outputting 480 by 864 frames to the screen. Below the DiT, an attention context holds memory, reference frames and the latest 20 chunks, and a Spatial Memory box stores 3 frames' K/V retrieved by camera pose."
  caption="The system as the project page draws it. The released runtime implements the left-to-right path from game engine to screen; the spatial-memory box and the coding agent are not in the shipped code (Matrix-Game 4.0 project page, overview figure)."
/>

## What the model actually sees

Each scene ships as its own checkpoint (`fps-castle/`, `fps-egypt/`, `driving_game/`), and each is a fine-tune of [Wan2.2-Fun-5B-Control](https://huggingface.co/alibaba-pai/Wan2.2-Fun-5B-Control), a [diffusion transformer](/architectures/diffusion-transformer) that takes a control video alongside its noise. I compared the headers:

| | Matrix-Game 4.0 (`fps-egypt`) | Wan2.2-Fun-5B-Control |
|---|---|---|
| Tensors | 825 | 827 |
| Parameters | 5,001,016,512 | 5,001,609,408 |
| Blocks / width / FFN | 30 / 3,072 / 14,336 | same |
| Patch embedding | 3,072 × 148 × (1, 2, 2) | same |
| Missing | — | `ref_conv` (592,896 params) |

(measured, from the safetensors headers; the parameter count matches `model.json` and the model card.) Every shape is identical. Matrix-Game dropped the base model's reference-image convolution and kept the rest, so its "reference frames" are not the base model's image channel; they are KV entries, below.

The 148 input channels are the interesting part. A comment in `inference/wrappers.py` spells out the layout: 48 noisy latent channels, 48 control channels, 48 inpainting channels that are always zero, and 4 mask channels that are always zero. The control channels are the low-poly frames passed through the codec. Wan2.2's VAE compresses 16× spatially and 4× in time, so a 480×864 frame becomes a 30×54 grid of 48-channel latents, and four RGB frames become one latent frame. After the (1, 2, 2) patch embedding that is 15×27 = 405 tokens per latent frame (measured, `FRAME_TOKENS = 15 * 27` in `rtx5090/stream.py`).

Text comes in by cross-attention from a umT5-XXL encoder: a fixed scene prompt for the shooters ("a third-person shooter game in an ancient Egyptian desert city with a great pyramid…"), and for racing optionally a caption the engine writes per chunk from what is in view. Those captions come from a catalog the engine ships (`racing/assets/caption-catalog.json`), not from a language model.

## One chunk at a time

The model is causal over latent frames. Each step generates one latent, which decodes to four RGB frames (one, for the very first chunk). The attention window is 21 latent frames including a "sink" at position zero (measured, `kv_window_latents: 21` in every `inference.json`). For the shooters the sink is a reference image of the scene, encoded once and written into the cache but never shown; for racing it is shown as frame zero. Every new chunk attends to the sink, the 20 most recent chunks, and itself: 405 queries against 8,505 keys (reasoned: 21 × 405).

`rtx5090/stream.py` lays out what happens per chunk, and its docstring is a good summary:

> Per chunk: encode the control frames, denoise, decode, hand the frames out, then commit the clean latent to the KV history.

In code, with the `rtx5090` preset:

1. **Encode.** The four low-poly frames go through TAEHV, a small streaming autoencoder that lives in Wan2.2's latent space (`base/taew2_2.pth`, 22.9 MB measured), instead of the 2.8 GB Wan VAE the `quality` preset uses. Result: one 48×30×54 control latent.
2. **Denoise.** Start from Gaussian noise, run the DiT once at timestep 1000 with the control latent in channels 48 to 95, and take the predicted clean latent. CFG is 1, so there is no second unconditional pass. One step only works because the model has been distilled for it; the project page says "distilled to one or two denoising steps", and a comment in `wrappers.py` refers to a DMD deployment setting. The training code is not released, so I cannot say more than that.
3. **Decode.** TAEHV turns the latent into four 480×864 RGB frames, which are packed to uint8 and copied to pinned host memory.
4. **Commit.** Run the DiT a second time on the clean latent at timestep 0, with the control channels zeroed (`drop_control=True`), purely to write this chunk's keys and values into the cache. On a full window it stops after the last layer has written its K/V and skips the output head.

Two things about step 4 matter for speed. First, it means every chunk costs two DiT passes, not one. Second, the RTX 5090 path queues it *after* the frame download starts, and "Nothing in a chunk waits for the GPU except the frame download." The frames leave while the GPU is still busy writing history. The clean pass costs throughput and no latency.

Why a second pass at all? Later chunks should condition on the frame the model actually produced, seen as clean input, not on the half-noisy activations of step 2. That is the [Self Forcing](/articles/vivix-w1) family's recipe against drift. Zeroing the control channels means the history carries appearance, not the proxy.

The rest of the RTX 5090 runtime is the systems work that makes a 5B model fit the budget:

- **FP8 linears.** All ten linear layers in each of the 30 blocks are converted to e4m3 at load time, one scale per weight matrix and one per activation call, computed on the GPU so nothing synchronizes with the host. That is 4,907,335,680 of the 5,001,016,512 parameters, 98.1% (reasoned from the header shapes). Embeddings and the head stay BF16.
- **A ring KV window.** The cache is a fixed buffer of 21 latent frames; a new chunk overwrites the oldest. In BF16 that buffer is 30 layers × K and V × 8,505 tokens × 3,072 × 2 bytes, about 3.14 GB (reasoned). It never grows, which is what makes a long session possible on a 32 GB card. See [attention and the KV cache](/architectures/attention-kv) for why this is the memory that usually runs out.
- **CUDA graphs.** The whole block stack is captured once during a 23-chunk warm-up and replayed, so the CPU launches one graph per pass instead of hundreds of kernels.
- **Optional SageAttention.** The `rtx5090-sage` preset keeps quantized keys and values per range of the ring and requantizes only the range that changed.

There is one more constraint worth knowing before you plan a long session. Positions are not recycled: "A session has a finite RoPE budget (8192 generated latent chunks by default)." That is 1 + 4 × 8,191 = 32,765 frames, about 9.1 minutes at 60 fps or 13.7 minutes at 40 fps (reasoned). Then you start a new session.

## Three speeds, one pipeline

Now the three numbers. The project page gives the clue the post leaves out: its stat strip reads 83 FPS "1-step denoising, RTX 5090", 112 ms "key press to first reacting frame", and 48 ms "to denoise one chunk (1 latent frame, 4 frames)" (reported).

**83.4 fps is the 48 ms, restated.** Four frames in 48 ms is 83.3 fps; 83.4 fps is four frames in 47.96 ms (reasoned). It is model throughput: frames produced per second of model time, the same thing the server logs as `model_fps`, which excludes waiting for the game.

**60 fps is the capture ceiling.** The server takes `--fps` from 1 to 60, and for the shooters it must divide the 120 Hz physics rate (measured, `server/app.py`). At 60 fps the engine produces a chunk every 66.7 ms. If the model needs 48 ms of GPU per chunk, it is busy 72% of real time and keeps up with room to spare (reasoned). If it cannot keep up, nothing is dropped: "When production is blocked, the game slows relative to wall time instead of running a catch-up burst" (`docs/architecture.md`). So 60 is what you get when 83.4 is available, and 83.4 is headroom, not something you see on screen.

**112 ms is a latency, and the chunk sets most of it.** A key press first has to land in an engine frame. That frame then has to wait for the rest of its chunk: a frame at the start of a chunk waits three more frame periods, 50 ms at 60 fps, before the GPU can start. Then encode, denoise and decode. The output frames are presented one period apart, so each frame is delayed by the same total: three frame periods of filling, plus the time to first frames out, plus up to one more frame period for the press to land. With 48 ms as the time to frames out that is 98 to 115 ms, and 112 sits inside it (reasoned). I cannot tell from the page whether the 48 ms includes the clean pass, or whether the 112 ms includes WebRTC and the browser; neither is said.

The widget below runs that schedule. Pick the capture rate and the number of steps, set how long a DiT pass and the codec take, and move the key press around inside a chunk.

<FrameBudget />

The defaults split 48 ms as 6 ms of codec and two DiT passes of 21 ms each, which is my assumption; Skywork publishes only the total. Under that split, a press reaches the screen 77 to 94 ms later before any network (reasoned), comfortably under 112 ms, which would leave about 20 to 35 ms for video encoding, transport and the browser. Toggle the clean pass to run before hand-out, as the reference path without early publishing does, and the range becomes 98 to 115 ms. Switch to two denoising steps and the GPU needs 69 ms per chunk, more than the 66.7 ms the engine allows at 60 fps, so the game slows slightly. That is why the README tells `quality` users to "reduce `--fps` to a sustainable measured capture rate".

Two lessons fall out of it that apply to any chunked streaming generator. Throughput is four frames divided by everything the GPU does, including work done after the frames have left. Latency is mostly the chunk itself: at 60 fps, three frame periods of filling is 50 ms before the model has started, and no amount of kernel work removes it. A chunk of one latent frame (four RGB frames) is already the smallest this VAE allows, so the next latency win would have to come from a higher capture rate or a codec with less temporal compression.

Is 48 ms plausible on one card? By my count each DiT pass is about 4.86 TFLOP at this window (3.52 in the linear layers, 1.27 in self-attention over 8,505 keys, a little in cross-attention), so two passes per chunk is about 9.7 TFLOP (reasoned). Doing that in 48 ms means roughly 200 TFLOP/s sustained, mostly through FP8 tensor cores. That is aggressive for a consumer card and well beyond what the same model reaches in BF16 on an A100, which the README measures at about 38 to 39 fps for the shooters, or roughly 104 ms per chunk (reported, reasoned). I could not test it, and the release does not include the measurement.

## What the repository claims, and what it doesn't

The README is more careful than the post. Its only performance table is for one A100-SXM4-80GB at 864×480 with the `realtime` preset: ~38-39 fps for Castle and Ancient Egypt, ~38.5 fps for racing with a fixed prompt and ~28.9 fps with dynamic captions (reported). The RTX 5090 section, added on release day, gives setup commands and no numbers. `--fps 40` is "a target capture cadence, not a throughput or latency guarantee." The model card goes further: "There is no guaranteed frame rate, consumer-GPU memory fit, or exact scene reconstruction. This model card does not claim a performance benchmark." The racing docs say none of `model_fps`, `online_fps` or browser FPS "alone is an action-to-photon latency measurement."

So 60 fps, 83.4 fps and 112 ms on an RTX 5090 are reported on the project page and in the post, and nowhere in the code or the model card (measured: none of those numbers appears in the repository). They are consistent with each other and with the runtime's schedule, as above. That is weaker than measured.

The project page also describes more than the release contains:

- **Spatial memory by camera pose.** The overview figure shows "3 frames' K/V by pose" retrieved into the attention context. The rollout code has a `kvbank` memory mode with `memory_max_frames=3`, but the streaming sessions hard-code `memory_mode="none"`, and the architecture doc is explicit: "No automatic cache reset, SDEdit initialization, retrieved memory or dynamic prompt generation in the released gameplay configuration" (measured). Turn around and walk back, and what you see is whatever the 20-chunk window and the low-poly proxy imply, not a retrieved memory.
- **The coding agent.** "Coding Agent turns ideas into playable game logic" describes a workflow, not a component. The README invites you to "customize the source with a coding agent", meaning your own: edit terrain, NPC behaviour and rules in C++, rebuild, play. Nothing in the release calls a language model.
- **Dynamic prompts** exist for racing only, from the engine's caption catalog, and cost throughput: 38.5 against 28.9 fps on the A100 (reported).

The model card lists the limits: the checkpoints are "specialized to the supplied low-poly scenes and camera profiles", and unseen geometry, fast camera motion, occlusion and long sessions "can cause appearance drift or geometry errors". "The same world model can support entirely different gameplay — by changing the code, not retraining the model" holds for new rules on the same map; it does not hold for a new map, which needs a new checkpoint.

## Is it the first?

"The world's first open-source video model game that runs in real time on a single consumer GPU" depends on what "video model game" means.

If it means any interactive generated world, no. [ABot-World-0](/articles/abot-world) (Apache-2.0, August) streams an action-conditioned 5B world from a single RTX 5090 at up to 16 fps, NVIDIA's [SANA-WM](/articles/sana-wm) generates a minute of camera-controlled world from 2.6B parameters (not in real time on one consumer card), and Matrix-Game's own 2.0 and 3.0 were real-time interactive world models before this. If it means a game whose rules run in an engine and whose every frame is rendered by a video model, then the closest prior work on this site is [StreamRender-H3](/articles/streamrender-h3), which needs eight GB200s for 24 fps, and the [Streaming Code World Model](/articles/streaming-code-world-model), which needs 5.1 s per 0.708 s chunk on an H200. Against those, Matrix-Game 4.0 at 60 fps on one consumer card, with code, weights and the three games released, is a real step. I have only checked the systems this site covers, so read "first" as "first I know of", and note that the consumer-GPU part is reported, not measured.

## Licence

The code is Apache-2.0 (`Matrix-Game-4/LICENSE`; the repository root is MIT for earlier versions). Scene weights, sink images and game assets are Apache-2.0, the shared Wan2.2 and umT5 files keep their upstream Apache-2.0 terms, TAEHV is MIT, and some racing audio is CC BY 3.0. Training code and data are not included.

## What I take from it

The interesting decision here is a division of labour, not a model. Put everything a rule can decide in an engine, give the network only the part that rules are bad at, which is how things look, and the network's job becomes small enough for one card at 60 fps. The cost is that it is a renderer per scene, not a world model you can drop into any game, and the model can be no more correct than the low-poly proxy it is given.

The speed claims hang together. 83.4 fps is the 48 ms chunk, 60 fps is the capture ceiling, and 112 ms is mostly the time to fill a four-frame chunk plus the model. They are still Skywork's numbers on Skywork's machine, published on a project page and not in the repository, which reports an A100 at about 38 fps. If you have a 5090, the runtime logs `model_fps` and per-stage timings with `--profile-chunks`. Those, plus a camera pointed at the screen while you press a key, would settle it.

*Built on Skywork AI's [Matrix-Game 4.0](https://matrix-game-v4-0.github.io/page/) project page, the [SkyworkAI/Matrix-Game](https://github.com/SkyworkAI/Matrix-Game) repository (`Matrix-Game-4/`, Apache-2.0) and the [Skywork/Matrix-Game-4.0](https://huggingface.co/Skywork/Matrix-Game-4.0) model card and configs. Both figures are reproduced from the project page for commentary. The frame-budget timeline is an original visualization of the release's chunk schedule; its stage split is an assumption, not a measurement.*