2026-07-04 · 9 min · world-models · diffusion · mixture-of-experts · physical-ai · explainer
A language model can tell you what will probably happen if you tip a full mug. A world model has to show you — render the next frames, consistent with gravity, contact, and the fact that the coffee ends up on the table. That is the bet behind physical AI: robots and agents need a model that can both reason about what to do and imagine the visual consequences of doing it. NVIDIA's Cosmos 3 (arXiv 2606.02800) is a world foundation model built around exactly that pairing — and the interesting part is how it fuses the two, not that it does.
The usual way to bolt reasoning onto a generator is to run an LLM, get some text, and feed it to a diffusion model as a prompt. Cosmos instead puts both in one network and one token sequence, so the generator can read the reasoner's internal state as it works. The mechanism is a Mixture-of-Transformers (MoT).
Two towers, one sequence
An MoT keeps separate weights per modality but runs one shared attention over a single sequence. Cosmos uses two towers: an autoregressive Reasoner that does the planning, and a diffusion Generator that produces video and world frames. Each token in the sequence is routed to its tower's weights, but they all attend together. Step through it:
Two towers, one sequence. The Reasoner is autoregressive and attends only causally over its own keys — it does the planning. The Generator is a diffusion model, and its query fans out over the concatenation [ K_AR ; K_DM ]— so it can read the reasoner's keys and values directly. That single cross-stream read is how a plan turns into pixels: reasoning is fused into generation inside one attention operation, not bolted on afterward.
The load-bearing detail is the attention pattern. The Reasoner is autoregressive, so its queries attend
only causally over its own keys (K_AR) — standard next-token reasoning. The Generator is a diffusion
model, and its queries Q_DM attend over the concatenation [K_AR ; K_DM] — the reasoner's keys and
its own. That one cross-stream read is the whole idea: the generator can look directly at what the reasoner
decided, so a plan flows into pixels inside a single attention operation rather than through a text
bottleneck. This is the paper's dual-stream joint-attention design:
![Architecture diagram of Cosmos 3. On the left, a Reasoner processes an AR subsequence of vision (ViT) and language tokens through layer norm, a shared multimodal attention block, and an MLP, using causal self-attention Attn(Q_AR, K_AR, V_AR). On the right, a Generator processes a diffusion subsequence of noisy vision (VAE), audio, and action tokens through the same shared attention block but with full attention Attn(Q_DM, [K_AR;K_DM], [V_AR;V_DM]). On the far right, an attention-mask grid shows AR queries attending causally (triangular) only to K_AR, while DM queries attend fully to both K_AR and K_DM.](/articles/cosmos-world-model/fig1.png)
If you have read our TwoTower explainer, the silhouette will look familiar — a frozen autoregressive tower feeding a diffusion denoiser — but the systems solve different problems. TwoTower is a language model that splits one job (context vs. denoising) to speed up text decode; Cosmos is a physical-AI world model whose two towers span different modalities and are trained jointly, with the generator reading the reasoner online rather than cross-attending to a frozen copy. Same family tree, different animal.
Two contrasts pin down what MoT is. A dense transformer would force one set of weights to be good at both causal reasoning and bidirectional denoising — the conflict TwoTower also fights. A Mixture-of-Experts model routes tokens to expert FFNs but shares one attention and one modality regime. MoT is the third option: separate weights per modality/role (the two towers), one shared attention — so the split is by what kind of token this is, and the fusion happens in attention. The diffusion half itself is a standard denoiser; if that machinery is unfamiliar, our diffusion and diffusion-language-model pieces cover it.
How the world becomes tokens
Before any of that attention can happen, five modalities have to become tokens in one sequence — and Cosmos encodes each differently. Understanding tokens land in the Reasoner subsequence; generation tokens land in the Generator subsequence. Trace a modality through its encoder:
The generation path uses a frozen video VAE: 4× temporal and 32×32 spatial compression (a 16×16 encoder plus a 2×2 patch merge), linearly projected into the transformer.
The asymmetry is deliberate. The understanding path uses a ViT encoder trained jointly with the backbone, so the reasoner sees vision the way its Qwen3-VL ancestor did. The generation path uses frozen VAEs — a Wan2.2 video VAE (4× temporal, 32×32 spatial compression) and an audio VAE — so the diffusion tower only has to produce latents a fixed decoder already knows how to render. And actions from every embodiment collapse into one shared latent action space, which is what lets a single model be a policy for arms, humanoids, and vehicles alike.
Planning in pixel space: Action-CoT
Reasoning about action is not the same as reasoning in words. To act in the world you need a plan expressed in terms of where things move. Cosmos's Action-CoT turns an instruction into a 2D motion plan on the image plane — a chain-of-thought drawn as a trajectory — before and while it generates. Pick an instruction and scrub the plan into existence:
The chain-of-thought here isn't a paragraph — it's a path. Cosmos maps the instruction to a 2D trajectory on the image plane: where the gripper (or the camera, or an object) should move, waypoint by waypoint. That plan then conditions the diffusion tower, so the frames it denoises are the ones that carry out the motion rather than merely look plausible. Reasoning about action happens in pixel space, where the physics has to hold.
Concretely: the model predicts a path of waypoints across the frame (the gripper's route to the mug, the block's slide, the drawer's pull), and that trajectory conditions the diffusion tower. The frames it denoises then have to realize the motion, not merely look plausible — the chain-of-thought lives in the same coordinate space the physics does. It is a neat answer to a real problem: language is a lossy way to specify a manipulation, and image-plane motion is exactly what a downstream controller can consume.
One backbone, six models
Because reasoning and generation share a backbone and everything is tokens, the same weights become six different task models just by choosing which modalities go in and which come out. Route it:
Nothing in the network changes between these six. The same Cosmos backbone becomes a vision-language model, an image or audio-visual generator, a policy, or a forward / inverse dynamicsmodel purely by choosing which modalities enter and which it's asked to produce. A forward dynamics model predicts the next video given a past and an action; an inverse one recovers the action that connects two frames — both fall out of one omnimodal world model.
The two dynamics models are the ones that matter for physical AI. A forward dynamics model predicts the next video given the past frames and an action — that is the world model as a simulator an agent can plan against. An inverse dynamics model recovers the action that connects two frames — useful for learning control from unlabeled video. Both are the same network with the arrows reversed.
Cosmos comes in three sizes, all built on the dual-tower MoT: Edge — 4B total on a dense 2B transformer
trained from scratch (28 layers, a later release), Nano — 16B total on a dense 8B, initialized from
Qwen3-VL-8B (36 layers), and Super — 64B total on a dense 32B, from Qwen3-VL-32B (64 layers).
The Qwen initialization is telling — the reasoner tower inherits a strong pretrained VLM, while the generator
is a flow-matching diffusion tower (it predicts a constant velocity, v* = ε − x₀) grafted on and trained
to read it. Everything is openly released under OpenMDW-1.1: the Nano and Super checkpoints, the code,
five synthetic SDG datasets (physics, robots, driving, digital humans, warehouses), and the Cosmos-HUE
evaluation benchmark.
The training pipeline
Two towers means two training tracks, joined where it counts. Click through the stages:
Task-specific post-training — Text-to-Image, Image-to-Video, and Robot Policy — with best-of-N sampling scored by the WMReward reward model.
The two tracks aren't independent: mid-training is where the diffusion Generator learns to attend over the Reasoner's keys, so by the time post-training specializes the model into six task heads, the plan-to-pixels pathway is already in place.
The Reasoner is extended from a VLM and fine-tuned on reasoning and Action-CoT data; the Generator is pre-trained as a flow-matching denoiser, then mid-trained jointly with the reasoner — the stage that actually wires up the dual-stream attention — before splitting into task-specific post-training for text-to-image, image-to-video, and robot policy. Several headline results lean on best-of-N sampling against a learned reward model (WMReward), which is worth holding in mind when reading the numbers.
The numbers, honestly
By the report's own tally, the post-trained models were the best open-source Text-to-Image and Image-to-Video models on Artificial Analysis, and the best policy model on RoboArena, at the time of writing — and Cosmos leads a physical-AI reasoning leaderboard among open models. Those are real, but they're open-model rankings, and the scope is easy to lose in a press release. On the reasoning benchmark where Cosmos Super posts its headline result, a closed frontier model still sits above it:
None of that makes the work less interesting — it makes the claim precise. As an open, openly-licensed omnimodal world model that fuses reasoning and generation in one attention operation and plans in image space, Cosmos 3 is a genuinely new capability tier for people building on open weights. It just isn't the best model in the world at everything, and the paper's own numbers say so if you read the parentheses.
The take
The idea worth keeping is the dual-stream joint attention. Most "reasoning + generation" systems chain two models and pay a text bottleneck between them; Cosmos makes the generator's queries attend over the reasoner's keys inside one operation, so the plan reaches the pixels without being flattened into a prompt. Pair that with Action-CoT — chain-of-thought as motion on the image plane — and you get a world model whose reasoning is expressed in the same space its physics has to hold. The Mixture-of-Transformers is the enabling structure: separate weights per modality, one shared sequence, fusion in attention. Whether the scoped-SOTA numbers hold as the leaderboards churn is beside the point; the architecture is the contribution, and it is a clean one.
Built on NVIDIA Cosmos 3 (arXiv 2606.02800; OpenMDW-1.1 license). The Reasoner/Generator MoT, dual-stream joint attention, and Action-CoT are described in the paper (Figures 5 and 1); the interactive diagrams are illustrations of the mechanism. Benchmark figures are quoted as scoped comparisons — best among open models, with a closed frontier model (Gemini 3.1 Pro) still ahead — and some results use best-of-N with the model's own reward model.