# Cosmos 3: a world model that reasons and generates in one sequence

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/cosmos-world-model
> date: 2026-07-04
> tags: world-models, diffusion, mixture-of-experts, physical-ai, explainer
A language model can tell you what will probably happen if you tip a full mug. A **world model** has to
*show* you — render the next frames, consistent with gravity, contact, and the fact that the coffee ends
up on the table. That is the bet behind **physical AI**: robots and agents need a model that can both
**reason** about what to do and **imagine** the visual consequences of doing it. NVIDIA's **Cosmos 3**
(arXiv 2606.02800) is a world foundation model built around exactly that pairing — and the interesting
part is *how* it fuses the two, not that it does.

The usual way to bolt reasoning onto a generator is to run an LLM, get some text, and feed it to a
diffusion model as a prompt. Cosmos instead puts both in **one network and one token sequence**, so the
generator can read the reasoner's internal state as it works. The mechanism is a **Mixture-of-Transformers
(MoT)**.

## Two towers, one sequence

An MoT keeps **separate weights per modality** but runs **one shared attention** over a single sequence.
Cosmos uses two towers: an **autoregressive Reasoner** that does the planning, and a **diffusion
Generator** that produces video and world frames. Each token in the sequence is routed to its tower's
weights, but they all attend together. Step through it:

<DualStream />

The load-bearing detail is the attention pattern. The Reasoner is autoregressive, so its queries attend
only **causally over its own keys** (`K_AR`) — standard next-token reasoning. The Generator is a diffusion
model, and its queries `Q_DM` attend over the **concatenation `[K_AR ; K_DM]`** — the reasoner's keys *and*
its own. That one cross-stream read is the whole idea: the generator can look directly at what the reasoner
decided, so a plan flows into pixels inside a single attention operation rather than through a text
bottleneck. This is the paper's dual-stream joint-attention design:

<Figure
  src="/articles/cosmos-world-model/fig1.png"
  alt="Architecture diagram of Cosmos 3. On the left, a Reasoner processes an AR subsequence of vision (ViT) and language tokens through layer norm, a shared multimodal attention block, and an MLP, using causal self-attention Attn(Q_AR, K_AR, V_AR). On the right, a Generator processes a diffusion subsequence of noisy vision (VAE), audio, and action tokens through the same shared attention block but with full attention Attn(Q_DM, [K_AR;K_DM], [V_AR;V_DM]). On the far right, an attention-mask grid shows AR queries attending causally (triangular) only to K_AR, while DM queries attend fully to both K_AR and K_DM."
  caption="The Mixture-of-Transformers: an AR Reasoner and a diffusion Generator share one sequence and one attention operation. AR queries attend causally over K_AR; the diffusion queries attend over the full concatenation [K_AR ; K_DM], so reasoning is fused into generation (paper, Figure 5)."
/>

If you have read our [TwoTower explainer](/articles/nemotron-twotower), the silhouette will look familiar —
a frozen autoregressive tower feeding a diffusion denoiser — but the systems solve different problems.
TwoTower is a *language* model that splits one job (context vs. denoising) to speed up text decode; Cosmos
is a *physical-AI world model* whose two towers span different modalities and are trained jointly, with the
generator reading the reasoner online rather than cross-attending to a frozen copy. Same family tree,
different animal.

Two contrasts pin down what MoT *is*. A **dense** transformer would force one set of weights to be good at
both causal reasoning and bidirectional denoising — the conflict TwoTower also fights. A [Mixture-of-Experts
model](/articles/mixture-of-experts-from-scratch) routes *tokens* to expert FFNs but shares one attention
and one modality regime. MoT is the third option: **separate weights per modality/role (the two towers),
one shared attention** — so the split is by *what kind of token this is*, and the fusion happens in
attention. The diffusion half itself is a standard denoiser; if that machinery is unfamiliar, our
[diffusion](/articles/set-diffusion) and [diffusion-language-model](/articles/illada-diffusion-language-model)
pieces cover it.

## How the world becomes tokens

Before any of that attention can happen, five modalities have to become tokens in one sequence — and
Cosmos encodes each differently. Understanding tokens land in the Reasoner subsequence; generation
tokens land in the Generator subsequence. Trace a modality through its encoder:

<TokenStack />

The asymmetry is deliberate. The **understanding** path uses a ViT encoder trained *jointly* with the
backbone, so the reasoner sees vision the way its Qwen3-VL ancestor did. The **generation** path uses
*frozen* VAEs — a Wan2.2 video VAE (4× temporal, 32×32 spatial compression) and an audio VAE — so the
diffusion tower only has to produce latents a fixed decoder already knows how to render. And **actions**
from every embodiment collapse into one shared latent action space, which is what lets a single model be
a policy for arms, humanoids, and vehicles alike.

## Planning in pixel space: Action-CoT

Reasoning about *action* is not the same as reasoning in words. To act in the world you need a plan
expressed in terms of *where things move*. Cosmos's **Action-CoT** turns an instruction into a **2D motion
plan on the image plane** — a chain-of-thought drawn as a trajectory — before and while it generates. Pick
an instruction and scrub the plan into existence:

<ActionCoT />

Concretely: the model predicts a path of waypoints across the frame (the gripper's route to the mug, the
block's slide, the drawer's pull), and that trajectory *conditions the diffusion tower*. The frames it
denoises then have to realize the motion, not merely look plausible — the chain-of-thought lives in the
same coordinate space the physics does. It is a neat answer to a real problem: language is a lossy way to
specify a manipulation, and image-plane motion is exactly what a downstream controller can consume.

## One backbone, six models

Because reasoning and generation share a backbone and everything is tokens, the *same* weights become six
different task models just by choosing which modalities go in and which come out. Route it:

<OneBackbone />

<Figure
  src="/articles/cosmos-world-model/fig2.png"
  alt="Overview diagram titled Cosmos 3, an Omnimodal World Model, with modality icons for Language, Image, Video, Audio, and Action. Below, six task models each show inputs flowing into a Cosmos 3 box and outputs coming out: Vision-Language Model, Image Generation Model, Audio-Visual Generation Model, Policy/World-Action Model, Forward Dynamics Model, and Inverse Dynamics Model."
  caption="One backbone spawns six task models — differing only in which modalities enter and exit, from vision-language to forward and inverse dynamics (paper, Figure 1)."
/>

The two dynamics models are the ones that matter for physical AI. A **forward dynamics** model predicts the
next video given the past frames and an action — that is the world model as a *simulator* an agent can plan
against. An **inverse dynamics** model recovers the action that connects two frames — useful for learning
control from unlabeled video. Both are the same network with the arrows reversed.

Cosmos comes in three sizes, all built on the dual-tower MoT: **Edge — 4B total on a dense 2B transformer
trained from scratch** (28 layers, a later release), **Nano — 16B total on a dense 8B**, initialized from
**Qwen3-VL-8B** (36 layers), and **Super — 64B total on a dense 32B**, from **Qwen3-VL-32B** (64 layers).
The Qwen initialization is telling — the reasoner tower inherits a strong pretrained VLM, while the generator
is a **flow-matching** diffusion tower (it predicts a constant velocity, `v* = ε − x₀`) grafted on and trained
to read it. Everything is openly released under **OpenMDW-1.1**: the Nano and Super checkpoints, the code,
five synthetic **SDG** datasets (physics, robots, driving, digital humans, warehouses), and the **Cosmos-HUE**
evaluation benchmark.

## The training pipeline

Two towers means two training tracks, joined where it counts. Click through the stages:

<TrainingStages />

The Reasoner is extended from a VLM and fine-tuned on reasoning and Action-CoT data; the Generator is
pre-trained as a flow-matching denoiser, then **mid-trained jointly** with the reasoner — the stage that
actually wires up the dual-stream attention — before splitting into task-specific post-training for
text-to-image, image-to-video, and robot policy. Several headline results lean on **best-of-N sampling
against a learned reward model (WMReward)**, which is worth holding in mind when reading the numbers.

## The numbers, honestly

By the report's own tally, the post-trained models were the **best open-source Text-to-Image and
Image-to-Video models on Artificial Analysis**, and the **best policy model on RoboArena**, at the time of
writing — and Cosmos leads a physical-AI reasoning leaderboard **among open models**. Those are real, but
they're *open-model* rankings, and the scope is easy to lose in a press release. On the reasoning benchmark
where Cosmos Super posts its headline result, a closed frontier model still sits above it:

<BenchBars
  title="Physical-AI reasoning benchmark (%) — scoped comparison"
  unit="%"
  bars={[
    { label: "Gemini 3.1 Pro", value: 77.5 },
    { label: "Cosmos Super 64B (best open)", value: 73.7, highlight: true },
  ]}
/>

<Callout type="warn">
Read the SOTA claims narrowly. **Best *open* model is not best model** — Gemini 3.1 Pro (77.5) beats Cosmos
Super (73.7) on the reasoning benchmark, and Veo-3.1 leads on audio generation. Any "#1" leaderboard
position is a **dated snapshot** that moves as models ship. The text-to-image comparison uses a
**provider-selected harness** — a setup its authors chose, so treat the framing as favorable. And several
reported results use **best-of-N sampling with Cosmos's own reward model**, not single-shot generation;
that is a legitimate technique but not the same as raw one-shot quality.
</Callout>

None of that makes the work less interesting — it makes the *claim* precise. As an **open**, openly-licensed
omnimodal world model that fuses reasoning and generation in one attention operation and plans in image
space, Cosmos 3 is a genuinely new capability tier for people building on open weights. It just isn't the
best model in the world at everything, and the paper's own numbers say so if you read the parentheses.

## The take

The idea worth keeping is the **dual-stream joint attention**. Most "reasoning + generation" systems chain
two models and pay a text bottleneck between them; Cosmos makes the generator's queries attend over the
reasoner's keys inside one operation, so the plan reaches the pixels without being flattened into a prompt.
Pair that with **Action-CoT** — chain-of-thought as motion on the image plane — and you get a world model
whose reasoning is expressed in the same space its physics has to hold. The Mixture-of-Transformers is the
enabling structure: separate weights per modality, one shared sequence, fusion in attention. Whether the
scoped-SOTA numbers hold as the leaderboards churn is beside the point; the architecture is the
contribution, and it is a clean one.

---

*Built on **NVIDIA Cosmos 3** (arXiv 2606.02800; OpenMDW-1.1 license). The Reasoner/Generator MoT, dual-stream
joint attention, and Action-CoT are described in the paper (Figures 5 and 1); the interactive diagrams are
illustrations of the mechanism. Benchmark figures are quoted as scoped comparisons — best among open models,
with a closed frontier model (Gemini 3.1 Pro) still ahead — and some results use best-of-N with the
model's own reward model.*