# Satyajit Ghana — full content corpus
> Head of Engineering @ Inkers Technology. I build deep-learning systems, 3D perception, and high-performance infra.
> Generated from the content layer. Index: /llms.txt
---
# About Satyajit Ghana
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/about
I build deep-learning systems, 3D perception, and high-performance infra.
Head of Engineering at Inkers, where I work on industrial AI — 3D perception, deep learning, and the systems that ship it to production.
I write custom neural networks, CUDA kernels, LiDAR/point-cloud pipelines, and high-performance C++/gRPC services. Previously taught MLOps (EMLO 2.0) and computer vision (EVA 4.0) at The School of AI.
## Links
- github: https://github.com/satyajitghana
- linkedin: https://www.linkedin.com/in/satyajitghana/
- x: https://x.com/thesudoer_
- medium: https://satyajitghana.medium.com/
- scholar: https://scholar.google.com/citations?user=rZCRakQAAAAJ&hl=en
- website: https://thesatyajit.com/
- email: satyajitghana7@gmail.com
---
# Resume — Satyajit Ghana
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/resume
> pdf: https://ai.thesatyajit.com/satyajit-ghana-resume.pdf
> json: https://ai.thesatyajit.com/resume.json
```json
{
"contact": {
"name": "Satyajit Ghana",
"title": "Head of Engineering",
"company": {
"name": "Inkers Technology",
"url": "https://inkers.ai"
},
"location": "Bengaluru, India",
"email": "satyajitghana7@gmail.com",
"github": "https://github.com/satyajitghana",
"linkedin": "https://www.linkedin.com/in/satyajitghana/",
"website": "https://ai.thesatyajit.com"
},
"summary": "Engineering leader and deep-learning systems builder. I lead engineering for industrial-AI products — 3D perception, LiDAR/point-cloud pipelines, and structural-defect analysis — and ship them as high-performance C++/CUDA/gRPC services. Two USPTO patents pending. Previously taught MLOps and computer vision.",
"experience": [
{
"organization": "Inkers Technology",
"url": "https://inkers.ai",
"location": "Bengaluru, India",
"roles": [
{
"title": "Head of Engineering",
"start": "2024-01"
},
{
"title": "Deep Learning Software Engineer",
"start": "2022-06",
"end": "2024-01"
},
{
"title": "Deep Learning Associate",
"start": "2021-07",
"end": "2022-06"
}
],
"summary": "Lead engineering for industrial-AI products across 3D perception, LiDAR/point-cloud pipelines, and structural-defect analysis.",
"highlights": [
"Lead engineering for industrial-AI products: 3D perception, LiDAR/point-cloud pipelines, and structural-defect analysis.",
"Design custom neural networks, CUDA kernels, and high-performance C++/gRPC services for production deployment.",
"Named inventor on 2 USPTO patents pending (assigned to Inkers); grew from Deep Learning Associate to Head of Engineering."
]
},
{
"organization": "The School of A.I.",
"location": "Remote",
"roles": [
{
"title": "MLOps Instructor",
"start": "2022-01",
"end": "2023-01"
}
],
"summary": "Designed and taught MLOps and contributed to computer-vision curriculum.",
"highlights": [
"Designed and taught EMLO 2.0 — an end-to-end MLOps course (training, packaging, deployment, monitoring).",
"Contributed to EVA 4.0, the deep computer-vision program."
]
}
],
"education": [
{
"institution": "M.S. Ramaiah University of Applied Sciences",
"location": "Bangalore, India",
"degree": "B.Tech",
"end": "2021",
"highlights": [
"CGPA 9.78/10",
"Silver Medalist"
]
}
],
"skills": [
{
"name": "Deep Learning",
"skills": [
"PyTorch",
"TensorFlow",
"Computer Vision",
"3D / Point Clouds / LiDAR",
"GenAI (SDXL, LLMs)"
]
},
{
"name": "Systems",
"skills": [
"C++",
"C",
"Rust",
"CUDA",
"gRPC",
"MongoDB"
]
},
{
"name": "MLOps",
"skills": [
"Kubernetes",
"AWS",
"GCP",
"Docker"
]
},
{
"name": "Web",
"skills": [
"TypeScript",
"React",
"Next.js"
]
}
],
"projects": [
{
"name": "torch-point-ops",
"description": "High-performance PyTorch operators for point-cloud and 3D geometry processing.",
"url": "https://github.com/satyajitghana/torch-point-ops"
},
{
"name": "PV-LIO-for-HBA",
"description": "Point-to-voxel LiDAR-inertial odometry adapted for hierarchical bundle adjustment.",
"url": "https://github.com/satyajitghana/PV-LIO-for-HBA"
},
{
"name": "ige_lio",
"description": "Iterated error-state Kalman-filter LiDAR-inertial odometry experiments.",
"url": "https://github.com/satyajitghana/ige_lio"
},
{
"name": "sdxl-dreambooth-finetune",
"description": "DreamBooth fine-tuning pipeline for Stable Diffusion XL subject personalization.",
"url": "https://github.com/satyajitghana/sdxl-dreambooth-finetune"
},
{
"name": "TSAI-DeepVision-EVA4.0",
"description": "Coursework and experiments from The School of AI's EVA 4.0 computer-vision program.",
"url": "https://github.com/satyajitghana/TSAI-DeepVision-EVA4.0"
},
{
"name": "PadhAI-Course",
"description": "Implementations and notes from the PadhAI deep-learning course.",
"url": "https://github.com/satyajitghana/PadhAI-Course"
}
],
"publications": [
{
"title": "Adaptive Visual Learning Using Augmented Reality and Machine Learning Techniques",
"publisher": "Journal of Computational and Theoretical Nanoscience",
"date": "2020-01-01",
"details": "Vol. 17, No. 11, pp. 4952–4956",
"doi": "10.1166/jctn.2020.8982",
"url": "https://doi.org/10.1166/jctn.2020.8982"
}
],
"patents": [
{
"title": "Method and System for Performing Structural Defect Analysis in a Structural Environment",
"status": "Pending",
"applicationNumber": "US 19/634,310",
"filed": "2026-03-31",
"assignee": "Inkers Technology"
},
{
"title": "Data Acquisition Device",
"status": "Pending",
"applicationNumber": "US 19/634,339",
"filed": "2026-03-31",
"assignee": "Inkers Technology"
}
]
}
```
---
# Health — biomarker panel
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/health
> panel date: 2026-05-15
```json
{
"panelDate": "2026-05-15",
"stack": [
{
"type": "device",
"name": "Apple Watch",
"url": "https://www.apple.com/watch/"
},
{
"type": "device",
"name": "Withings Body+"
},
{
"type": "supplement",
"name": "Vitamin D3"
},
{
"type": "supplement",
"name": "Omega-3"
},
{
"type": "supplement",
"name": "Creatine"
},
{
"type": "supplement",
"name": "Magnesium Glycinate"
}
],
"biomarkers": [
{
"key": "ldl-c",
"label": "LDL-C",
"value": 128,
"unit": "mg/dL",
"category": "cardiovascular",
"weight": 2,
"optimalRange": {
"max": 100
},
"note": "Elevated — primary driver of atherosclerotic risk.",
"status": "elevated"
},
{
"key": "hdl-c",
"label": "HDL-C",
"value": 52,
"unit": "mg/dL",
"category": "cardiovascular",
"optimalRange": {
"min": 40
},
"status": "optimal"
},
{
"key": "apob",
"label": "ApoB",
"value": 105,
"unit": "mg/dL",
"category": "cardiovascular",
"weight": 1.5,
"optimalRange": {
"max": 90
},
"note": "Elevated — atherogenic particle count above target.",
"status": "elevated"
},
{
"key": "triglycerides",
"label": "Triglycerides",
"value": 88,
"unit": "mg/dL",
"category": "cardiovascular",
"optimalRange": {
"max": 150
},
"status": "optimal"
},
{
"key": "hba1c",
"label": "HbA1c",
"value": 5.4,
"unit": "%",
"category": "metabolic",
"weight": 1.5,
"optimalRange": {
"max": 5.7
},
"status": "optimal"
},
{
"key": "fasting-glucose",
"label": "Fasting Glucose",
"value": 96,
"unit": "mg/dL",
"category": "metabolic",
"optimalRange": {
"min": 70,
"max": 99
},
"status": "optimal"
},
{
"key": "fasting-insulin",
"label": "Fasting Insulin",
"value": 8.1,
"unit": "µIU/mL",
"category": "metabolic",
"optimalRange": {
"max": 8
},
"note": "Borderline — marginally above optimal fasting insulin.",
"status": "borderline"
},
{
"key": "alt",
"label": "ALT",
"value": 22,
"unit": "U/L",
"category": "liver_kidney",
"optimalRange": {
"max": 40
},
"status": "optimal"
},
{
"key": "ast",
"label": "AST",
"value": 24,
"unit": "U/L",
"category": "liver_kidney",
"optimalRange": {
"max": 40
},
"status": "optimal"
},
{
"key": "creatinine",
"label": "Creatinine",
"value": 0.95,
"unit": "mg/dL",
"category": "liver_kidney",
"optimalRange": {
"min": 0.7,
"max": 1.3
},
"status": "optimal"
},
{
"key": "egfr",
"label": "eGFR",
"value": 99,
"unit": "mL/min",
"category": "liver_kidney",
"optimalRange": {
"min": 90
},
"status": "optimal"
},
{
"key": "tsh",
"label": "TSH",
"value": 2.1,
"unit": "mIU/L",
"category": "hormonal",
"optimalRange": {
"min": 0.5,
"max": 4
},
"status": "optimal"
},
{
"key": "testosterone",
"label": "Testosterone",
"value": 610,
"unit": "ng/dL",
"category": "hormonal",
"optimalRange": {
"min": 300,
"max": 1000
},
"status": "optimal"
},
{
"key": "vitamin-d",
"label": "Vitamin D",
"value": 24,
"unit": "ng/mL",
"category": "nutritional",
"weight": 1.5,
"optimalRange": {
"min": 30,
"max": 100
},
"note": "Low — below the optimal 25-OH vitamin D range.",
"status": "low"
},
{
"key": "vitamin-b12",
"label": "Vitamin B12",
"value": 540,
"unit": "pg/mL",
"category": "nutritional",
"optimalRange": {
"min": 400,
"max": 1000
},
"status": "optimal"
},
{
"key": "ferritin",
"label": "Ferritin",
"value": 38,
"unit": "ng/mL",
"category": "nutritional",
"optimalRange": {
"min": 50,
"max": 300
},
"note": "Low — iron stores below the optimal floor.",
"status": "low"
},
{
"key": "hemoglobin",
"label": "Hemoglobin",
"value": 15.1,
"unit": "g/dL",
"category": "blood_panel",
"optimalRange": {
"min": 13.5,
"max": 17.5
},
"status": "optimal"
},
{
"key": "wbc",
"label": "WBC",
"value": 6.2,
"unit": "10³/µL",
"category": "blood_panel",
"optimalRange": {
"min": 4,
"max": 11
},
"status": "optimal"
},
{
"key": "platelets",
"label": "Platelets",
"value": 245,
"unit": "10³/µL",
"category": "blood_panel",
"optimalRange": {
"min": 150,
"max": 400
},
"status": "optimal"
},
{
"key": "resting-hr",
"label": "Resting HR",
"value": 58,
"unit": "bpm",
"category": "vitals",
"optimalRange": {
"min": 40,
"max": 60
},
"status": "optimal"
},
{
"key": "bp-systolic",
"label": "BP Systolic",
"value": 132,
"unit": "mmHg",
"category": "vitals",
"weight": 1.5,
"optimalRange": {
"max": 120
},
"note": "Elevated — systolic pressure above the optimal ceiling.",
"status": "borderline"
},
{
"key": "vo2max",
"label": "VO₂max",
"value": 47,
"unit": "mL/kg/min",
"category": "vitals",
"optimalRange": {
"min": 42
},
"status": "optimal"
}
]
}
```
---
# Now
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/now
```json
{
"updated": "2026-06-04",
"items": [
"Building ai.thesatyajit.com — a dual-native personal site maintained by a crew of Claude agents that ship every change as a reviewed PR.",
"Leading industrial-AI 3D perception at Inkers: turning LiDAR and camera capture into structural-defect analysis that runs in production.",
"Writing CUDA point-cloud kernels — fused nearest-neighbour, sampling, and grouping ops to take the Python side out of the hot path.",
"Reading deeply on LiDAR odometry and point-cloud registration, with an eye toward tighter, faster SLAM front-ends."
]
}
```
---
# Uses
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/uses
```json
[
{
"section": "Workstation",
"items": [
{
"name": "Custom GPU workstation",
"note": "NVIDIA RTX-class GPU for CUDA + training"
},
{
"name": "Linux",
"note": "Ubuntu — primary dev OS"
}
]
},
{
"section": "Editor",
"items": [
{
"name": "VS Code",
"note": "daily driver"
},
{
"name": "Claude Code",
"note": "agentic pair-programming in the terminal"
},
{
"name": "Fonts",
"note": "JetBrains Mono today, migrating toward IBM Plex Mono"
}
]
},
{
"section": "Terminal",
"items": [
{
"name": "zsh",
"note": "shell"
},
{
"name": "tmux",
"note": "session multiplexing"
}
]
},
{
"section": "Stack highlights",
"items": [
{
"name": "PyTorch + CUDA",
"note": "custom kernels for point-cloud ops"
},
{
"name": "C++ / gRPC",
"note": "high-performance perception services"
},
{
"name": "Next.js + Tailwind",
"note": "this site"
}
]
}
]
```
---
# Reading
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/reading
```json
[
{
"type": "paper",
"title": "PV-LIO: A Probabilistic Voxel-based LiDAR-Inertial Odometry Framework",
"status": "reading",
"note": "PLACEHOLDER — voxel-probabilistic front-end ideas for tighter LIO."
},
{
"type": "book",
"title": "Programming Massively Parallel Processors",
"author": "Hwu, Kirk, El Hajj",
"status": "reading",
"note": "PLACEHOLDER — the CUDA reference I keep coming back to."
},
{
"type": "paper",
"title": "FAST-LIO2: Fast Direct LiDAR-Inertial Odometry",
"status": "read",
"note": "PLACEHOLDER — iterated Kalman filter without feature extraction."
},
{
"type": "book",
"title": "Multiple View Geometry in Computer Vision",
"author": "Hartley & Zisserman",
"status": "queued",
"note": "PLACEHOLDER — geometry fundamentals refresher."
},
{
"type": "paper",
"title": "3D Gaussian Splatting for Real-Time Radiance Field Rendering",
"status": "queued",
"note": "PLACEHOLDER — splatting as a reconstruction primitive."
}
]
```
---
# Vritti UI
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/vritti
> stack: React 19, Next.js 16, Tailwind v4, Motion, Three.js, TypeScript
Components crafted for design engineers — animated components and blocks you copy
and paste into your shadcn/ui project. No package installs, full ownership.
```bash
npx shadcn@latest add "https://vritti.thesatyajit.com/r/shimmer-button"
```
**560+ components** across 14 categories — backgrounds, animations, text effects, buttons,
cards, charts, layouts, navigation, carousels, cursors, inputs, media, shaders, and the
genuinely weird ones. **240+ blocks** — pre-built page sections for real apps: auth,
pricing, e-commerce, billing, dashboards, modals, testimonials, footers, FAQ, AI & Web3.
Beyond components, it ships a set of **creative tools** that run online or install into
your project: Art Studio (9 generative-art tools), Background Studio, Shape Studio (an SVG
editor), Texture Studio (30+ stackable WebGL filters), Shader Studio (87 production-ready
shaders), Dither Studio, a visual Theme Editor (42 presets, Google Fonts, contrast
checker, CSS export), and native haptics. There's an interactive playground for tuning
component props live, and AI components (chat streaming, image generation, voice, AI
search) for LLM-native UIs.
Everything is copy-paste, shadcn-style — you own the source. It's the most ambitious thing
on this page: a full component ecosystem, a design-tool suite, and a theme system, built on
React 19, Next.js 16, Tailwind v4, Motion, and Three.js.
---
# torch-point-ops
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/torch-point-ops
> stack: PyTorch, CUDA, C++, Python
> repo: https://github.com/satyajitghana/torch-point-ops
A PyTorch library for 3D point operations — nearest-neighbour, sampling, and grouping
primitives — implemented as fused CUDA kernels for throughput on large point clouds.
Built to back LiDAR and 3D-perception pipelines where the Python-side ops were the
bottleneck. Exposes a clean `torch` autograd-compatible API over the native kernels.
---
# fabrik
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/fabrik
> stack: TypeScript, React, Vercel AI SDK, Zod, Motion
> repo: https://github.com/satyajitghana/fabrik
Most LLM apps are text in, text out. **fabrik** lets the model answer with *interface* —
it picks and fills real React components (cards, forms, charts, pickers) instead of
emitting a wall of prose. The AI decides what to show.
It's provider-agnostic: wire it to any model through the Vercel AI SDK (Gemini, OpenAI,
Anthropic, …), keep the API key server-side in a single route handler, and render the
streamed component tree on the client. Components are validated with Zod schemas, so the
model can only produce UIs you've defined — no arbitrary markup, no injection surface.
The bet behind it: as models get better at tool use, the next interface isn't a chat
bubble — it's a UI the model assembles on the fly for the task at hand.
---
# rentree
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/rentree
> stack: Next.js, React, TypeScript, Supabase, Drizzle ORM, Tailwind v4
> repo: https://github.com/satyajitghana/rentree
Rentree connects people with organic farms across India. Instead of buying produce off a
shelf, you rent a plot or adopt a tree: your farmer grows your food, sends real photo
updates as it grows, and ships it when it's ready.
Rent a plot for a season (tomatoes, herbs, rice — whatever grows), or adopt an apple,
orange, or mango tree and get the whole harvest. Every tree carries a QR code, so you can
scan and trace the full journey from planting to your plate. A clean Next.js 16 app on
Supabase + Drizzle, with the provenance trail as the trust mechanism that makes the model
work.
---
# Quantum Orbital Visualizer
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/atoms
> stack: Next.js, Three.js, Rust (WASM), C++23 (WASM), Tailwind v4, Zustand
> repo: https://github.com/satyajitghana/atoms
Each visualization is the probability density |ψ|² of a hydrogen-like electron, computed
from the exact solutions to the Schrödinger equation and rendered as up to 500K glowing
particles in real time. Drive the quantum numbers (n, l, m) and watch the probability
cloud morph as you go.
The fun engineering bit is the **dual WASM backends**: the same physics is implemented
three ways — JavaScript, Rust→WASM (~41 KB), and C++23→WASM (~28 KB) — and you can switch
the compute engine at runtime. Same orbitals, different implementations, side-by-side for
benchmarking.
There's also an Element Explorer: the full periodic table, where clicking any of the 118
elements parses its electron configuration into individual orbitals you can then visualize
one by one.
---
# BOTCHA
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/botcha
> stack: Next.js, TypeScript, Redis, crypto (SHA-256 / HMAC / JWT)
> repo: https://github.com/satyajitghana/botcha
Only an autonomous agent with runtime access to HTTP, cryptography, and byte manipulation
can pass — which is the whole joke, and the whole point.
Each challenge is 256 random bytes plus 2–4 byte-level transformation steps written in
randomized natural language, inside a 30-second window: fast enough for a machine,
hopeless for a human copy-pasting into a REPL. The agent must decode the base64, execute
each transform in order, concatenate the raw byte outputs, `SHA-256` the result, then
`HMAC-SHA256(key=nonce, message=answer)` and submit both — proving it actually did the
computation. On success it gets a short-lived JWT.
It's a small, sharp take on a real question the agent era raises: if the web increasingly
wants to *let bots in* and keep humans out, what does that gate look like?
---
# penora
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/penora
> stack: TypeScript, React, Canvas, npm package
> repo: https://github.com/satyajitghana/penora
Type a string and penora animates it into natural handwriting — not a font fade-in, but
stroke-by-stroke drawing driven by the glyph contours, with pen physics layered on top:
pressure tapering at stroke ends, seeded jitter so each render is subtly different, and
micro-wobble for that hand-drawn quality. Export the result as video or GIF.
Shipped as an npm package (`npm i penora`) and as a shadcn registry component you can drop
straight into a project. It's the kind of small, self-contained library that's satisfying
to build: a tight problem (make text look handwritten and *alive*) with a lot of room for
craft in the details.
---
# Hyr
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/hyr
> stack: Next.js, TypeScript, Tailwind v4, Google Gemini
> repo: https://github.com/satyajitghana/hyr
Drop in any PDF resume and Hyr parses it into a structured, fully editable format —
inline editing for every section, no forms or modals. From there it tailors the resume
per job posting, aligning keywords and tone to the listing so it gets past ATS screening
and reads as a real fit.
Build, tailor, optimize, and track the whole job search in one place. The AI work runs on
Gemini behind a server route; the parsing-to-structured-data step is the part that makes
the rest of the editing experience feel clean rather than fighting a PDF.
---
# MockLab
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/mocklab
> stack: Next.js, Tailwind v4, shadcn/ui, Motion, html-to-image
> repo: https://github.com/satyajitghana/mocklab
MockLab generates pixel-perfect social mockups across 21+ platforms — social posts (X,
LinkedIn, Instagram post/story, Reddit, Threads, YouTube, TikTok), chat messages
(WhatsApp, Telegram, Slack, Discord, iMessage, Snapchat), and Gmail. Edit on the left,
watch the live preview on the right, download as 2× PNG.
Everything is configurable: verified badges, reactions, timestamps, read receipts, media
uploads, and per-message editing for the chat mockups — all with dark/light theming for
both the app and every mockup. It's a deceptively deep UI project: each platform is its
own little pixel-accurate design system to reproduce.
---
# tapcn
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/projects/tapcn
> stack: React Native, Expo, TypeScript, CLI
> repo: https://github.com/satyajitghana/tapcn
tapcn brings the shadcn model to mobile: not an npm component library, but a CLI that
copies beautifully-designed, accessible component *source* directly into your Expo
project. No dependency lock-in, no version conflicts — just code you control and can edit.
```bash
npx @tapcn/cli init
npx @tapcn/cli add button card input text
```
Components work on iOS, Android, and Web from a single source. The whole appeal of the
shadcn approach — own your components, theme them freely, no black-box library — has been
missing on React Native; tapcn fills that gap.
---
# My site's chatbot was stuffing 273k tokens into every message. I gave it tools instead.
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/blog/site-agent-dynamic-tools
> date: 2026-07-20
> tags: agents, llm, tools, retrieval, systems
The assistant embedded on this site — the `⌘K` console, `/api/ask`, and the `ask_satyajit`
MCP tool — worked. It was also doing something faintly absurd. Every single request built its
system prompt like this:
```ts
// the old lib/chat.ts, abridged
const sections = []
for (const page of ["about", "resume", "health", "now", "uses", "reading"])
sections.push(await dataPageMarkdown(page))
for (const item of getAllContent()) // every article, blog, log, digest…
sections.push(contentMarkdown(item.kind, item.slug))
return [PERSONA, RULES, "=== CONTENT CORPUS ===", sections.join("\n---\n")].join("\n")
```
It pasted **the entire site** into the prompt and let the provider's context cache eat the cost.
That's a defensible move when the corpus is a page or two. Mine isn't anymore:
```text
old full-corpus system prompt: 1,090,854 chars ≈ 273,000 tokens
```
The code even carried a comment — *"if the corpus ever approaches ~100K tokens, switch to
retrieval"* — that reality had quietly sailed past nearly 3× over. Two hundred seventy-three
thousand tokens is past the input window of the model serving it. The chat was running on
whatever survived truncation. Time to do the thing the comment said.
## The fix: retrieve, don't dump
I rebuilt the assistant as a small **tool-using agent**. The system prompt now carries only a
compact **catalog** — every page's `kind/slug`, title, and one-line description, about **5k
tokens** — and the agent pulls the bodies it actually needs through tools. Three of them:
```ts
// lib/chat.ts — the whole tool set
search_content({ query, limit }) // BM25 over the site (ranked, returns section + snippet)
get_content({ kind, slug }) // fetch one page's full markdown, on demand
think({ thought }) // a reasoning scratchpad; records the thought, returns nothing
```
A question now flows: read the catalog → `search_content` (or jump straight to `get_content` if
the catalog already names the page) → read one or two pages → optionally `think` to reconcile
them → answer, with citations. The base prompt is fixed and small; the variable cost is the
one-to-three pages it fetched, not the other sixty-five it didn't.
```text
before: ~273k tokens, every request, whether relevant or not
after: ~5k catalog + ~2–10k per page actually fetched
```
That's the same **dynamic loading** idea from
[Kimi K3's tool-calling guide](https://platform.kimi.ai/docs/guide/kimi-k3-tool-calling-best-practice) —
their point is that a big upfront payload "eats up context and makes the model more likely to
pick the wrong thing." Kimi loads *tool schemas* on demand; my corpus is the payload, so I load
*content* on demand. Same principle, one layer down.
## Three tools, on purpose
The tool count is a decision, not an accident. Mario Zechner's
[pi coding agent](https://mariozechner.at/posts/2025-11-30-pi-coding-agent/) makes the case that
"four tools are all you need" — `read`, `write`, `edit`, `bash` — and that MCP servers which
"dump their entire tool descriptions into your context on every session" are the anti-pattern.
A read-only site agent needs fewer still: search, fetch, think. Each description is a couple of
sentences. The combined tool surface is under a thousand tokens, so keeping it declared upfront
(rather than lazily loading schemas, which only pays off with dozens of tools) is the right call
here — the honest version of "dynamic tools" for a small surface is *don't have a big one*.
The one genuinely new tool is `think`, from Anthropic's
[think-tool post](https://www.anthropic.com/engineering/claude-think-tool). It does nothing —
literally logs the thought and returns:
```ts
think: tool({
description: "Think out loud: plan which pages to fetch, or check a draft answer against the sources you read. Records the thought and returns nothing new.",
inputSchema: z.object({ thought: z.string() }),
execute: async ({ thought }) => ({ ok: true, thought }),
}),
```
That looks pointless until you watch a multi-step tool run. Between `search_content` and the
final answer, the model has raw tool output sitting in context and no designated place to reason
over it before committing. `think` is that place — a scratchpad that keeps the "what did I just
read, and does it actually answer the question" step from being skipped. Anthropic reports it
buying a large margin on multi-step tool tasks; the appeal for a retrieval agent is exactly that
it makes the model *check the page it fetched* instead of answering from the snippet.
## The loop
The agent loop itself is one call, courtesy of the Vercel AI SDK — `stopWhen` bounds how many
tool round-trips it may take before it has to answer:
```ts
const result = streamText({
model: chatModel("main"),
system, // persona + rules + the 5k-token catalog
messages,
tools: agentTools(), // search_content, get_content, think
stopWhen: stepCountIs(6), // search → read → think → answer is ~4; 6 is headroom
})
```
The same tool set backs all three surfaces — the streaming `⌘K` console (`/api/chat`), the
one-shot `/api/ask`, and the `ask_satyajit` MCP tool — so there's one harness, not three. And
`search_content` is the [Contextual BM25](/blog/site-search-contextual-bm25) engine I'd already
built for the site's `/search`, now doing double duty as the agent's retriever. The pieces
compose: the search work made the harness possible.
Honest scope. This is retrieval over a small, single-author corpus, not a coding agent — the
lessons transfer but the stakes are lower. `think`'s benefit is task-dependent (Anthropic saw
big gains on policy-heavy multi-step tasks, near-zero on simple ones); on a two-hop lookup it
mostly earns its keep by stopping the model from answering off the snippet. And the retriever is
lexical BM25 — no embeddings yet, so a pure paraphrase with no shared terms can still miss the
right page. The catalog is the safety net there: the model can see every title even when search
comes up short.
## The take
The old design wasn't wrong when it was written — it was wrong at 273k tokens. The rebuild is the
[agent-harness](/articles/agent-harness) framing applied to my own site: the model barely changed,
but the loop, the tool set, and the context policy around it changed completely. Give the agent a
map and three tools and let it fetch, instead of force-feeding it the whole library and hoping the
answer survives the truncation. Retrieve, don't dump.
---
# Giving my own search the Contextual BM25 treatment
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/blog/site-search-contextual-bm25
> date: 2026-07-20
> tags: search, retrieval, bm25, rag, information-retrieval
I recently wrote two pieces back to back: one on [BM25](/articles/bm25), the ranking
function that refuses to die, and one on [Contextual Retrieval](/blog/contextual-retrieval),
Anthropic's fix for chunks that forget where they came from. Then I opened `lib/search.ts` —
the thing powering this site's own `/search` and the `/api/search` endpoint agents hit — and
found this:
```ts
// the old lib/search.ts, abridged
for (const item of getAllContent()) {
const fields = [item.title, item.description, item.tags, item.body]
for (const text of fields) {
if (text.toLowerCase().includes(q)) { // q = the whole lowercased query
results.push({ /* … a ±120-char window around the hit */ })
break
}
}
}
```
A substring `indexOf`. It has three problems, and the third is the one that actually bites:
1. **No ranking.** The first document that contains the substring wins, in date order. A tight,
on-topic match and an incidental mention are indistinguishable.
2. **No notion of rarity or length.** Matching "the" counts the same as matching "kalman".
3. **Multi-word queries fall off a cliff.** `q` is the *entire* query string, so
`includes(q)` needs that **exact phrase** somewhere in the text. Nobody searches in verbatim
phrases. `"bm25 length normalization"` returns **zero results** — those three words are all
in the BM25 article, just never contiguous.
So I ate my own cooking. Here's the rebuild.
## BM25, the same formula I just wrote up
The [BM25 walkthrough](/articles/bm25) has the whole derivation; the code is a direct
transcription of it. Lucene's non-negative IDF, term-frequency saturation at `k1 = 1.2`, length
normalization at `b = 0.75`:
```ts
// lib/search.ts — the ranking core, straight from the article's formula
const K1 = 1.2
const B = 0.75
function idf(term: string, ix: Index): number {
const n = ix.df.get(term) ?? 0
return Math.log(1 + (ix.n - n + 0.5) / (n + 0.5))
}
// per query term present in a chunk:
const f = chunk.tf.get(term) ?? 0
const norm = K1 * (1 - B + (B * chunk.len) / ix.avgdl)
score += idf(term, ix) * (f * (K1 + 1)) / (f + norm)
```
Two things fall out for free. The query is **tokenized** into terms and each is scored
independently, so multi-word queries just work — no phrase has to exist. And rare terms
dominate: `kalman` carries far more weight than `filter`, because `idf` collapses for common
words. No stopword list, same as the article promised.
I keep a **postings map** (`term → chunk indices`) so a query only touches chunks that actually
share a term, instead of scanning the whole corpus. The corpus is one person's writing, so this
is overkill — but it's the same inverted-index shape a real engine uses, and it's three lines.
## Contextual chunks, minus the Claude call
BM25 alone still has the chunk-amnesia problem from the [Contextual Retrieval
post](/blog/contextual-retrieval): if I split an article into passages and index each one alone,
a passage that reads "it drops 41% at parity" has no idea it's *about the Harness Effect, in the
section on the controlled swap*. A query for either misses it.
Anthropic's fix is to have Claude write a one-line context for every chunk before indexing.
Mine is cheaper and dumber: the context a chunk lost is sitting right there in the document's
frontmatter and headings. So before indexing, every chunk inherits its document's **title,
description, tags, and nearest heading** — deterministically, at request time, no model, no
build step:
```ts
// each chunk is packed with weighted terms: its own body, its heading, and the
// document context it would otherwise have lost when the body was split.
const contextTokens = tokenize([title, description, tags.join(" ")].join(" "))
for (const t of bodyTokens) add(t, 1)
for (const t of headingTokens) add(t, 1) // section-local: full weight
for (const t of contextTokens) add(t, CONTEXT_WEIGHT) // 0.5 — present, not dominant
```
The honest caveat: this is a **poor man's** Contextual Retrieval. Claude's per-chunk context can
say things the frontmatter can't ("the previous quarter's revenue was \$314M"); my version can
only replay the structural context the document already carries. And prepending the *same* title
to every chunk of a document inflates those terms' document-frequency, which is exactly why the
context terms get `CONTEXT_WEIGHT = 0.5` instead of full weight — present enough to make the
chunk findable by its subject, quiet enough not to drown the passage's own words. It's the 80%
of the win for 0% of the inference cost, which is the right trade for a static personal site.
Each result also reports **which section** it matched — the nearest heading rides along as the
result's `field`, so `/api/search` tells an agent not just *which* document but *where* in it.
## Before / after
Same queries, old substring search vs. the new Contextual BM25, on the live corpus:
```text
query substring indexOf contextual BM25 (top hit)
--------------------------------------------------------------------------------------
"bm25 length normalization" 0 results (no verbatim articles/bm25
phrase) [Start with TF-IDF, and its two flaws]
"revenue grew quarter" 0 results blog/contextual-retrieval
[The problem: chunks forget …]
"kalman filter" first dated doc that articles/kalman-filter (8.4)
contains the phrase, + fast-lio2 (8.6, cites it)
unranked ranked by relevance, not date
"reciprocal rank fusion" 0 results blog/contextual-retrieval
[A dependency-free repro]
```
The multi-word queries are the story. Under substring, anything that isn't a verbatim phrase —
which is almost everything a person types — returned nothing. Under BM25 every term contributes,
so the query finds the document even when its words are scattered across a paragraph, and the
contextual prefix pulls in matches on the *subject* of a passage, not just its literal words.
## It's live
Try it: [`/search?q=length+normalization`](/search?q=length+normalization). Agents get the
ranked, scored JSON with the section label:
```bash
curl -s "https://ai.thesatyajit.com/api/search?q=contextual%20retrieval&limit=3" | jq '.results[] | {slug, field, score}'
```
What this still isn't: it's the **lexical half** only. The [contextual retrieval
post](/blog/contextual-retrieval) makes the case that the real win is *fusing* BM25 with a dense
retriever, and that reranking the shortlist is what takes the failure rate the last mile. There
are no embeddings here yet — a query has to share actual terms with a chunk, so a pure paraphrase
with no lexical overlap can still miss. For a corpus this size, lexical BM25 over contextualized
chunks is the honest 90% solution; the semantic half is the next commit, not this one.
## The takeaway
The whole change is `lib/search.ts` — a few hundred lines, no dependencies, no index server, no
model at request time. It's the two ideas I'd just written about, applied to the smallest
possible target: my own search bar. BM25 for ranking, structural context for the chunk-amnesia
problem, and an honest note about the half I haven't built yet. Writing about a technique is a
good way to understand it; running it in your own site is a better one.
---
# Contextual Retrieval, with a runnable repro and a browser playground
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/blog/contextual-retrieval
> date: 2026-07-17
> tags: rag, retrieval, claude, embeddings, playground
[Anthropic's Contextual Retrieval post](https://www.anthropic.com/engineering/contextual-retrieval)
has been on my to-read list for a while. It targets the oldest bug in RAG, so I finally sat down,
built a dependency-free reproduction, and wired up a browser playground. This is the write-up.
## The problem: chunks forget where they came from
Standard RAG splits documents into chunks and indexes each chunk on its own. That destroys context.
Anthropic's example is perfect:
> `The company's revenue grew by 3% over the previous quarter.`
Which company? Which quarter? The chunk can't say. A query like *"how did ACME's Q2 revenue change?"*
has **no terms to match** against that chunk — the words "ACME" and "Q2 2023" live in the *document*,
not the *chunk*. Both lexical (BM25) and semantic (embedding) retrieval miss it.
## The fix: let Claude situate each chunk
Contextual Retrieval prepends a short, chunk-specific context to each chunk **before** you embed it and
**before** you build the BM25 index — Anthropic calls these **Contextual Embeddings** and **Contextual
BM25**. The context is generated by Claude, given the whole document. The same chunk becomes:
> `This chunk is from an SEC filing on ACME corp's performance in Q2 2023; the previous quarter's revenue was $314 million. The company's revenue grew by 3% over the previous quarter.`
Now the owner and the period are *in the chunk*, so retrieval can find it.
## Watch it happen
Here is the whole pipeline in your browser — the document, its chunks, the chunks with context prepended,
then retrieval. The scoring is a real BM25 and a TF-IDF cosine (a stand-in for an embedding model); the
contextual prefixes stand in for Claude's output. Step to **4 · retrieve**, pick a query, and watch the
right chunk climb from the standard column to the contextual one:
Toggle between BM25, embeddings, and their fusion. The pattern holds across all three: the answer chunk
is buried on standard chunks and near the top once each chunk carries its context.
## The one prompt that does the work
This is the prompt from the post, verbatim. It runs once per chunk, with the full document supplied as
context:
```text
{{WHOLE_DOCUMENT}}
Here is the chunk we want to situate within the whole document
{{CHUNK_CONTENT}}
Please give a short succinct context to situate this chunk within the overall document for the purposes of improving search retrieval of the chunk. Answer only with the succinct context and nothing else.
```
The obvious worry is cost: you re-send the whole document once per chunk. **Prompt caching** kills that.
Cache the document once and every chunk in it reads from the cache, which is why Anthropic quotes a
one-time **$1.02 per million document tokens**. The document is the stable prefix, so it takes the
`cache_control` breakpoint; the chunk and instruction vary and come after it:
```python
from anthropic import Anthropic
client = Anthropic()
CONTEXT_PROMPT = """Here is the chunk we want to situate within the whole document
{chunk}
Please give a short succinct context to situate this chunk within the overall document \
for the purposes of improving search retrieval of the chunk. Answer only with the \
succinct context and nothing else."""
def situate(doc: str, chunk: str) -> str:
resp = client.messages.create(
# One Claude call per chunk. For a large index you'd typically drop to a
# cheaper model like claude-haiku-4-5 — which is what Anthropic's ~$1.02 /
# 1M-doc-token estimate assumes — trading a little context quality for cost.
model="claude-opus-4-8",
max_tokens=200,
messages=[{
"role": "user",
"content": [
# The whole document is the stable prefix: cache it ONCE, then every
# chunk of this document reads from the cache instead of re-paying for it.
{"type": "text",
"text": f"\n{doc}\n",
"cache_control": {"type": "ephemeral"}},
{"type": "text", "text": CONTEXT_PROMPT.format(chunk=chunk)},
],
}],
)
return "".join(b.text for b in resp.content if b.type == "text").strip()
```
Verify the cache is actually working: `resp.usage.cache_read_input_tokens` should be non-zero on every
chunk after the first for a given document. If it's zero, something upstream is mutating the document
bytes (a timestamp, non-deterministic JSON) and invalidating the prefix.
## A dependency-free repro
To convince myself the lift was real and not marketing, I wrote a ~180-line pure-stdlib script: naive
chunking, BM25 from scratch, a TF-IDF cosine as an embedding stand-in, and reciprocal rank fusion. The
retrieval core is small — here is BM25 and the fusion step:
```python
class BM25:
def score(self, query, i):
dl, tf, s = len(self.docs[i]), self.tf[i], 0.0
for t in query:
if t not in tf:
continue
num = tf[t] * (self.k1 + 1)
den = tf[t] + self.k1 * (1 - self.b + self.b * dl / self.avgdl)
s += self.idf(t) * num / den
return s
def rrf(rankings, k=60): # reciprocal rank fusion of BM25 + embedding rankings
scores = Counter()
for ranking in rankings:
for rank, item in enumerate(ranking):
scores[item] += 1 / (k + rank + 1)
return [i for i, _ in scores.most_common()]
```
Then I index the chunks two ways — plain, and with a one-line context prepended — and measure where the
correct chunk lands for a couple of "which entity, which period" queries. The actual output:
```text
query method plain rank ctx rank
------------------------------------------------------------------------------------
How did ACME Corp revenue change in Q2 2023? bm25 4 2
How did ACME Corp revenue change in Q2 2023? emb 4 1
How did ACME Corp revenue change in Q2 2023? hybrid 4 2
What happened to Beta Industries revenue in Q3 2023? bm25 5 2
What happened to Beta Industries revenue in Q3 2023? emb 5 1
What happened to Beta Industries revenue in Q3 2023? hybrid 5 2
recall@1 (fraction of queries where the right chunk ranks #1):
bm25 plain 0/2 contextual 0/2
emb plain 0/2 contextual 2/2
```
Contextualizing the chunks moves the answer from rank **4–5** to rank **1–2** across BM25, embeddings, and
their fusion — and embeddings recall@1 goes from **0/2 to 2/2**. On this toy corpus fusion lands the answer
at #2 rather than #1 (a small-N artifact of RRF), which is a good reminder that the fusion win is an
*aggregate* effect — which is exactly what Anthropic's real evaluation measures.
## What the real evaluation found
On Anthropic's benchmark (top-20 retrieval failure rate, i.e. `1 - recall@20`), stacking the techniques
compounds:
- **Contextual Embeddings** alone: `5.7% → 3.7%` — a **35%** cut in the failure rate.
- **+ Contextual BM25**: `5.7% → 2.9%` — **49%**.
- **+ reranking**: `5.7% → 1.9%` — **67%**.
The lexical half matters more than you'd guess — BM25 nails exact identifiers (error codes, ticker symbols,
function names) that embeddings smear together, so contextualizing *both* indexes and fusing them beats
either alone.
## Things worth copying from the post
- **Retrieve top-20, not top-5/10.** Anthropic found 20 the most performant cut for the final context.
- **Rerank the shortlist.** Retrieve ~150 candidates, then rerank down to 20 for the answer prompt — that's
the step that takes the failure rate from 2.9% to 1.9%.
- **Embedding model matters.** Gemini and Voyage embeddings were the standouts in their tests.
- **Chunking still matters.** Size, boundary, and overlap all move the numbers — Contextual Retrieval sits
on top of good chunking, it doesn't replace it.
- **A domain-tuned context prompt beats the generic one.** The template above is a floor, not a ceiling.
Be honest about what this costs. Contextual Retrieval adds a Claude call **per chunk** at index time —
cheap per token with caching, but real latency and spend when you're indexing millions of chunks, and it
has to re-run when documents change. My repro is a minimal reproduction of the *core mechanism* (context
lifts rank); the `35 / 49 / 67%` figures are Anthropic's, on their corpus. And my playground scores with a
TF-IDF cosine, not a real embedding model — it shows the *shape* of the effect, not production numbers.
## The takeaway
The move is almost embarrassingly simple: spend a cheap, cached Claude call per chunk to write down the
context a human would need to make sense of it, then index that. It attacks the failure at its source
instead of papering over it downstream, it helps lexical and semantic retrieval at the same time, and — as
the little repro above shows — you can watch the right chunk climb the rankings the moment the context goes
in.
---
*Source: [Introducing Contextual Retrieval](https://www.anthropic.com/engineering/contextual-retrieval)
(Anthropic). The prompt and the `35 / 49 / 67%` numbers are theirs; the repro and the browser playground are
mine, and the playground's scoring runs entirely client-side.*
---
# Qwen-BIM: a 14B model that matches 671B DeepSeek-R1 — by knowing its domain
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/blog/qwen-bim-domain-beats-scale
> date: 2026-06-09
> tags: llm, fine-tuning, domain-models, bim, paper-notes
Here's the headline from [Qwen-BIM](https://arxiv.org/abs/2602.20812) (Lin et al.,
Tsinghua, Feb 2026): a fine-tuned **14B** model scores **0.83 on G-Eval** for
BIM-based design tasks — essentially tied with **DeepSeek-R1 at 671B** (0.84), and
ahead of its own 72B sibling. A model ~48× smaller, matching the frontier on a
specific domain.
That result is not surprising on its own — "fine-tune a small model on your domain"
is folklore by now. What makes the paper worth reading is the *anatomy*: where exactly
general LLMs fall over on engineering work, and which one design choice did most of the
lifting. I work on industrial AI at [Inkers](https://inkers.ai), so domain models over
3D/BIM data are close to home. These are my notes.
## The actual problem: a BIM model isn't text
A Building Information Model is a structured graph of components — walls, slabs, beams,
each with geometry, materials, and relationships. An LLM can't read it. So step one of
*any* LLM-on-BIM pipeline is an unglamorous one the field mostly skips past: **turn the
model into text.**
The authors do exactly that, carefully. Revit models of five building types (malls,
offices, dormitories, teaching buildings, museums) are sliced into spatial blocks of
~10–15 components each, defects are injected, and each block is serialized to plain text
plus 22 templated questions with **hard-coded reference answers**. That last detail
matters: the ground truth is computed by rules, not by another model, so the benchmark
isn't measuring one LLM against another LLM's opinion.
The questions ladder up in difficulty on purpose: from "list the wall IDs"
(extraction) through "compute each slab's area" (calculation) to "is this wall
thickness suspicious given residential norms?" (domain reasoning). It's a clean way to
see *which rung* a model falls off.
The whole pipeline, end to end, is just: project the structured model into text, turn it
into supervised Q&A, add reasoning traces, and LoRA-fine-tune a small open model on it.
## Where general LLMs actually fail
They evaluated 11 general models (ChatGLM, Qwen, DeepSeek — including the 671B
DeepSeek-V3/R1). The failure modes are specific and, honestly, familiar from any
engineering-LLM project:
- **Arithmetic.** Asked for a slab's planar area, Qwen-max picks the *right formula* and
still returns the wrong number. The bottleneck isn't understanding — it's calculation.
- **Natural-language literalism.** Models misread parentheses in the answer template, or
a naming rule ("wall IDs start with Q"), and confidently apply the wrong transform.
- **Missing domain knowledge.** Asked to infer a building's floor height, the 14B base
model reasons that floor height ≈ slab thickness (120 mm) — coherent chain of thought,
wrong mental model, because it was never taught what "floor height" means in practice.
The pattern: general models clear extraction and counting, then degrade sharply on
calculation, multi-step reasoning, and anything needing design common sense. On the
domain-specific design-review tasks, G-Eval is mostly **below 0.8** — not reliable
enough to trust.
## The one choice that mattered: reasoning supervision
This is the part I'd underline. They built two datasets from the same BIM text:
- **BIM-QA** — 2,129 plain question→answer pairs.
- **BIM-QRA** — 1,364 question→**reasoning**→answer triples, where the intermediate
steps are supervised, not just the final answer.
Then they LoRA-fine-tuned Qwen2.5-14B on different mixes. The result is the kind of
finding that should change how you build these datasets:
| Fine-tuning data | Size | G-Eval |
|---|---|---|
| 100% QA | 2,129 | 0.69 |
| 80% QA + 20% QRA | 2,661 | 0.77 |
| 60% QA + 40% QRA | 2,500 | 0.77 |
| **100% QRA** | **1,364** | **0.83** |
The **smallest** dataset — pure reasoning triples — won, by a wide margin. More
reasoning supervision monotonically improved G-Eval, and quality beat quantity outright.
Teaching the model *how to get there*, on a third of the data, beat teaching it *what the
answer is* on the full set.
## Bigger is not better (and the paper shows it twice)
Two clean data points against scale-maximalism:
1. On the general benchmark, **QwQ-32B out-scored DeepSeek-R1 (671B)** on G-Eval. The
giant model's verbose reasoning actually *hurt* — it padded answers with redundant
text, tanking format and text-similarity scores without improving correctness.
2. After fine-tuning, **Qwen-BIM (14B) matched DeepSeek-R1 (671B)** and beat the 72B and
32B Qwen models on the domain G-Eval.
The improvement from fine-tuning is also *targeted* exactly where you'd want it:
| G-Eval | Base 14B | Qwen-BIM | Δ |
|---|---|---|---|
| General tasks | 0.810 | 0.874 | +0.06 |
| Domain-specific tasks | 0.588 | 0.801 | **+0.21** |
| Overall | 0.689 | 0.834 | +0.15 |
General ability barely moved (it was already fine); the entire gain is concentrated in
the domain tasks that were broken. That's the signature of fine-tuning doing the right
thing — adding domain competence without trading away the base model's generality.
## What I'd flag
It's a careful paper, but keep the scope honest:
- **2D only.** Early tests showed the models couldn't do 3D geometry (collision
detection), so those questions were cut. The hard part of real BIM reasoning is 3D.
- **One narrow task family**, five building types, rule-generated Q&A. G-Eval is an
LLM-as-judge metric — better-correlated with humans than BLEU/ROUGE here, but still a
proxy. "Data available on request" rather than released.
- The "matches 671B" comparison is on *this* benchmark. It's a domain-competence claim,
not a general-capability one.
## Why it's the right playbook anyway
Strip away the BIM specifics and this is a template for industrial domain models, the
kind I think about constantly: you rarely need a frontier model. You need (1) a faithful
**text projection of your structured/3D data**, (2) a benchmark with **rule-computed
ground truth** so you're measuring competence, not vibes, and (3) **reasoning-supervised**
fine-tuning data — quality and chain-of-thought over raw volume. Get those three right
and a 14B model on two A6000s reaches the same place a 671B model does, at a fraction of
the inference cost. For anyone shipping AI into a real engineering vertical, that economics
is the whole game.
---
*Paper: [Developing large language model for BIM-based design with domain-specific
benchmark and dataset](https://arxiv.org/abs/2602.20812) — Lin, Cai, Ni, Zhou, Pan
(2026), arXiv:2602.20812.*
---
# This site is managed by Claude
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/blog/hello-world
> date: 2026-06-03
> tags: meta, ai, nextjs
GitHub READMEs are dead. After Claude and the wave of coding agents, your homepage
isn't a static profile — it's a living, agent-readable artifact you can hand to an LLM.
This site is **dual-native**: every page is both a clean human document and a
machine-readable surface. Try fetching [`/blog/hello-world.md`](/blog/hello-world.md)
or [`/llms.txt`](/llms.txt) — an agent gets structured text, you get the rendered page.
The whole site is maintained by a crew of Claude agents. New posts, logs, and
data updates are authored by skills that validate themselves before shipping.
## What's under the hood
The content layer is a single source of truth: MDX files validated with Zod, surfaced
identically to humans (this page), to agents (the `.md` variant), and to tools
(the MCP server). More on that soon.
---
# AA-3D: what 10,000 blind votes on three.js scenes can and cannot tell apart
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/aa-3d
> date: 2026-10-13
> tags: benchmarks, agents, agentic-coding, 3d, pricing
For a year the way people compared models at 3D has been a screenshot thread. Someone asks for a pelican riding a
bicycle, or a spinning black hole, or a playable pinball table, posts what three models made, and the replies argue
about lighting. It is fun and it measures nothing, because nobody controls the prompt, the tools, the time or who is
judging.
Yesterday Artificial Analysis [turned that into a benchmark](https://x.com/ArtificialAnlys/status/2109791341802885209).
AA-3D gives 21 models the same 55 written briefs, the same offline sandbox and the same three-hour clock, has each one
build a three.js web page, and asks a recruited panel to pick between two scenes built from the same brief without
knowing who made them. The votes become an Elo rating. Claude Sonnet 5.5 is first at 1194, and the post's first finding
is that "four models are statistically tied at the top: Claude Sonnet 5.5 (1194), Claude Opus 5.5 (1190), GPT-6 Astra
(1165), and Claude Fable 5.1 (1131)."
I wanted to know how much a leaderboard like this can actually tell apart. A human-preference arena is a statistics
problem wearing a nice interface: everything rests on a Bradley-Terry fit and the intervals around it. AA publishes more
than most arenas do. The [methodology page](https://artificialanalysis.ai/methodology/aa-3d) names the estimator,
the [leaderboard](https://artificialanalysis.ai/evaluations/aa-3d) embeds every model's rating, interval, win rate by
domain, cost and tool use in the page, and the [launch article](https://artificialanalysis.ai/articles/aa-3d) lists
every scene. That was enough to rebuild the fit and check the claims against AA's own numbers.
Most of it holds up, and the analysis around the leaderboard is better than the leaderboard. Three things don't hold up so well.
The "four tied" sentence uses overlapping intervals as the test, which is the wrong one, and AA's own rank-range column
disagrees with it. The correlation with general intelligence is real but leans on three very weak models. And a
model that fails to produce a scene loses nothing for it.
## What a model is asked to build
The task set is 55 briefs across seven domains: landmarks and real places (10), machines (9), natural phenomena (9),
living bodies (8), games, sports and piloting (8), space (6), and stories and fantasy (5). The briefs are deliberately
the kind of thing people ask for: the Colosseum "from a one-line request, as a typical user would ask for it", a
counterweight trebuchet you load and fire at a castle, a beer pour with a developing foam head, a Kerr black hole with
"the cinematic presence of Gargantua". Two are in-jokes. One asks for a steeplechase in which the runners are language
models placed month by month from the Artificial Analysis leaderboard, another for an Artificial Analysis office on the
Moon. And yes, the pelican is there.
A brief has a fixed shape: what the scene must show, how people should interact with it, and how it starts. The one
AA shows in full is the T. rex: an "anatomically accurate, fully articulated T. rex skeleton walking through a
three-dimensional scene", with foot contact, weight transfer and a counterbalancing tail coordinated "so the skeleton
reads as a moving animal rather than independently oscillating bones", plus a pause, free inspection, a scrubbable
walk cycle and an attack on another animal. It's a hard brief, and one a panelist can only judge by
eye.
Only the one-line summaries of the other 54 are public, along with the scenes themselves. The shared "delivery
contract" every task gets, the system prompt and those automated-check fields are not. So you can look at every
result but you cannot rerun the benchmark, which matters for what "independent" can mean here. I come back to it at the
end.
## The sandbox, and the harness under it
Each model runs as an agent in [Stirrup](https://github.com/ArtificialAnalysis/Stirrup), AA's open-source agent
harness, one sandbox per task with no network. The methodology page lists the box: 4 CPU cores and 8 GB of memory,
no GPU, WebGL rendered in software, three.js 0.186.0, Rapier 0.20.0 and Blender 4.5 (both optional), Python 3, Node.js
22 and a headless Chromium for self-testing. There is an optional CC0 library of lighting, materials, models,
characters and animation clips from Poly Haven, ambientCG, Kenney, Quaternius and KayKit. Output is one HTML page with
its assets, up to 200 MiB. One attempt per task, up to 250 turns, three hours.
I cloned Stirrup (commit `247f24d`, MIT) to see what a "turn" is, since turn counts show up all over AA's analysis.
A turn is one assistant message, which can carry several tool calls; `_get_turn_count` in `src/stirrup/core/agent.py`
(lines 198 to 201) literally counts `AssistantMessage` objects. Three defaults in `src/stirrup/constants.py` shape
a long run like this one, if AA kept them. The context is summarised once it passes 70% of the window (`CONTEXT_SUMMARIZATION_CUTOFF =
0.7`, line 22). From 20 turns before the limit, the agent gets a "You have N turns remaining" message every turn
(line 23). And by default any image the agent looks at is shrunk to one megapixel before the model sees it (`RESOLUTION_1MP`,
line 26, applied in `core/models.py` line 162). That last one means the model is judging its own scene from a
downscaled screenshot rendered by a software rasteriser, while the panel sees it in a real browser on a real GPU.
The repository is the general harness. Nothing 3D-specific is in it: no brief, no delivery contract, no browser
check. So I can say how the loop works and not what the agent was told.
AA's replays add the texture that a leaderboard hides. On one task GPT-6 Astra has a complete page after 6 minutes
and finishes at 68; Claude Sonnet 5.5 shows nothing until minute 42 and works until 2 hours 58 minutes. Sonnet writes
6.2 times the output tokens of Astra on that run and costs 1.3 times as much, \$14.93 against \$11.62, because its
list prices are a fifth of Astra's. Across the benchmark, browser tests take 82% of tool time: at 33 to 38 seconds
each they fill about 25 of the 30.7 minutes of tool time per scene, while writing code is 6%. An agent that builds
a three.js page spends most of its wall clock looking at it.
## Who gets a vote: the eligibility gate
Before a scene can be voted on, a headless browser with software rendering gives it up to 60 seconds to load and
draw, and it must keep rendering, show a non-blank image and run offline without errors. Then a person reviews it.
Fail either step and the scene is out.
The methodology does not spell out what happens next. A scene that is out simply has no votes. Bradley-Terry only
sees the comparisons that happened, so a model that fails a brief is rated on the briefs it passed. I counted the
scenes in the launch article's explorer: 1,030 of the 1,155 possible (21 models times 55 briefs). GPT-6 Sol and Muse
Spark 1.3 have all 55. Claude Sonnet 5.5, the leader, has 47; the eight missing are the Sydney airport, the tube
radio, the watch complications, the mountain-bike trail, the pole vault, the glider, the James Webb telescope deployment and the
pelican. Claude Opus 5.5 has 50, Nemotron 3 Ultra 36.
I can't tell from public data why those eight are missing. Some may have failed the check, some may have run out of
time, and the explorer may not list every built scene (AA says GPT-6 Luna "builds all 55 scenes", and the explorer
shows 54). AA's own detail analysis mentions 1,008 scenes plus 23 that break on the voting page, which is 1,031, so
the counts are close but not identical. Either way the leaderboard answers "when it produces a working scene, how good
is it?" and not "how often does it produce one?". Those are different questions for anyone who will use the model
unattended, and a completion rate next to each Elo would answer the second.
## The vote
A panelist sees two scenes built from the same brief by two different models, interacts with both, and picks one.
The vote unlocks only after each has been open for at least 30 seconds. Names are never shown, even after the vote.
About one comparison in four has "a strong consensus answer" and works as a check; a session that fails those checks
is thrown out with all its votes, and the check votes themselves are not counted. Public votes on the
[try-it-yourself page](https://artificialanalysis.ai/3d/vote) do not count either. The result is "over 10,000 counted
votes from over 1,000 panelists", covering all 210 pairs of the 21 models.
This is a careful protocol. The 30-second floor and the hidden checks are exactly what an open arena lacks. But
it is worth being clear about what a vote is: one person's preference after half a minute or so with each of two
scenes. Thirty seconds is enough to see the lighting, the detail and how many controls there are. It is not enough to
play a pinball game properly, check whether a weightlifter's organs are where they should be, or notice that a
"Late Bronze Age" Troy has the wrong walls. I'll come back to what that does to the ranking.
## From votes to a rating
Bradley-Terry is a short idea. Give each model a strength $\theta_i$ and say the chance that $i$ beats $j$ depends only
on the difference:
$$
P(i \succ j) = \frac{1}{1 + e^{-(\theta_i - \theta_j)}}
$$
Then pick the strengths that make the observed votes most likely. If $w_{ij}$ is the number of times $i$ beat $j$, the
log-likelihood is $\sum_{i \ne j} w_{ij} \log P(i \succ j)$, and maximising it is a small convex problem: Newton's
method converges in a handful of steps. Only differences are identified, so one model is pinned. AA pins GPT-6 Luna at
1000, then rescales to Elo points, where 400 points is a factor of ten in the odds:
$$
R_i = 1000 + \frac{400}{\ln 10}\,\theta_i, \qquad P(i \succ j) = \frac{1}{1 + 10^{-(R_i - R_j)/400}}
$$
It is called Elo for familiarity, but nothing is updated vote by vote as in chess. The fit sees all the votes at
once, so their order does not matter. AA's methodology says every vote is weighted equally here, and that its Image,
Video and Music arenas use a different, power-law weighted fit.
The anchor matters more than it looks. Luna's own row on the leaderboard shows no interval, because it is fixed. Every
other "±" is therefore the uncertainty of that model's gap to Luna. Keep that in mind
for the next section.
I checked the scale against the data. AA's leaderboard embeds each model's win rate in each domain. If the ratings are
on the 400-point scale above and the panel paired models roughly evenly, a model's overall win rate should be its
average expected score against the other twenty. For Sonnet that is 76.7%, and its domain win rates, weighted by scene
count, average 76.5%. Inkling: 12.0% predicted, 12.0% observed. Across all 21 models the two never differ by more
than two points. AA's own chart makes the same point pair by pair: Sonnet and Luna are 194 points apart, the fit expects
75%, and the panel gave Sonnet 75%.
## What "statistically tied" should mean
AA charted the leaderboard with the claim drawn on.
The published intervals are ±32 to ±45. I wanted to know whether they are what 10,000 votes should give, so I simulated
it: take the published ratings as the truth, draw 10,000 votes between uniformly random pairs, fit Bradley-Terry
anchored at Luna, and read the intervals off the inverse Hessian. Over 20 draws a model's interval comes out at about
±36, and the bottom two widen to the mid-40s because their votes are lopsided and carry less information. AA publishes
±34 at the top and ±41 and ±45 for Inkling and Nemotron. The intervals are honest.
The test is the problem. Two 95% intervals that overlap do not mean two models are indistinguishable. What decides it
is the interval of the difference, and when both ratings are measured against the same anchor, they share Luna's
uncertainty. The covariance subtracts out, and the interval on a gap ends up about the size of a single model's
interval rather than 1.4 times it. In my simulations it is about ±37 for every top pair. Against that:
- Sonnet minus Opus is 4 points. Tied, and nowhere near separable.
- Sonnet minus Astra is 29. Tied.
- Astra minus Fable is 34. Tied, just.
- Sonnet minus Fable is 63, and Opus minus Fable 59. Separated.
So at this vote count the top is three models, with Fable fourth and level with Astra, GPT-6 Sol and GPT-6.1 Sol. AA's
own leaderboard says the same thing in a column the launch post does not use. Its "Range", described as "the range of
plausible rankings based on this model's Elo confidence interval", gives Sonnet and Opus 1 to 3, Astra 1 to 4, and
Fable 4 to 6. A model whose plausible ranks start at 4 is not tied for first.
None of this changes who is good. It is a narrower point about the sentence everyone will quote: "four statistically
tied" is true only under a test that is too lenient, and the table one click away shows it.
The widget below runs the same fit in your browser. It starts with 10,000 simulated votes drawn from the published
ratings. Add votes and watch the intervals shrink by the square root; start from zero and watch them open up; cast a
run of upsets by hand and see how little one model's rating moves when everything else is pinned by thousands of
other votes. The bottom panel tests the pairs that matter with the difference interval.
Two things to try. First, switch pairing to "near neighbours" and add 10,000 votes. Pairing each model with models
rated close to it tightens the intervals on the gaps at the top, where the question is, while the intervals against
the distant anchor get wider. AA does not say how it pairs; its matrix covers all 210 pairs, which suggests something
close to uniform. Second, try separating Sonnet from Opus. A 4-point gap needs an interval of ±4, and with only
direct comparisons near even odds that takes about $\left(1.96 \times 2 \times 173.7 / 4.1\right)^2 \approx 27{,}600$
votes between those two models alone. With 10,000 votes spread over 210 pairs, each pair gets about 48. The order of
the top two is not something this panel can settle, and AA does not claim it can.
## Is it just intelligence again?
The second finding is that AA-3D tracks AA's general Intelligence Index closely: "each point on the Artificial
Analysis Intelligence Index is worth about 18 Elo (Pearson 0.92)", a tighter fit than Terminal-Bench 4.0 (0.81) or
SciCode (0.76). If that were the whole story, a 3D benchmark would be an expensive way to re-measure intelligence.
The leaderboard page carries each model's Index, Terminal-Bench 4.0 and SciCode scores, so I refitted it. With all 21
models the slope is 17.9 Elo per Index point and Pearson's r is 0.923. Terminal-Bench gives 0.81 and SciCode 0.76,
exactly as stated. The residuals match too: GPT-6 Luna sits 141 above the line, GPT-6 Sol 93, GPT-6 Astra 46, and
GLM-5.3 197 below.
Then I dropped the three models with an Index under 30: MiniMax-M3, Inkling and Nemotron 3 Ultra, the three weakest
on both axes. The Index correlation falls to 0.79. Terminal-Bench falls to 0.63 and SciCode to 0.41. The ordering
survives, so the Index still tracks AA-3D better than either single benchmark, but a good share of the "0.92" comes
from three points at the bottom-left corner, which any two capability measures would agree on. Among the 18 models a
person would actually choose between, the Index explains about two-thirds of the variance in AA-3D Elo, against
the 85% that a 0.92 implies.
It is also less surprising than it sounds that an index beats any single benchmark. The Intelligence Index averages
ten evaluations, Terminal-Bench and SciCode among them, so its noise is lower by construction. The interesting part
is the residuals. The GPT-6 family builds better scenes than its Index predicts, by 46 to 141 points. GLM-5.3, which
cannot view images and so never sees its own scene, falls 197 below; AA notes that its sibling GLM-5.3-Flash, which
can, scores 149 higher with a lower Index. That residual looks like a fact about 3D, and about whether the agent can
look at its work.
## What thirty seconds of looking rewards
The best section of AA's launch article is the one where they take their own leaderboard apart, and it tells you what
a preference panel is measuring.
Detail wins. The most detailed fifth of scenes in each task, by four screenshot measures, has screenshots twice as
large (168 KB against 84 KB) and is a third as empty, and wins 63% of its votes against 31% for the least detailed
fifth. Control count tracks Elo with a Pearson of 0.90: the top five show a median of 15 to 20 buttons and inputs,
the bottom three 6 to 8. Models that look at their own render early rank higher (Spearman 0.82 across the 19 that
can view images): the top five first look after a median of 10 to 13 turns. Reading files and docs goes the other way,
with a Pearson of −0.74.
AA is careful about cause. Elo alone predicts 61% and 33% for those detail fifths, so most of the detail effect is
"strong models build detailed scenes". Within one model, a scene one standard deviation more detailed wins only about
1.5 points more of its votes. Of 86 scene measures, 11 survive multiple-comparison correction, and none moves votes by
more than 4 points.
I read that as good news and a warning. Good news, because the panel is not trivially fooled: a weak model cannot buy
votes by cranking up the polygon count. The warning is what the measured things have in common. Detail, controls,
visual polish and how soon the agent checked its own screenshot are what a person notices in half a minute. Nothing in
the protocol checks whether the brief was met. The T. rex brief asks for anatomical accuracy and an attack sequence
that "should involve the body and balance"; a panelist can reward the skeleton that looks most impressive without
knowing whether its pelvis is right, or ever finding the attack button. Frame time does not separate anyone either:
every model's median frame takes under 1.3 ms of GPU time on AA's test machine, far below the 16.7 ms a 60 fps frame
allows, so nobody is losing votes to jank on that machine. Whether that holds on a phone, nobody measured.
We have seen this pattern on the site before. On Artificial Analysis's image arena, the
[Ming-Image-0.1-Design](/articles/ming-image-0-1-design) "#1 among open-weight models" claim held in 3 of the arena's
20 views, and in the category it was quoted for the same model was 16th of 129 once closed models went back in. [Kimi K3](/articles/kimi-k3) held first on
WebDev Arena while sitting eighth on Text Arena, which says more about what each arena's voters reward than about the
model. Preference Elo is real information about preference. It is a different quantity from "meets the spec", which
is what rubric-graded evaluations such as AA's Harvey LAB-AA try to measure. Even AA's GDPval-AA, which grades office
work, ranks by blind pairwise Elo; the [Claude Haiku 5.5](/articles/claude-haiku-5-5) piece plots it against cost.
The domain table is where to be most careful.
AA says a top-five model has the highest win rate in six of seven domains, and that in living bodies GPT-6.1 Sol
(78.6%) ties Sonnet (78.5%). With seven or so scenes per cell and a 12-point noise floor, the honest reading is
narrower: the top five are good everywhere, which AA also says, and no domain result says a model is a specialist.
Sonnet's empty stories-and-fantasy cell is the eight-missing-scenes problem again: it has four scenes there, under
the five-scene floor, because the pelican is one of the missing.
## What a scene costs
AA prices every scene from token counts, split into fresh input, cache reads, cache writes, reasoning and answer.
A scene costs from \$0.20 (GPT-6 Luna) to \$32.08 (Claude Fable 5.1). Input is 69% of the spend, because an agent
resends its context every turn; input is 97% to 99% of all tokens, mostly read from cache. I recomputed the cost
frontier from the embedded per-scene costs and it matches: GPT-6 Luna at \$0.20 and 1000, Muse Spark 1.3 at \$3.21
and 1035, GPT-6 Sol at \$3.46 and 1123, Claude Sonnet 5.5 at \$15.18 and 1194. Kimi K3 spends 68 times what Luna does
per scene for a slightly lower rating.
The frontier ignores intervals, which matters at its cheap end. Because Luna is the anchor, Muse Spark's published
interval, 1003 to 1067, is literally the interval on its gap to Luna, and it clears 1000 by three points. The
frontier's second step exists, just. GPT-6 Sol over Muse Spark (87 points) and Sonnet over Sol (71) are both clear.
If I were picking a model to build these scenes, the frontier reads simply. GPT-6 Sol at \$3.46 is fifth overall
and 4.4 times cheaper than Sonnet; Sonnet buys roughly 70 Elo, which is a 60% head-to-head preference, for the
extra \$11.72; and nothing past Sonnet buys anything the panel can detect. Luna at 20 cents is the surprise: eighth
overall, with 54 of its 55 scenes in the explorer. One caveat on the axis: it is "mean cost
per scene built". AA does not say whether runs that produced no eligible scene count in the denominator, and for a
model that misses eight of 55 the difference could be real.
## What I'd take from it
AA-3D is the first 3D-generation benchmark I would cite. The protocol is sound where open arenas are weak: a fixed
brief, a fixed sandbox with a clock, blind pairs from the same brief, a viewing floor, hidden check questions,
recruited panelists, and an estimator and interval that are named and, as far as I can reproduce them, correctly
computed. The launch analysis is unusually self-critical, and the scenes are all public, so you can look at what a
1194 means.
How I would read it:
- Treat the top three as one tier and everything from Fable to GPT-6.1 Sol as the next. For two models, test the
gap against the interval of the gap, which is about ±37 at this vote count, not whether their bars overlap. The
leaderboard's Range column agrees with that test at the top, so read it before the bars.
- Read a rating as quality given a working scene. If you will run an agent unattended, ask AA for completion rates,
or count scenes in the explorer as I did.
- Read Elo as what a person prefers after half a minute. Visible detail, more controls and an agent that checks its
own screenshots early win those votes. Whether the brief was met is not scored.
- Don't over-read the Intelligence Index fit. It is real, and three weak models make it look tighter than it is
among the models you would actually choose between.
The thing it cannot be yet is reproducible. Stirrup is open, the scenes are open, and the ratings can be rebuilt from
the page, as this article does. The briefs in full, the delivery contract, the automated-check fields and the votes
themselves are not, so nobody outside AA can rerun a model or refit with a different pairing. For a benchmark whose
whole value is that a third party ran it, publishing the vote table would cost nothing and settle most of the
questions above.
If you build three.js scenes with agents, [PixiJS 3D and benchy](/articles/pixijs-3d-benchy) and
[Voxel Musou](/articles/voxel-musou) are two things built that way, read file by file, and
[One-shot launch videos](/articles/one-shot-launch-videos) looks at the same idea for video.
## How I checked
- Read the X post and its thread through the fxtwitter mirror, and the launch article, methodology page and
leaderboard at artificialanalysis.ai on 13 October 2026. Every rating, interval, rank range, Intelligence Index,
Terminal-Bench 4.0 and SciCode score, domain win rate with scene count, per-scene cost and tool-use breakdown here
is parsed from the data those pages embed.
- Recomputed the Elo-vs-Index fit (ordinary least squares and Pearson's r) for all 21 models and for the 18 with an
Index of 30 or more, and the same for Terminal-Bench 4.0 and SciCode. Recomputed the cost frontier from per-scene
costs.
- Checked the Elo scale by comparing each model's expected score against the other 20 under the 400-point logistic
with its scene-weighted domain win rate. This assumes roughly even pairing and per-scene weights stand in for
per-vote weights, so it is a consistency check, not a refit.
- Simulated the panel: 10,000 votes between uniformly random pairs drawn from the published ratings, a Bradley-Terry
fit by Newton's method with GPT-6 Luna anchored at 1000, intervals from the inverse Hessian, 20 seeds. The widget
runs the same fit in the browser (`components/articles/aa-3d/bt.ts`). AA uses a sandwich estimator, which agrees
with the Hessian one when the model is right; its vote table is not public, so I could not refit the real votes.
- Counted scenes per model and the missing briefs from the launch article's scene explorer.
- Cloned Stirrup at `247f24d` (MIT) and read the agent loop, constants, image handling and summariser prompt. The
AA-3D task configuration is not in it.
- Not verified: why any particular scene is missing, how AA pairs models for voting, and whether failed runs count in
cost per scene.
---
# Astronex-World 1.0: a world model trained on two L20s, and what its cache really holds
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/astronex-world
> date: 2026-10-13
> tags: world-models, video-generation, diffusion, distillation, kv-cache
Most open world models on WBench come from labs with a rack of H100s behind them. Astronex Robotics says it did the whole post-training run for Astronex-World 1.0 on two NVIDIA L20 48 GB cards: a 5B model, Apache-2.0, that scores 70.0 on WBench Full and, they say, streams 832x480 at 24 fps on one of those cards. I first saw the name a few rows down the WBench leaderboard while checking [XGEN-Jing's #1 claim](/articles/xgen-jing-dao). The ModelScope announcement then landed with weights, code, a 25-page technical report, sampling presets and post-training recipes. That is enough to read the thing end to end instead of taking the bullet points on trust, so I did.
The short version: the engineering inside the causal sampler is careful, unusually well commented, and more interesting than the headline. Some of the headline is softer than it reads. The 70.0 is real, but it puts the model 11th of 16 on the live Full table. Two of the four control channels are mostly scaffolding in the released weights. And the "real time on one L20" sentence is the one claim I tried hardest to reproduce on paper and could not.
## One trunk, two attention masks
Astronex-World is a post-trained [Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B). It keeps Wan's video VAE (48 latent channels, 4x temporal and 16x16 spatial compression), its UMT5-xxl text encoder and its 30-layer [diffusion transformer](/architectures/diffusion-transformer): width 3072, 24 heads, FFN 14,336. Wan2.2's 1x2x2 patch embedding means a single 832x480 latent frame becomes a 15 by 26 grid, 390 tokens. Keep that number in mind; most of the arithmetic later comes from it.
The idea the report builds everything on is that one set of weights can be sampled two ways. In the bidirectional form, every latent frame attends to every other one across a 20-latent-frame clip, sampled with 50 steps and CFG 8.0 (`inference/bidirectional.yaml`). That is a normal video diffusion model: good motion, fixed length, nothing you can steer mid-clip. In the causal form the temporal mask becomes block lower-triangular. Frames inside the current block of eight still see each other, but they only see the past, through a KV cache, and each block is denoised with eight UniPC steps at CFG 3.0. Once a block is done its keys and values go into the cache and the next block starts from fresh noise. That is what makes the video extendable and steerable while it runs.
The bidirectional checkpoint is not a throwaway teacher here. It is a released form, it is the real-score model in the last training stage, and the repo's own front end says when to prefer it: "Pick causal for interaction and length, bidirectional for a fixed-length clip where motion has to be right" (`inference/generate.py`). Both forms load the same 898 tensors. I counted them in the three safetensors headers, read by HTTP range request without downloading the 10.7 GB of weights, and the README's breakdown holds exactly: 825 tensors of Wan backbone, 60 for the PRoPE output projections, 7 for the action embedder, 6 for the action-output head. All 898 are bf16.
## What the cache actually holds
Figure 2 draws a 20-frame window with four sink frames sitting outside it, which would make 24 frames of attention context. The code says otherwise. The cache is allocated as `kv_cache_size = self.local_attn_size * self.frame_seq_length` (`inference/pipelines/causal_diffusion_inference.py:1300`), `local_attn_size` is 20 in `causal.yaml`, and the eviction rule keeps the first four frames at the front of that same buffer. So at steady state a block sees 20 latent frames in total: four sink frames, eight frames of recent history, and its own eight. The README's version of the sentence, "sink + recent history + current block", is the accurate one.
That is a short memory. Eight frames of history is 32 output frames, 1.33 seconds of video at 24 fps. Everything older than that, except the opening four latent frames, is gone. Three pieces of engineering try to make that short memory behave.
The sink stays in view forever, and not only because it is never evicted. The cache stores raw keys, and every step the code re-applies [RoPE](/architectures/rope) to the resident window with the window's start as frame 0 (`models/causal_model.py:885`). The comment there says this fixes the rotation distance between sink and query at the window size for good, "so the reference frame never decays out of attention." In image-to-video the reference frame is frame 0, so it conditions every block until the end. A comment in `inference/pipelines/causal_inference.py:218` measures the cost of that: with the reference pinned, the content distance from the first frame shrinks over a 15-second rollout (0.052 to 0.044), while the same model with no reference drifts away from where it started (0.036 to 0.131). "The scene is held, not explored." There is an opt-in `cond_release_frame` to unpin it. I like that this is written down rather than discovered by a user.
The second piece is the one the report calls retrieval, and it turned out to be different from what I expected. I expected a memory bank: old frames stored somewhere, pulled back when the camera returns. What the released sampler does is choose what to evict. When a new block arrives and the cache is full, the newest four history frames are pinned, and the other four history slots go to whichever resident frames best overlap the view the new block is about to draw. Overlap is measured geometrically in `_frustum_overlap` (`post_train/objectives/camera_diffusion.py:15`): a 5x5 grid of rays is pushed through the query camera's frustum to depth 3, and each candidate frame scores the fraction of those points that land inside its image and in front of it. A frame needs a score of at least 0.3 to be kept on merit, at most two kept frames can change per block, and anything left over is filled by recency (`utils/cache_selection.py`). Nothing outside the cache is ever consulted. Once a frame has been evicted, no score brings it back.
The widget below is a top-down port of that bookkeeping, with the window sizes and thresholds from `causal.yaml` and the step sizes Control2V's poses quantise to (0.08 units and 3 degrees per latent frame). Try the strafe path with retrieval on: frames 10 and 11, from early in the outbound leg, survive in the cache through the whole return, while recency alone would have dropped them at block 3. Then pan right and back, and watch the kept frames fall under 0.3 and stay anyway, because the swap limit holds the previous picks.
This matters for how to read the long-horizon claims. Overlap-aware eviction helps a zigzag or a short out-and-back, where the frames you need are still resident when you turn round. It cannot help a return after more than a few blocks, because by then the frames that saw the place are gone. The report is candid about where this leaves the model: its failure figure shows an abstract tunnel losing its geometry under `W-right-left`, and a warehouse fading to near-black over 15.8 seconds. The README says retrieval is used for "revisit trajectories"; the report says the final WBench runs enabled it only for round trips, loops, repeats, L-shapes and zigzags. The released `causal.yaml` turns it on for everything.
A third piece exists in the code but is off by default: `sink_refresh`, which swaps the sink's contents for still-visible frames once the original sink stops overlapping the view. Its comment records why: on one WBench case "the sink's content correlation falls under 0.3 by frame 48, and the visible damage follows about a hundred frames later". Dropping the sink entirely "hallucinates gold objects and colour noise". These are the comments of someone who ran the experiments. I would rather read this than another diagram.
## Where the four controls enter
The announcement lists camera trajectories, 64-D continuous actions, embodiment IDs and timed text events. They enter the network in three different places, and the released weights train them to very different degrees.
Cameras go through PRoPE, which turns each frame's intrinsics and world-to-camera transform into a projective positional encoding, so attention works in relative camera geometry. The report says PRoPE is applied in all 30 layers. In the code it is a second self-attention running beside the ordinary RoPE one, on its own KV cache, whose output passes through a zero-initialised projection, `prope_o`, and is added to the block's output (`models/causal_model.py:1218`). That is the ControlNet trick applied to attention: at initialisation the model is exactly Wan, and the camera branch earns its influence. It is not free. The 30 `prope_o` projections are 283 million parameters, about 5% of the model, and every causal step runs two self-attentions per layer instead of one.
Actions go through modulation. A 64-D vector per latent frame passes through a two-layer MLP, the embodiment embedding is added, and a zero-initialised linear layer turns the result into the same shift, scale and gate values Wan's timestep embedding already produces (`models/action.py`). The docstring explains the choice in one line I'll borrow: camera goes through PRoPE, which edits attention geometry; actions go through modulation, which edits feature statistics. It also records that the 64 and the 32 were chosen to match NVIDIA's [Cosmos 3](/articles/cosmos-world-model), whose action channel has the same shape.
What fills those 64 numbers is narrower than "64-D continuous actions" suggests. `utils/action_encoding.py` lays out the lanes the trained embedder reads: two for the left stick, two for the right stick, 17 binary buttons, and 43 lanes of zeros marked reserved. WBench's keyboard commands become stick values: W is forward on one lane, the arrow keys turn on another. So in the release the action stream is a gamepad, which matches its training data: DROID robot actions in stage I and CrossFPS game footage in the later stages. Driving and manipulation are described as post-training targets, not current behaviour.
The embodiment table tells the same story more starkly. It has 32 rows of 3072, and I read all of them by range request. One row is non-zero. The other 31 are still exactly zero, as `ActionEmbedder` initialises them. "32 embodiment IDs" is a capacity, and one of them has been trained.
The action-output head, which the README presents as the route to inverse dynamics, ships untrained, as the config says plainly: "included but untrained". I checked how untrained. The report and README describe the output layer as zero-initialised. In the shipped tensors the hidden layer is zero too: every value I sampled from the 3072x3072 matrix, and both biases, are 0, with only the LayerNorm's weight at its default of 1.0. What ships is the scaffold for a head, with nothing learned in it.
Events are the most interesting control, and the cleanest. An event is a second caption appended to the prompt from a chosen latent frame onwards. Frames before the switch are already in the cache and never recomputed; the cross-attention keys and values are rebuilt from the new text, so every later block reads prompt plus event while still attending to the cached scene. That is why an edit like "the building collapses" lands inside the existing scene instead of replacing it, and why only the causal form can do it: the bidirectional form has no finished past to protect. The README notes the release has no dedicated event branch and routes events through the caption (`event_in_caption: true`), and that the event text has to stay in plain language, because rewriting it in abstract technical vocabulary stops it triggering.
One number in that section puzzled me. Against an otherwise identical run with no event, the report measures frames before the switch differing by a mean of 1.8/255 and frames after by 9.5/255. If the prefix is computed causally from identical inputs it should be identical, and the code's own comment says earlier chunks are "bit-identical to a run with no event at all". Something after sampling touches the whole clip. The colour stabiliser in `causal.yaml` (`rgb_chroma_stabilize_strength: 1.0`) is my guess, but I did not confirm it. Either way, 1.8 against 9.5 shows the event staying where it was put.
## Five stages, two cards, and what ships
The recipe is the now-familiar path from a bidirectional video model to a fast causal one, done carefully and on a small budget. Stage I adds the controls to Wan with full attention: 2,000 camera updates plus 400 for direction, then 2,000 action updates on DROID with PRoPE frozen (DROID's cameras barely move, so training PRoPE there would teach it that cameras are static), then 200 updates to restore the camera response the action branch had weakened. The result, `restore-2200`, is the released bidirectional model. Stage II switches to the block-causal mask and trains with teacher forcing on clean history for 500 updates. Stage III is online trajectory distillation in the style of Causal Forcing++, a descendant of [CausVid](https://arxiv.org/abs/2412.07772): a frozen copy of the stage II model runs a 25-step UniPC trajectory, the student follows a 12-step one from the same noise, and one student transition per sample keeps its gradient. It runs for 20 updates. Stage IV is a 540-update SFT on a mixture of Control2V, CrossFPS, DrivingDojo and NVIDIA PhysicalAI with rolling forcing, keeping the first frame clean 70% of the time so the same weights do text-to-video and image-to-video. Stage V is asymmetric DMD: the causal student is matched against the frozen bidirectional model as the real score, with a fake-score network trained online, for 45 updates.
Two things in that list are worth stopping on. All of it uses LoRA at rank 64 on a frozen backbone, which is why two cards suffice: the config records the final export as `W = base.weight + (lora_b @ lora_a) * 128/64`, folded in fp32. And the update counts are tiny. Stage III is 20 updates at effective batch 2; stage V is 45. This is not a model that learned a world from scratch. It is Wan2.2 with a new attention mask and a set of controls grafted on, nudged until it streams. The efficiency claim is honest in that sense, and it is also the main reason the comparison with 14B models trained on H100 clusters is less surprising than it sounds.
The price of stage V is in the report's own internal diagnostic: a ball's bounce amplitude on one test sequence.
| Model state | Bounce amplitude | vs bidirectional |
| --- | --- | --- |
| Bidirectional teacher | 88.8 px | 100% |
| SFT1500 (after stage IV) | 36.1 px | 41% |
| DMD1515 | 21.2 px | 24% |
| DMD1545 (the release) | 16.6 px | 19% |
The released causal model keeps 19% of the teacher's motion on this test. The report says so plainly and calls it DMD over-staticization; VBench's Dynamic Degree of 0.254 for text-to-video agrees. It is also why the README says to use the bidirectional form "when the motion has to be exact". If you need things to move, budget for that.
Then there is what "the complete five-stage training pipeline ... runs on two 48GB GPUs; weights, code, sampling presets, and post-training recipes are included" means in practice. The code release has three post-training recipes, `camera`, `action` and `sft`, all running the same teacher-forced flow-matching trainer with LoRA on the released weights (`post_train/train.py`). There is no stage III distillation trainer and no DMD trainer in the repo; a docstring in `camera_diffusion.py` notes that this trainer "is not their DMD, which needs a teacher score model and a critic this trainer does not build". The datasets are not included either. The report credits minWM as the framework the stages were run in. So the five stages are documented, and you can continue training the release, but you cannot rerun the recipe that produced it from this repo. The tweet's wording leaves room for both readings.
The post-training README is worth reading anyway, for two bugs it describes that would bite anyone. Non-reentrant gradient checkpointing kept the KV cache alive through `saved_tensors_hooks` and ran out of memory on the third step. Reentrant checkpointing silently built no graph at all, because block 0's input comes off a frozen patch embedding: 600 trainable tensors, all with `grad is None`, and 26 checkpoints byte-identical to their start. Its advice is to compare start and end checkpoints tensor by tensor before believing any measurement from a run. Good advice for anyone doing LoRA on a frozen DiT.
## The real-time claim, counted
The report, the README and the announcement all say the causal form streams 832x480 at 24 fps in real time on one L20. None of them gives a measured throughput. The code has a timer, but it records the latency of the first block, excluding the text encoder and the VAE decode, and no number from it is published. So I counted the work instead.
The released sampler (`CausalDiffusionInferencePipeline`) does this per block: for each of eight UniPC steps, it runs the DiT once with the prompt and once with the negative prompt, on separate KV caches, and combines them with CFG 3.0. After the last step it runs one more clean pass per branch to write the block's keys and values into the cache. That is 18 passes of the 5B DiT for one block. A block is eight latent frames, which the VAE turns into 32 output frames, so all 18 passes have to fit into 1.33 seconds.
Each pass processes 3,120 query tokens (8 frames of 390). The linear layers cost about 9.25 GFLOP per token, which is 28.9 TFLOP for one pass. The two self-attentions over the 7,800-token window add about 17.9 TFLOP, cross-attention to the text another 0.6. Call it 47.4 TFLOP a pass and 853 TFLOP a block. To finish that in 1.33 seconds you need to sustain about 640 TFLOPS of bf16 matrix multiplies. The L20 is an Ada-generation card; the dense bf16 figure I remember from NVIDIA's datasheet is about 120 TFLOPS. I could not re-fetch the sheet from this machine, so the widget lets you set it.
Even with every corner cut, the arithmetic does not close on that card. Four steps without CFG is five passes, about 237 TFLOP a block and roughly 178 TFLOPS sustained, still above the peak I have for an L20. At the released settings and a generous 50% utilisation, the count says about 14 seconds of compute per 1.33 seconds of video. I would be glad to be wrong, and there are ways I could be: the claim may have been measured on a different configuration or at lower resolution, or "real time" may mean the first-block latency the code times. But nothing in the release shows 24 fps being produced, and the count says it would need several times the card's peak.
The memory side checks out, which makes the speed claim stand out more. The README measures a 37.9 GB peak for `causal.yaml` and 23.2 GB for `causal_consumer.yaml`, which drops the caches before the VAE decode. From the shapes: there are four caches (prompt and negative prompt, each with a RoPE and a PRoPE copy), each 20 frames x 390 tokens x 3072 x keys and values x 2 bytes x 30 layers, about 2.9 GB, so 11.5 GB of cache next to 10.7 GB of weights. That lands just under the 23.2 GB they report, and explains their remark that below about 24 GB the caches themselves stop fitting. The consumer config is a real contribution: same output, "bit-identical on both a 23-frame and a 47-frame rollout", on a 32 GB RTX 5090.
One smaller inconsistency: the README says the sampler "also accepts 4 steps for latency experiments", the report says four-step sampling "is also supported", and the shipped `config.json` says "4 steps degrades visibly -- cabinets and signage smear -- so it is not a supported mode". Believe the config.
## 70.0 against which field
The WBench numbers in the report match the live leaderboard to the decimal: 70.0 on Full 289 and 73.5 on Navi 158, with Quality 78.3, Setting 73.8, Interaction 47.6, Consistency 82.4 and Physical 68.1 on Full. They are not flagged as self-evaluations in the leaderboard's markup, unlike the rows at the top of the board that the [XGEN-Jing article](/articles/xgen-jing-dao) looked at.
Context is where the announcement gets selective. The report's comparison table lists seven open models on Full. The live Full table today has 16 rows, and Astronex-World is 11th. Above it, besides the API models (Kling 3.0, Wan 2.7, Seedance 1.5) and the self-evaluated leaders, are HY-Video 1.5 at 74.3, NVIDIA's Cosmos3-Super at 72.5, LTX 2.3 at 70.9 and Cosmos 2.5 at 70.4. The report's table includes HY-Video 1.5, an 8.3B model 4.3 points ahead; the tweet's "outperforming evaluated models with 13.6B and 14B parameters" quietly skips it. The three Cosmos rows are not in the report's table at all, though two of them score higher. The report says its table is a snapshot of 13 September 2026, so some rows may have arrived later, and I can't tell from the page when each was added. Below it are LongCat-Video (69.9), Helios (69.7), Cosmos3-Nano (69.4), YUME 1.5 (68.9) and Kairos 3.0 (65.7).
The Full split is also a thin field. Most dedicated interactive world models only report Navi. On Navi, where 44 models are listed, Astronex-World's 73.5 is 36th; three other 5B open models are well ahead of it there, including Zing-0.5 at 81.0, DreamX-World at 75.0 and [ABot-World](/articles/abot-world) at 74.7. The "5B beats 14B" framing works on Full because the interactive specialists mostly are not on that table.
The weakness is in the report too, stated plainly: Interaction is 63.4 on Navi but 47.6 on Full, because event editing, subject actions and perspective switches go through the one-caption-switch interface, and the WBench adapter merges multiple timed events in one case into a single switch. Quality is the strength. Its 78.3 is the highest of the seven models in the report's table, and also the highest of any open-source row on the live Full table; only API models and self-evaluated entries score above it.
VBench is a partial run, and the report says so: 240 of 6,220 text-to-video prompts and 118 of 5,590 image-to-video prompts, in official order without selection. Smoothness (0.990), flicker (0.987) and background consistency (0.985) are high; dynamic degree (0.254) is low, which is the DMD motion loss again. I would not read more than that into 4% of a benchmark.
## What I'd use it for
The model card and the code are both linked here so you can check my reading against them.
If you want a small, permissively licensed world model to study or post-train, this is a good one to start from, mostly because of the code. The KV-cache bookkeeping, the overlap-aware eviction, the window-relative RoPE and the event mechanism are compact and documented with the experiments that motivated them. The `camera` and `action` recipes train one pathway while leaving the other bit-identical, which makes ablations answerable. The consumer config puts inference within reach of a 24-32 GB card.
I would not pick it for interactive streaming on an L20 until someone publishes a measured frame rate. I would not count on the action interface for robotics or driving yet either: what's trained is one embodiment and a gamepad's worth of lanes, and the output head is zeros. And if motion matters, I'd sample the bidirectional form, which keeps the teacher's 88.8 px bounce instead of the student's 16.6.
The broader result stands: a careful five-stage recipe on a frozen 5B prior and two 48 GB cards gets within a point of several much larger open models on WBench Full. That is worth knowing. It is a statement about the Wan2.2 prior and about distillation recipes having matured, more than about this model's world.
## How I checked
I read the ModelScope card, the technical report PDF and the code release, shallow-cloned at commit `27584bf` (2026-09-18), and the configs (`config.json`, `transformer/config.json`, `vae/config.json`, `inference/*.yaml`, `post_train/configs/*.yaml`). I read the three safetensors headers from the Hugging Face mirror (`Astronex-Lab/Astronex-World`) by HTTP range request, counted tensors and parameters, and range-read the small graft tensors (the embodiment table, the action head, slices of the modulation and PRoPE projections) to see which were still at their initial values. The header totals 5,359,018,240 parameters including the untrained head; the card's 5.35B is a rounded figure.
The cache sizes, the retrieval rule and the per-block pass count come from reading `models/causal_model.py`, `inference/pipelines/causal_diffusion_inference.py`, `utils/cache_selection.py` and `post_train/objectives/camera_diffusion.py`. The window widget is a port of that logic to a flat top-down world with five rays instead of 25. The FLOP count is my own arithmetic from the model shapes and counts matrix multiplies only. I did not run the model: I have no L20, and the house rule is not to execute third-party code. The L20 peak throughput is from memory, not a datasheet I could fetch. Leaderboard positions are from the live WBench page (meituan-longcat.github.io/WBench), read on 2026-10-13.
---
# BudgetPix: a pixel diffusion model that reads its own first draft to decide where the tokens go
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/budgetpix
> date: 2026-10-13
> tags: diffusion, diffusion-transformers, image-generation, flow-matching, tokenization, inference
AK posted the title and a Hugging Face link. A pixel diffusion transformer spends the same work on every 16×16 patch: empty sky costs what fur costs. BudgetPix asks whether one fine-tune can turn that cost into a dial you set at inference time.
The paper is **BudgetPix: Compute-Adaptive Tokenization for Pixel-Space Image Diffusion** ([arXiv 2610.12307](https://arxiv.org/abs/2610.12307)), by Ozgur Kara, Yujia Chen, Daniel Watson, David Forsyth, James Matthew Rehg, Wen-Sheng Chu and Du Tran, from UIUC and Google. It adapts released pixel-space models (JiT, MiniT2I, PixelDiT and the one-step pixel MeanFlow) and reports that one checkpoint degrades gracefully from the dense grid down to a tenth of it.
The quadtree is the easy part. The interesting part is a chicken-and-egg problem the paper states itself: to decide where an image is detailed you need the image, and a generator does not have one yet. How it gets around that decides what the method costs.
There is no code and no weights. The [project page](https://karaozgur.com/BudgetPix/) shows a "Code (soon)" button that is disabled, its tooltip reading "Code and checkpoints will be released". Hugging Face's paper page links a repository, `ozgurkara99/budgetpix`; I cloned it, and at commit `f48ce48` (12 October 2026) it holds a README and a `.gitignore`, with the line "Code will be released soon!". The Hugging Face API lists no models. Everything below is checked against the paper, its long appendix and the project page.
## Why uniform patches waste compute in pixel space
A pixel-space model skips the VAE and patchifies raw RGB, so the token count is set by one number, the patch size: a 512×512 image in 16-pixel patches is $(512/16)^2 = 1{,}024$ tokens, and in 32-pixel patches it is 256. [PixelUMM and PixelDense](/articles/pixelumm-pixeldense), from the same fortnight, explain why pixel models predict the clean patch rather than the noise; BudgetPix keeps that and changes only what a token covers.
The paper's motivating example is a flag against open sky: MiniT2I-L spends 1,024 tokens on it, and BudgetPix reconstructs the scene with 307 (reported, Figure 2). The same MiniT2I-L is the text-to-image base of [Representation Guidance](/articles/representation-guidance), from the same week, which leaves the token count alone and changes the training target instead.
How much compute does a token cut buy? Count one standard transformer block with width $D$ and $N$ tokens. The QKV and output projections cost $8ND^2$ FLOPs, an MLP of ratio 4 costs $16ND^2$, and attention scores plus mixing cost $4N^2D$. At a token fraction $f$:
$$
\frac{\text{FLOPs}(fN)}{\text{FLOPs}(N)} = \frac{6Df + Nf^2}{6D + N}.
$$
The attention term is quadratic, but big pixel patches keep $N$ small next to $6D$. With the widths in the paper's Table 16 and the token counts in Table 21, attention is about 4% of a block's FLOPs for JiT-L/32 at 512² ($N = 256$, $D = 1{,}024$), 12% for MiniT2I-L/16 at 512² ($N = 1{,}024$, $D = 1{,}248$) and 31% for PixelDiT at 1024² ($N = 4{,}096$, $D = 1{,}536$) (reasoned). So cutting tokens to 25% cuts a block's FLOPs to between 19.2% (PixelDiT) and 24.3% (JiT-L/32), not to 6.25% (reasoned). The saving is mostly linear, which matters once the paper's "compute budget" turns out to mean tokens.
## The entropy quadtree
BudgetPix borrows its layout rule from APT, an adaptive-patch vision transformer. Take token scales $p$, $2p$ and $4p$ (16, 32 and 64 pixels for a 16-pixel parent). Start from the coarsest cells. For each cell $R$, build a histogram of pixel luma and take its Shannon entropy:
$$
H(R) = -\sum_{k=1}^{K} q_k(R)\,\log_2 q_k(R),
$$
where $q_k(R)$ is the fraction of the cell's pixels in bin $k$. The appendix pins the details: $K = 512$ bins, so $H$ runs from 0 to 9 bits. A $4p$ cell stays one token if $H \le \tau_{4p} + \delta$, otherwise it splits into four $2p$ cells; each of those stays one token if $H \le \tau_{2p} + \delta$, otherwise it splits into four $p$ tokens. The thresholds are $\tau_{4p} = 4.0$ and $\tau_{2p} = 6.0$ bits, taken from APT "not tuned" (reported, App. D.7). The global offset $\delta$ is the dial.
There are two ways to set $\delta$. In threshold mode it is fixed, and each image takes whatever token count its content gives. In budget mode, which the paper uses for every reported number, $\delta$ is bisected per image so the layout has at most $B$ tokens. The two lie on nearly one quality curve when you compare them at the same mean token count (reported, Table 14). Threshold mode gives up predictability: at $\delta = +1$ on JiT-L/32 the mean is 56.0% of the grid with a standard deviation of 18.2 points across images (reported). A server with a latency target cannot take that spread.
Drag the budget down and watch which regions go first. The sky merges into 32-pixel tokens, then into 64-pixel ones; the tree crown holds out longest. At the default of 30% the scene lands on 76 tokens of 256 (measured, in the widget). In the entropy view, look at the kite. Its four panels are a handful of flat colours, so its histogram has only a few occupied bins and its cells score lower than the grass, even though the kite has the sharpest edges in the frame, and it merges before the grass does. A luma histogram counts distinct values, not their arrangement; a two-colour checkerboard scores 1 bit. The paper's conclusion lists "learning where to place tokens, not deriving it from entropy" as the next step.
Uniform patchify on this grid gives 256, 64 or 16 tokens and nothing in between. The quadtree gives any count from 16 to 256 and spends the cut where the picture has least to lose. [Level-of-Token Diffusion](/articles/lot-diffusion), on arXiv three days earlier, tiles a DiT's token grid the same way, but its layouts come from a planner or a reference image and its count is whatever the thresholds produce. BudgetPix guarantees the count and needs no outside layout; the price of that comes up below.
## Getting pixels into and out of mixed-size tokens
The central transformer is the released model, fine-tuned, with its own global position embedding evaluated at each token's centre. What is new sits on either side of it.
**The encoder** has two paths. A $p \times p$ patch goes through the parent's own patch embedder $E$, unchanged. A larger patch $P_i$ of side $np$ is bilinearly downsampled to $p \times p$ and embedded with the same $E$, which loses everything inside it. A second stream, the Patch Aggregator, puts it back: it splits $P_i$ into $n^2$ sub-patches, embeds each with $E$, adds a learned position within the patch, and mixes them with a small convolutional stack into one correction vector. Both paths add a scale embedding:
$$
\mathbf{x}_i = E\big(\mathrm{resize}_p(P_i)\big) + \mathbf{e}_{\mathrm{scale}}(s) + \mathrm{PA}_s\big(\mathrm{split}(P_i)\big), \qquad n > 1.
$$
**The decoder** mirrors it. The parent's head turns a token into a $p \times p$ patch; for a coarse token that patch is bilinearly upsampled to $np \times np$, right in layout but smooth inside. A Patch Refiner adds the missing texture as a residual: it cuts the predicted patch into 4×4-pixel pieces, runs two small self-attention blocks over them under adaptive layer norm conditioned on the token's final hidden state, and projects to an $np \times np$ residual.
The added modules are 2.39M parameters on JiT-B/16, 3.50M on MiniT2I-L's 911.8M and 4.37M on PixelDiT's 1302.5M (reported, Tables 16, 18 and 19). Their last layers are zero-initialised, so at step 0 the adapted model reproduces the released one, with identical dense-grid FID (reported, Table 17). PixelDiT already ends in a full-resolution pixel stage, so there BudgetPix corrects each cell's conditioning of that stage instead, and the stage decodes every pixel at every budget.
On JiT-B/32 at 512² and a 50% budget, FID-50k is 60.61 with neither (the released weights fed coarse tokens), 11.58 with the aggregator and 8.30 with the refiner added too (reported, Table 2). The aggregator does most of the work.
## Training one checkpoint for every budget
Every weight is fine-tuned with the parent's own $x$-prediction flow-matching loss, $\mathbb{E}\big[w(t)\lVert x - \hat{x} \rVert_2^2\big]$, with $w(t)$ the parent's weighting. The new part is which layout each training image gets:
- **dense grid**, with probability $a = 0.15$, to anchor the model;
- **random layout**, on $b = 0.10$ of the rest (8.5% of all images): the quadtree of a fake entropy map drawn uniformly from 0 to 9 bits, so coarse tokens also land on detail;
- **content layout** otherwise: the quadtree of the clean image's real entropy.
For the last two, $\delta$ is drawn from $\mathcal{U}(0, 2)$ per image and step, which the paper says covers "roughly 25% up to 100% of the original dense grid". Table 14's threshold sweep agrees: $\delta = +2$ gives a mean of 28.0% of the grid on JiT-L/32 (reported). So budgets below about a quarter ask for offsets the model never trained on (reasoned), and FID shows it: 11.54 at 25%, 18.12 at 20%, 44.62 at 10% (reported, Table 14). The introduction's "coherent, high-quality images using as little as 10%" rests on the sample grids; the number at 10% is 44.62 against 2.71 dense.
## Where the layout comes from when there is no image yet
In training, the layout is the clean image's quadtree. At sampling time there is only noise. BudgetPix's answer:
1. Run the first $n_w$ steps on the dense grid, so the model sets the composition.
2. Because the parents predict $x$ directly, every forward pass already outputs a clean-image estimate $\hat{x}$. Compute its quadtree and bisect $\delta$ to fit the budget $B$.
3. Keep that layout for $m = 5$ steps, then rebuild it from the newest $\hat{x}$.
The defaults are $n_w = 5$ for the 512² JiT parents, MiniT2I and PixelDiT, and $n_w = 10$ for JiT at 256² (reported, Tables 18, 19 and 21). The layout is not predicted from the prompt or fixed in advance. It is the model's own first draft, measured with the training rule.
The appendix tests how much each choice matters, on JiT-B/32 at 512² with FID-10k (reported, Table 27). Random placement with the same token count is far worse: 19.51 against 10.99 at 50%. An oracle, the quadtree of a finished 50-step dense draft from the same noise, is no better than the default (11.30). Rebuilding every step, every 25 steps or only once after the warm-up all land between 10.96 and 11.04. The one knob that moves FID is the warm-up length: 12.44 with none, 10.99 with five, 9.68 with twenty.
So the layout only has to be roughly right. Good, because the warm-up is not free. JiT samples with 50 Heun steps under classifier-free guidance, four network calls per step, the last step Euler: 199 calls. Five dense steps plus the estimate the first layout is built from are 21 of them, at full price (reported, Table 21).
Put the reported FIDs on a compute axis and a trade appears (reasoned, using the block FLOP ratio for $N = 256$, $D = 768$). Twenty warm-up steps at a 25% budget cost about 55% of the dense FLOPs and score 15.73. Five warm-up steps at a 50% budget cost about 54% and score 10.99. At equal compute, spend it on tokens, not on a longer warm-up.
A one-step generator has no trajectory to read an estimate from. For the one-step pixel MeanFlow model the main text says BudgetPix uses "the random layout", but the appendix describes something else: "each image takes the entropy quadtree of one of 20 training images of its class, picked by the image index and bisected to the budget" (reported, App. B.6). The one-step layouts are borrowed from real ImageNet photos of the same class: a fair prior for class-conditional ImageNet, and one that does not exist for a text prompt.
The wall-clock runs use a layout "built once, then frozen" after the warm-up, not rebuilt every five steps (reported, Table 21); Table 27 scores the two the same, so the timed sampler is close to, but not literally, the evaluated one. The paper also names the obvious failure in its conclusion: the layout comes from intermediate predictions, "so it can miss details they do not yet show".
## Checking the headline claims
### "25% of the original compute budget"
The abstract says BudgetPix matches MiniT2I-L at 512² and PixelDiT at 1024² "using just 25% of the original compute budget". Figure 1's caption settles what the percentage is: "Percentages show the fraction of tokens used." Table 21 lists the budgets as token counts. The compute budget is the token budget, applied after the dense warm-up.
For MiniT2I-L, one block at 25% of the tokens is 22.7% of the dense FLOPs (reasoned, from the formula above). MiniT2I samples with 100 Euler steps, two calls each, plus one call for the estimate: 201 calls, 11 of them dense. That puts the whole sample at about 27.0% of the dense FLOPs (reasoned; it ignores the text tokens and the added modules). The paper's own measurement is a 3.59× wall-clock speed-up on one H100, which is 27.9% of the released model's time (reported in Table 6; the inversion is mine). FLOPs and time agree here, but not everywhere. At the same 25% budget, the measured speed-ups are 1.95× for JiT-L/32, 1.95× for PixelDiT and only 1.36× for JiT-B/32 (reported, Table 6). For JiT-L/32 my FLOP estimate is about 32% of dense, warm-up included, against a measured 51% of the time (reasoned against reported). Short 256-token sequences do not keep an H100 busy on matrix FLOPs (my reading). PixelDiT's floor is its pixel stage, which the paper names as a limit.
Is BudgetPix's overhead counted? Yes, and it rounds to zero: BudgetPix at 25% on MiniT2I-L takes 1.376 s per image, exactly what the untrained base model takes at the same budget through the same sampler (reported, Table 7). Token count buys the speed. BudgetPix makes the low-token images worth having.
The cheap end of the dial buys nothing. At a 90% budget MiniT2I-L is 1.01× and JiT-B/32 0.92×, and PixelDiT is 0.79×, slower than dense (reported, Table 6). Table 21 notes that PixelDiT's attention is masked when a reduced budget pads the layout, which is my best guess at the cause. From about 80% down the gains start.
Then quality. GenEval at 25% is 0.874 against the released MiniT2I-L's 0.882, and 0.725 against PixelDiT's 0.721 (reported, Tables 1 and 5). With 553 prompts of four images each, GenEval's standard error near 0.87 is roughly 0.007 (reasoned, treating it as one proportion), so the MiniT2I gap is noise. The PixelDiT comparison flatters BudgetPix: its own dense score is 0.741, because the fine-tune on PD12M improved PixelDiT before any tokens were cut (held-out FID 43.7 to 36.2 on that data, reported in Table 17). Against its own dense grid, BudgetPix at 25% loses 0.016 GenEval and 1.5 DPG-Bench points. "Matches the fidelity" holds for PixelDiT against the released model, not against itself.
### "0.8-point increase in FID" with MeanFlow
The table is App. B.6, Table 15: the one-step pixel MeanFlow model pMF-L/16, class-conditional ImageNet at 256², 256 tokens dense. BudgetPix scores FID-50k 2.65 at 100% and 3.46 at 60%, which is 154 tokens (reported). That is the 0.81 in the introduction. The baseline is BudgetPix's own dense score; the released model reads 2.54 in the same protocol, so against it the rise is 0.92 (reasoned). And 0.81 on a base of 2.65 is a 31% rise in FID (reasoned). "Near-zero quality degradation" is generous for that.
The other rows are what make it impressive. At 60%, the released weights fed BudgetPix's layouts score 257.20, ToMe 36.04 and FeatSim 100.26 (reported). Even at 25% BudgetPix is 5.44. Nothing times the one-step model, so there is no wall-clock behind this result.
### "Human and VLM judges" against "prior budget-adaptive baselines"
The baselines are the base model (released weights pushed to the budget through BudgetPix's sampler, no training), ToMe and FeatSim (two token-merging methods wrapped around the released weights), and RTI, a LoRA adapter that consumes region tokens, run from its released checkpoint. RTI, the only real competitor, is concurrent (August 2026), not prior, and exists only for MiniT2I, so the studies run on MiniT2I-L at 512².
Both studies measure faithfulness to each method's own full-budget image, not quality. The first asks at which budget a difference first becomes visible. People (50 per form, 20 GenEval prompts) say BudgetPix can be cut by 51.8% [48.0, 55.7] before they notice, against 38.0% [31.8, 44.0] for RTI and 32.5% or less for the rest (reported, Table 20). A Gemini 3.8 Flash judge on 100 prompts gives 36.2% against 21.1% for RTI. The intervals do not overlap; that part holds.
That contest is closer. Head to head, people prefer BudgetPix to RTI 63.5%, 63.4%, 57.2% and 62.3% of the time at 80, 60, 40 and 20% budgets, ties counted as half (reported). At 40% the interval is [45.3, 69.5], which includes a coin flip. The VLM judge at 40% gives 54.5% [46.0, 63.0], also including 50%. "Significantly improved" over RTI holds at three of four budgets.
RTI does not collapse like the merging methods: at 25% on MiniT2I-L it scores 0.859 GenEval, 83.0 DPG-Bench and 1.16 ImageReward against BudgetPix's 0.874, 84.3 and 1.17 (reported, Table 5). Where RTI loses is speed, 2.16× against 3.59× at that budget, and pixel faithfulness (Figure 7, bottom row).
## What I would watch for
- **No code.** Every number here is the authors'. The appendix is unusually complete, so the claims will be checkable once code lands.
- **Entropy is not detail.** Flat graphics, text and line art have sharp structure and low entropy; the layout spends tokens on noisy texture before clean edges.
- **Tokens are not time.** With short sequences like JiT's 256, or a full-resolution stage like PixelDiT's, wall-clock gains sit well under the token cut.
- **The warm-up is a fixed tax.** About a tenth of JiT-L/32's calls run dense before the dial does anything. Fewer sampling steps shrink the trajectory the layout is read from; the one-step case already needed borrowed layouts.
- **Below 25%, you are extrapolating.** The training offsets cover roughly a quarter of the grid and up.
Neighbours on this site attack the same cost elsewhere. [LoT Diffusion](/articles/lot-diffusion) takes the layout as an input from a planner. [MrFlow](/articles/mrflow-diffusion-acceleration) moves the cost along the time axis. [EVA](/articles/eva-empirical-vae) works on the latent side BudgetPix avoids. For why token count grows with the square of the resolution in the first place, see the [Diffusion Transformer page](/architectures/diffusion-transformer#tokens-grow-with-the-square-of-the-resolution).
## How I checked
I read the paper in the arXiv HTML (v1, 8 October 2026) and, where the HTML dropped text after a less-than sign, in the project page's PDF. Figures 4, 5 and 7 are the project page's full-resolution copies, Figure 6 is from the arXiv HTML, and Figure 41 is a 300 dpi render of PDF page 55. I shallow-cloned `ozgurkara99/budgetpix` and queried the Hugging Face API.
The FLOP ratios, warm-up accounting, GenEval standard error, inverted speed-ups and MeanFlow percentages are my arithmetic on the paper's tables. The FLOP model is one standard block with an MLP ratio of 4; it ignores conditioning tokens, the added modules and kernel efficiency, so treat it as a direction. The quadtree widget implements the appendix's rule on a scene I drew; its token counts are measured on that scene only. I could not run BudgetPix or reproduce any reported number.
---
# Data-space iteration: a four-step image generator that never asks what time it is
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/data-space-iteration
> date: 2026-10-13
> tags: diffusion, flow-matching, distillation, image-generation, diffusion-transformers
When I took FastH3 apart a few weeks ago, one line in its checkpoint metadata stuck with me: `dmd_denoising_steps: [999, 749, 500, 250]`. Four network calls, at four fixed noise levels, for every video, every prompt, every pixel. As far as I could tell nothing about it was specific to that model. It is the uniform trailing schedule that few-step work on diffusion transformers treats as the default, and in [the FastH3 write-up](/articles/fastvideo-fasth3) it was simply there, baked into a 33B student.
So when Shanchuan Lin (Peter Lin on X) posted [Few-Step Generation via Data-Space Iteration](https://arxiv.org/abs/2610.12102), with a thread that opened on exactly that complaint, I read it the same day. Lin is the person behind SDXL-Lightning, AnimateDiff-Lightning and the adversarial post-training work at ByteDance Seed, so few-step distillation is his day job. His claim is that the schedule is not a hyperparameter to tune better. It should not exist.
What surprised me is how small the change is. The paper's own code puts the new rollout next to the old one, and the difference is about five lines. The training loss is untouched. Everything interesting is in *what gets passed from one network call to the next*.
## What a flow model actually gives you
Start from the model being distilled. A flow-matching model is trained on straight lines between a data sample $x_0$ and Gaussian noise $x_1$. The paper uses the convention where $t = 0$ is clean and $t = 1$ is pure noise:
$$
x_t = (1 - t)\,x_0 + t\,x_1, \qquad t \in [0, 1].
$$
The network $v_\phi(x_t, t)$ learns the velocity $\mathrm{d}x_t/\mathrm{d}t$, whose per-pair target is $x_1 - x_0$. To sample, you draw noise and integrate the ODE from $t = 1$ back to $t = 0$. The teacher here is the public SiT-XL/2 checkpoint, and the paper's Table 3 shows the price of doing that properly: FID 8.26 at 250 network calls without guidance, 2.06 with guidance, which doubles the calls to 250×2.
One identity carries most of what follows. If you know the velocity at $(x_t, t)$, you can jump straight to a guess at the clean endpoint:
$$
\hat x_0 = x_t - t\, v_\phi(x_t, t).
$$
This is $x$-prediction. It is exact for the straight line of a single pair, and for the learned average velocity it gives the average of all clean images consistent with $x_t$. At high noise that average is blurry, because many different images are consistent with almost pure noise. Pixel-space models such as JiT go further and make the network output $\hat x_0$ directly, for a different reason (noise is high-dimensional and the data manifold is not), which I covered in [the PixelUMM piece](/articles/pixelumm-pixeldense#predict-the-clean-image-not-the-noise). Keep the identity in mind. Data-space iteration turns out to be this formula evaluated at $t = 1$, forever.
## Two ways to spend four calls
Distillation trades the 250-call integral for a generator $G_\theta$ that gets there in a handful. The paper sorts existing few-step generators into two families, and the split is about what state crosses the gap between calls.
The first family, which the paper calls flow-renoise, passes a noisy image at a chosen time. At schedule point $t_r$ the generator predicts a clean image, and the sampler mixes that prediction with *fresh* noise to land at the next point:
$$
\hat x_0 = G_\theta(x_{t_r}, t_r), \qquad \hat x_{t_{r+1}} = (1 - t_{r+1})\,\hat x_0 + t_{r+1}\,\varepsilon, \quad \varepsilon \sim \mathcal{N}(0, I).
$$
Multistep consistency-model sampling works this way, and so does the original DMD. Every intermediate output is a complete image, which is convenient: you can stop after any call.
The second family, flow-map methods, also passes a noisy image, but learns the jump itself. The generator takes $x_{t_r}$ to $x_{t_{r+1}}$ directly, with no fresh noise:
$$
\hat x_{t_{r+1}} = G_\theta(x_{t_r}, t_r, t_{r+1}).
$$
Progressive distillation, SDXL-Lightning, shortcut models and MeanFlow are in this family by the paper's sorting. MeanFlow learns an average velocity $u$ over an interval and steps $x_{t_{r+1}} = x_{t_r} - (t_r - t_{r+1})\,u$; Phased DMD, the baseline the paper uses here, trains each interval with a distribution-matching loss. The intermediate states are not images, they are noisy latents at time $t_{r+1}$, so a four-step flow-map model only has a four-step answer.
Both families need a list $1 = t_0 > t_1 > \cdots > t_R = 0$. For a model you sample with an ODE solver, that list is an inference-time choice. For the generators that matter in production it is not, because they are trained by **self-rollout**: during training, the generator runs its own earlier steps to produce the inputs for the step being trained. Streaming video distillation such as [Self Forcing](https://arxiv.org/abs/2506.08009), the recipe behind [StreamRender-H3](/articles/streamrender-h3), works this way. The states the student sees during training are defined by the schedule, so a different schedule means a different training run. Lin's thread says this is where the idea came from: choosing a discretization was a practical problem while they were working on autoregressive streaming video with self-rollout distillation.
There is also a softer argument in the introduction. One global $t$ advances every sample, and every spatial location within it, through the same denoising schedule, and there is no reason a goldfish and an ostrich, or a sky and a face, want the same step sizes. I will come back to how much of that the paper actually tests.
## The change: pass the clean guess
Data-space iteration keeps a clean-image estimate as the state. Start it at zero, then apply one shared generator $R$ times, handing it the current estimate, the *original* noise $z$ and the step index $r$:
$$
\hat x_0^{(0)} = \mathbf{0}, \qquad \hat x_0^{(r+1)} = G_\theta\!\left(\hat x_0^{(r)},\, z,\, r\right), \quad r = 0, \ldots, R-1.
$$
No fresh noise is added between calls, so nothing the model already got right is destroyed. No intermediate flow time is ever constructed, so there is no schedule to pick. Each call is trained to output the best finished sample it can, given what the previous call produced.
The implementation is where it gets neat. They build $G_\theta$ from the pretrained velocity network with three edits: the noise $z$ goes through the original input projection; the current estimate gets its own new projection, summed with the first; an embedding of $r$ is added to the timestep-conditioning path. The original time input is pinned at $t = 1$, and the output is
$$
\hat x_0^{(r+1)} = z - g_\theta\!\left(\hat x_0^{(r)}, z, r, t = 1\right).
$$
This is the $x$-prediction identity from above with $t = 1$ and $x_1 = z$. So every call is the teacher's one-jump-from-pure-noise prediction, with a side channel carrying the previous answer. Read the initialization with that in mind. The new projection has no bias (Appendix A), and the first estimate is all zeros, so on the very first call the new input contributes exactly nothing. At initialization, call one *is* the teacher's single-step $x$-prediction from noise, plus the $r$ embedding drawn from $\mathcal{N}(0, 0.02)$. Training only has to teach the later calls how to use the side channel.
The appendix gives the parameter bill, and I redid it because "the model gets smaller" is a strange claim for a method that adds inputs. SiT-XL/2 has 675,129,632 parameters. The paper says the new projection and the $r$ embedding add 24,192, and dropping the timestep MLP in favour of a constant $t = 1$ embedding removes 1,626,624, for 673,527,200. The total checks out: 675,129,632 + 24,192 − 1,626,624 = 673,527,200. The pieces mostly check out too. A bias-free 2×2 patch projection from a 4-channel latent into 1,152 dimensions is 16 × 1,152 = 18,432 weights, and the remaining 5,760 is exactly five 1,152-wide vectors, which fits an embedding table for the step index with one spare row. DiT's timestep MLP (256 → 1,152 → 1,152, with biases) is 1,624,320 by my count, 2,304 short of the paper's figure. That gap is two 1,152-wide vectors and I could not tell from the text what they are. The saving is real either way, and it comes from the fact that a model that never sees $t$ does not need the network that embeds it.
## Training it with DMD, unchanged
The paper's second claim is that this is a drop-in change for distribution matching distillation. To see why, it helps to see what DMD optimises.
DMD wants the generator's output distribution to match the data distribution, and measures the mismatch with a reverse KL. The gradient of that KL with respect to a generated sample is the difference of two scores: the data's score and the generator's own score. Neither is available directly, so DMD estimates both at a blurred version of the sample. Take a generated $\hat x_0$, noise it to a random level $\tau$, and ask two flow models for their velocity there:
- $v_\text{real}$: the frozen teacher, which knows the data distribution at every noise level.
- $v_\text{fake}$: a copy of the teacher that is continually retrained on the generator's own noised samples, so it knows the *generator's* distribution.
Where the two disagree, the generator's samples are in the wrong place, and the difference points the way to move them. In the paper's velocity form:
$$
\mathcal{L}_\text{dmd}(\theta) = \mathbb{E}\left[\, w(\tau)\; \mathrm{sg}\!\left[v_\text{real}(\hat x_\tau, \tau) - v_\text{fake}(\hat x_\tau, \tau, r)\right]^{\top} \hat x_0^{(r+1)} \right],
\qquad \hat x_\tau = (1 - \tau)\,\hat x_0^{(r+1)} + \tau\,\varepsilon.
$$
The stop-gradient makes the bracket a fixed arrow, and the dot product with $\hat x_0$ turns it into a gradient on the generator. The fake model is trained alongside with a plain flow-matching loss on the generator's noised samples, four fake updates per generator update here.
Nothing in that loss cares how $\hat x_0^{(r+1)}$ was produced, and the drop-in claim rests on exactly this. Training picks a step $r$, runs the first $r$ calls without gradients, runs call $r+1$ with gradients, and applies DMD to its output. The only addition for data-space is that $v_\text{fake}$ is conditioned on $r$, because each step has its own output distribution to track. The paper's Algorithm 1 prints the two rollouts side by side; this is the data-space one, verbatim:
```python
# arXiv 2610.12102, Appendix E, Algorithm 1
def rollout_data_space(gen, z, c, r, track_grad):
x_hat = torch.zeros_like(z)
for i in range(r + 1):
with torch.set_grad_enabled(track_grad and i == r):
x_hat = z - gen(x_hat, z, c, i)
return x_hat
```
and the flow-renoise one it replaces:
```python
def rollout_flow_renoise(gen, z, c, r, track_grad, R=4):
x_hat = torch.zeros_like(z)
for i, t_r in enumerate(torch.arange(R, 0, -1, device=z.device)[:r + 1] / R):
t_r = t_r.expand(z.shape[0])
t_r4 = t_r[:, None, None, None]
eps = z if i == 0 else torch.randn_like(z)
x_tr = (1.0 - t_r4) * x_hat + t_r4 * eps
with torch.set_grad_enabled(track_grad and i == r):
x_hat = x_tr - t_r4 * gen(x_tr, t_r, c)
return x_hat
```
Put them next to each other and the whole paper is visible. The renoise loop computes a time $t_r$ from the schedule (`arange(R, 0, -1) / R` is the uniform 1, 0.75, 0.5, 0.25), draws fresh noise after the first call, builds a noisy input, and converts the network's velocity to $\hat x_0$ with $x_t - t\,v$. The data-space loop has no time, no fresh noise, and converts with $z - v$. Same `dmd_step` afterwards. Since only one call per training step carries gradients either way, there is no extra memory or compute against a matched DMD baseline, which the paper states and the code makes obvious.
The setup is deliberately bare. No GAN loss, no extra regression term, which the paper says is so the iteration method is the only thing that varies. $R = 4$, Adam at $10^{-4}$, batch 256, 40 epochs, $\tau \sim \mathrm{Beta}(2.5, 1)$ with $w(\tau) = 1$, EMA 0.9999, TF32. The appendix says every model reproduces in about a day on one eight-GPU server with 80 GB cards.
## Watching three rollouts on a toy
There is no released code beyond the two algorithms in the appendix, and a SiT-XL/2 DMD run is a day on eight 80 GB GPUs. So I did the next best thing and ran the paper's Algorithm 1 and 2 on a problem small enough to see: a ring of eight Gaussian blobs in 2D, where the teacher's velocity can be written down exactly.
The setup copies the paper wherever it can. Three small generators (three hidden layers of 64, SiLU) get the three rollouts from the appendix: data-space and flow-renoise from Algorithm 1, flow-map with Phased DMD's re-noising from Algorithm 2. All three train with the same DMD step, $\tau \sim \mathrm{Beta}(2.5, 1)$, $w = 1$, Adam with $\beta_1 = 0$, $r$ cycling through 0 to 3, the uniform schedule, 6,000 generator updates and three seeds each. I made two shortcuts that only work in 2D. The "real" velocity is the mixture's exact probability-flow velocity, not a teacher network. The "fake" velocity is computed exactly from a fresh batch of 512 of the generator's own samples at the same step, rather than learned by a second network. Nothing here starts from a pretrained model, and nothing here is ImageNet.
Two things came out, and only one of them agrees with the paper.
The behaviour reproduces cleanly. I took 240 shared noise draws and asked which of the eight blobs each one lands in after the first call and after the fourth. Under flow-renoise, 198 of the 240 end up in a different blob from where call one put them. Under data-space, 14 do. Figure 2's goldfish-to-carrot effect survives the trip down to two dimensions: re-noising to $t = 0.75$ throws away which mode you were in, and the next call picks again. The mean distance a point moves between calls tells the same story, about 1.86 then 0.89 then 0.30 for flow-renoise against 0.26, 0.12 and 0.07 for data-space, averaged over seeds. Follow one dot in the widget and you can watch it.
The ranking does not reproduce. I scored each method with a sliced Wasserstein distance against 20,000 draws from the ring, scaled by 100; two independent draws of the ring itself score 1.39. At four calls, averaged over the three seeds, flow-map scores 4.33, data-space 6.17 and flow-renoise 6.73. Data-space does improve with every call, 9.40 then 7.05, 6.39 and 6.17, while flow-renoise barely moves from 6.98, so the "every extra call buys something" shape is there. But in this toy the flow-map generator wins clearly, and data-space is the worst of the three after one call, which is the CIFAR-10 pattern rather than the ImageNet one.
I do not read that as evidence against the paper. A 2D ring has nothing like the per-sample and per-region variation the paper's argument relies on, my generators are tiny and start from scratch rather than from a teacher, and my exact "fake" is not the learned one. What the toy does show is that the mechanism is real and does not need ImageNet to appear: passing a clean guess instead of a re-noised one turns iteration from resampling into editing. Whether editing also wins on quality is the empirical question, and for that the paper's tables are the evidence, not my ring.
## The numbers, and what they are measured against
Read the table in two passes, as the paper does.
The first pass is the standard setting: everyone uses the uniform schedule [1.0, 0.75, 0.5, 0.25], which the paper notes is what prior few-step work on this architecture uses. Data-space wins every column. Without guidance it reaches 2.46 at four calls against 2.75 for DMD and 2.55 for Phased DMD; with guidance 2.29 against 2.61 and 2.86. At one call it is also ahead, 3.85 against 4.45, which I come back to below because the paper itself does not think that gap means much.
The second pass is the honest one. The flow methods get an "oracle": three schedules, each a separately trained generator, best one reported. The schedules come from the shift function used by Stable Diffusion 3,
$$
\operatorname{shift}(t;\gamma) = \frac{\gamma t}{1 + (\gamma - 1)\,t},
$$
with $\gamma$ of 0.5, 1 and 2, which I recomputed to [1.00, 0.60, 0.33, 0.14] and [1.00, 0.86, 0.67, 0.40], matching Table 2(c). The board below lets you flip through them.
With the oracle, the margin mostly closes. Unguided, the best flow-map schedule reaches 2.48 against data-space's 2.46. The gap is 0.02, and the appendix says the reported FIDs for these two methods are means of three training runs with a standard deviation below 0.05, so the fair reading is a tie. With guidance, flow-map at $\gamma = 2$ gets 2.37 against 2.29, a gap of 0.08, which is outside that spread but not by a lot. Flow-renoise never catches up under any schedule.
So the claim in the abstract is worded correctly: data-space "outperforms standard discretization baselines and matches or improves upon variants selected through schedule search". The real point is the cost of the oracle. Getting 2.48 took three flow-map training runs. Getting 2.46 took one, and there was nothing to search.
Two more things in the board are worth staring at. Under the flow schedules, which $\gamma$ wins is not even consistent: unguided flow-map prefers $\gamma = 2$ (2.48), flow-renoise at four calls prefers $\gamma = 2$ too (2.72) but at two calls prefers 0.5 (2.84), and with guidance flow-renoise at one call prefers $\gamma = 2$ while at three it prefers uniform. The paper's words are that the shift "does not exhibit a predictable correlation with final performance". To me this is the strongest evidence in the paper for its motivation, and it is not a FID win at all. It shows that the knob being removed was not one anyone could set by reasoning.
The second is the step curve. Flow-renoise gets *worse* from three calls to four on the uniform schedule, 2.67 to 2.75 unguided and 2.51 to 2.61 guided. Data-space flattens instead: 2.47 to 2.46, and 2.33 to 2.29. The paper says that training with more than four steps did not improve FID further, so four looks like where this model's capacity runs out, which is what the abstract means by "the best sample permitted by its capacity".
### The ablations that set the recipe
Table 1 is a search over the training recipe for the data-space generator, and some of it transfers to any DMD setup. The weighting of the noise level $\tau$ matters a lot. Two weightings that correspond to the original score-based DMD update, $(1-\tau)^2/\tau$ and $(1-\tau)/\tau$, blow up near $\tau = 0$ and diverge even with $\tau$ clipped at 0.02. Weighting low noise heavily with $1 - \tau$ gives 5.66 at four steps. Constant weight gives 3.61; weighting by $\tau$ gives 2.87; by $\tau^2$, 2.99. Then they swap weighting for importance sampling, drawing $\tau$ from $\mathrm{Beta}(2.5, 1)$ with constant weight, and get the final 2.46. The appendix shows why the Beta family is the natural next step: $\mathrm{Beta}(a, 1)$ has density $a\tau^{a-1}$, so $\mathrm{Beta}(2, 1)$ is the sampling twin of $w = \tau$ and $\mathrm{Beta}(3, 1)$ of $w = \tau^2$. The paper's explanation for why sampling beats weighting is simple: a minibatch stops spending rows on timesteps that then get almost no weight, and the generator and the fake model now see the same $\tau$ distribution.
The rest is short. Conditioning the generator on $r$ helps: without it, 2.63 at four steps instead of 2.46. Adam with $\beta_1 = 0$ beats the usual 0.9, 2.46 against 2.66, so momentum hurts here. And an optimal-transport regularizer, which pulls every step's output toward the same transport of $z$, slightly hurts (2.47). The model does not need to be a fixed point.
### A number the paper does not headline
Table 2(a) has one row I keep coming back to. To match Phased DMD exactly, they also trained data-space with Phased DMD's $\tau$ sampling, where step $r$ only sees $\tau \in [t_{r+1}, 1]$. That model's first call is useless, FID 44.91, because it was only ever trained against high-noise targets. But its third call reaches 2.24, the lowest unguided ImageNet FID in the paper, below the headline model's 2.46 and below the guided 2.29.
I think the paper is right to leave it out of Table 3, because it gives up the property that every prefix of the rollout is a usable image. Still, it says something: the data-space generator, given a curriculum where early steps only learn coarse structure, spends its later calls better. Call it a schedule sneaking back in through the loss, and nobody has searched it.
## Refine, or replace
The picture that convinced me is Figure 2. Read across a flow-renoise row and the content changes at every call: in the goldfish row the scene goes from a pink blur to a goldfish, then to a hand holding a carrot, then to two red fish. Fresh noise is doing exactly what it was asked to. Each call keeps only what survives being mixed back to noise level $t_{r+1}$, and at 0.75 that is not much. The data-space row keeps the same fish, in the same pose, and sharpens it.
The appendix puts a number on it. Table 6 measures the average L2 change between consecutive outputs over 50k samples, decoded to pixels in [0, 1]:
| change between calls | 1 → 2 | 2 → 3 | 3 → 4 |
|---|---|---|---|
| flow-renoise | 82.72 | 51.18 | 29.91 |
| data-space | 59.65 | 44.25 | 35.98 |
Flow-renoise changes the image more in the first two transitions, as the pictures show. The detail I did not expect is the last column: data-space moves *more* than flow-renoise between calls three and four, 35.98 against 29.91. The renoise sampler at $t = 0.25$ is mostly polishing. The data-space generator is still doing real work at its last call, and its steps shrink slowly rather than collapsing. If you squint, that is the per-sample "step size" the introduction asked for, chosen by the network rather than by $t$. But the paper does not measure it per sample or per region, and the motivating claim that "spatial regions with different frequency content might benefit from different forms of refinement" stays a hypothesis here. Figure 3 shows signed per-pixel differences that concentrate on edges and the subject for data-space and spread over the whole frame for renoise, which is suggestive, on four images.
The full-metric table in Appendix C adds some texture to the FID story. At four steps without guidance, data-space has recall 0.59 against Phased DMD's 0.56, but lower precision (0.81 against 0.83) and a lower Inception Score (228.59 against 245.76). So the FID win comes from coverage more than from sharpness. That fits a generator that keeps its first composition, rather than one that gets three chances to land on a stereotypical class image. It is also the kind of trade-off that FID alone hides.
### The one-step result is not evidence
At one call, data-space and flow-renoise are the same thing: one network evaluation from pure noise to an image. The paper says so in its CIFAR-10 appendix, and the CIFAR numbers show why to take the ImageNet one-step gap (3.85 against 4.45) with salt. On CIFAR-10 with a 55M UNet, data-space is *worse* at one step, 12.20 against DMD's 11.21. From two steps on it pulls well ahead, 8.28 against 11.14, and at four 8.62 against DMD's 10.03 and Phased DMD's 11.72. The authors attribute the one-step difference to optimisation dynamics and ask readers to judge the method at more than one step. I agree, and I would extend the caution a little: if optimisation alone can move one-step FID by more than half a point in either direction, a 0.02 gap at four steps is noise and an 0.08 gap is a lean, not a law.
One more oddity worth knowing: every distilled model here beats the unguided teacher by a mile, 2.46 to 2.75 against the teacher's 8.26. The paper credits DMD's reverse-KL objective, which is mode-seeking. It means unguided comparisons against the teacher flatter everyone. The guided teacher at 2.06 still beats them all, at about 125 times the calls of a four-step guided student (500 against 4).
## Where it sits
The paper is careful about relatives, and I think the comparisons are fair.
Self-conditioning (Analog Bits and its descendants) also feeds the model its previous prediction. The difference is what the iteration is. Self-conditioning still integrates an ODE and uses the previous $\hat x_0$ to improve the velocity at the next time; here the data estimate *is* the state and there is no ODE.
Deep equilibrium models solve for $x^\ast = G(x^\ast, z)$. Data-space iteration is not trying to converge: the step index changes the update, and the loss is applied at every step rather than at the fixed point. The failed optimal-transport regularizer is the experiment that tests the equilibrium intuition, and it loses.
Looped transformers reuse blocks to get depth, which makes them the closest relative of something this site covered last week, [LiFT](/articles/lift-loop-flow-transformer), which loops a DiT core inside each step and indexes loops with a continuous depth coordinate (the [looped-transformer page](/architectures/looped-transformer) has the background). Lin's own Adversarial Flow Models paper trained a looped generator end to end through the unrolled loop. Data-space iteration gets the loop without backpropagating through it: only one call carries gradients per training step. This is also why it drops into an existing DMD codebase without new memory tricks.
JiT came up in the replies, where someone asked whether this is the same idea as "Back to Basics", since both talk about data-space prediction. Lin's answer: "JiT's data prediction is designed to avoid the Transformer bottleneck, but it still operates within the probability-flow framework." I think he is right. JiT changes what one network call outputs; this paper changes what a sampler iterates on.
## What I would do with it
If I were distilling a model with self-rollout today, especially a streaming video model where the schedule is baked into training, I would try this first. The reasons are practical. It removes a hyperparameter that the paper shows cannot be set by reasoning. It costs nothing at training time. It makes the model slightly smaller. Every prefix of the rollout is a usable image, so the step count can be picked at inference, up to the trained maximum. Lin's thread adds a recipe for large models: when an ODE-based initialisation is preferable, first consistency-distil to a one-step model, then apply data-space DMD.
What I would not do is read Table 3 as "data-space beats flow-map". Against a searched flow-map schedule it ties without guidance and leads by 0.08 with it. The evidence is one benchmark (ImageNet 256, class-conditional, one 675M backbone) plus a CIFAR-10 appendix, and the paper itself leaves text-to-image and video for future work, which is exactly where the motivation came from. The GAN and regression losses that production DMD stacks use were left out on purpose, so how the gain survives inside those is open. There is no code release beyond the two appendix algorithms, and no checkpoints.
The idea I expect to stick is the framing: the thing passed between calls is a design choice, and "a noisy image at a time someone picked" was never the only option.
## How I checked
Everything in the paper sections comes from arXiv 2610.12102v1: the equations from §2, the training recipe from §3.1 and Table 4, the FIDs from Tables 1, 2, 3, 5 and 8, the step-to-step L2 numbers from Table 6, the code from Algorithms 1 and 2. Figures 1 to 3 are cropped from the PDF; the Table 3 image is the one Lin posted on X, which I compared against the HTML table cell by cell. I recomputed the parameter arithmetic (it adds up; 2,304 parameters of the timestep MLP's figure I could not attribute), the shifted schedules (they match Table 2(c)), and the 125× call ratio. The thread and replies came from the fxtwitter mirror of post 2109634731142787145. I looked for a code release on Lin's site and under the ByteDance-Seed and PeterL1n GitHub accounts and found none. The toy is my own numpy re-implementation of Algorithms 1 and 2 (single-threaded, about 100 seconds per run on a CPU), described in its section; its numbers are mine and say nothing about ImageNet.
---
# DiffusionART: why DiffusionNFT needed EMA, and the 12x that follows from fixing it
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/diffusionart
> date: 2026-10-13
> tags: diffusion, reinforcement-learning, video-generation, flow-matching, post-training, grpo
The post that sent me here said Kling AI's new diffusion RL method trains video models with "12x fewer GPU hours than DiffusionNFT and Flow-GRPO-Fast." My first reaction to any GPU-hour ratio in RL is suspicion. Two reward curves never stop at the same place, so a ratio between them depends on which point you pick, which run you pick, and what each method counts as a step.
So I read the paper ([arXiv:2610.12572](https://arxiv.org/abs/2610.12572), Zhenglin Wan and colleagues at NUS HPC-AI Lab, Kling AI and HKUST), its LaTeX source, the training code at [github.com/vanzll/DiffusionART](https://github.com/vanzll/DiffusionART), and the two checkpoints in the [Hugging Face collection](https://huggingface.co/collections/vanzl/diffusionart-6acc498d285b7e7c4dbd16b7).
The 12.02x holds up as arithmetic. It just comes from a different run than the paper's main video experiment, and two of the four milestones behind it are easier to read once you know a DiffusionART "step" is half a batch. What I did not expect is that the most useful part of the paper is not the speedup at all. It is a diagnosis of why DiffusionNFT, the method it beats, needs an exponential moving average to stay alive. That diagnosis is clean, it comes with ablations, and it explains the fix.
## Two ways to turn a score into an update
Online diffusion RL has the same problem every time. You generate a batch of images or videos, a reward model scores each finished sample, and that single number has to become a gradient for a network that was called sixteen or thirty times along the way. I walked through the landscape in the [DiffusionOPSD article](/articles/diffusionopsd); the short version is that there are two families.
**Reverse-process methods** like Flow-GRPO and DanceGRPO treat each denoising step as a stochastic policy action. That needs an ODE-to-SDE conversion so a transition has a likelihood, then a PPO-style ratio on sampled transitions, weighted by a group-normalised advantage. It trains on states the model actually visited. Its weakness, in the paper's words, is credit assignment: each transition gets pushed along its own sampling noise, scaled by an advantage that the transition only partly caused. Flow-GRPO is also what [the AUK speech-editing model](/articles/auk-speech-editing) used for its TTS stage, so it is not just an image technique.
**Forward-process methods** like DiffusionNFT skip likelihoods entirely. Take the clean samples, re-noise them with fresh noise, and regress the velocity with a reward-weighted positive branch and a negative branch. It is supervised flow matching with weights, which is why it is fast. It is also, per its own authors, unstable without two things: training both branches together and rolling out from an EMA copy of the policy whose decay has to be tuned per task.
DiffusionART keeps NFT's regression and drops the re-noising. That one change is the whole method, and the paper spends its first half explaining why it matters.
## What NFT's loss is really asking for
DiffusionNFT maps each sample's raw reward to a goodness weight $r \in [0,1]$ by centering within the prompt's group and clipping. With $\beta > 0$ it builds $v_{\theta,\pm} = v_{\text{old}} \pm \beta(v_\theta - v_{\text{old}})$ and minimises
$$
\mathcal{L}_{\text{NFT}} = \mathbb{E}\left[ r\,\lVert v_{\theta,+} - u_t \rVert^2 + (1-r)\,\lVert v_{\theta,-} - u_t \rVert^2 \right]
$$
where $u_t = x_0 - \epsilon$ is the rectified-flow velocity target for a sample $x_0$ re-noised to $x_t$. The paper rewrites this, usefully, as a single regression (its Equation 1): with $a = (2r-1)/\beta$, the loss equals $\beta^2\,\mathbb{E}\lVert v_\theta - v_{\text{tgt}}\rVert^2$ plus a constant, where
$$
v_{\text{tgt}} = a\,u_t + (1-a)\,v_{\text{old}}.
$$
A good sample pulls the velocity toward its own noising velocity; a bad one pushes away from it. At the on-policy point, every sample's force is proportional to $(2r-1)(u_t - v_{\text{old}})$.
That residual, $u_t - v_{\text{old}}$, is where the trouble hides.
## The puzzle: each branch alone collapses
DiffusionNFT's motivating result is that the ideal velocity fields of the positive, original and negative endpoint distributions are collinear. If you move toward the positive field, you are moving away from the negative one. Either branch on its own should be enough.
It isn't. The paper reruns NFT without EMA and trains each branch separately. Both collapse to a GenEval score of zero within about ten steps, while the joint objective climbs to around 0.9.
The explanation, Proposition 1, comes from separating two velocity fields that NFT quietly treats as one. There is $v^{\text{old}}_{\text{rvs}}$, the field the sampler actually uses to generate the endpoints. And there is $v^{\text{old}}_{\text{fwd}}$, the ideal velocity you would get by forward-noising those same endpoints and asking for the conditional mean of $u_t$. A perfectly trained flow model has the two equal. A real one does not. Call the gap $e = v^{\text{old}}_{\text{fwd}} - v^{\text{old}}_{\text{rvs}}$.
With $g$ the reward-improving direction (Lemma 1 shows it is a conditional covariance between the goodness weight and $u_t$) and $q$ the expected goodness at a given state, the branch forces at the on-policy point are
$$
\mathcal{F}_+ = g + q\,e, \qquad \mathcal{F}_- = g - (1-q)\,e, \qquad \mathcal{F}_{\text{NFT}} = 2g + (2q-1)\,e.
$$
The positive branch drags the velocity toward the forward field. The negative branch shoves it away. Together they agree on $g$ and disagree on $e$, and what survives is $(2q-1)e$: a force set by how much the sampler disagrees with its own samples, with no reward in it.
Prompt-level reward centering makes $q$ average to one half across a prompt's samples. That cancels the term on average, which is why joint training works at all. It does not cancel it at each state, and it does not make $\sum_i (2r_i - 1)\,e_i$ vanish, because $e_i$ varies from state to state. States that happen to sit in high-reward territory get pulled toward the forward field; low-reward states get pushed off it. The paper calls this the forward-reverse consistency force.
I like this argument because it predicts the ablation instead of explaining it after the fact. Remove one branch and you remove the cancellation entirely; the force is then a full $q\,e$ or $(1-q)\,e$, unopposed, and the model is told to fix a mismatch it cannot see in its reward. The proof in Appendix B.2 is short algebra on Equation 1, and the decomposition is exact rather than a first-order approximation.
## EMA was restoring, not going off-policy
The second half of the diagnosis is about EMA. DiffusionNFT samples from a lagged copy of the weights. The obvious story is that slow-moving rollouts help, the usual off-policy argument. The paper tests it and finds a different mechanism.
Proposition 2 splits the NFT loss with the drift $d = v_\theta - v_{\text{old}}$ and the residual $h = u_t - v_{\text{old}}$:
$$
\mathcal{L}_{\text{NFT}} = \mathbb{E}\left[-2\beta(2r-1)\langle d, h\rangle + \beta^2\lVert d\rVert^2\right] + \text{const}.
$$
The linear term carries the guidance and the consistency force. The quadratic term pulls the policy back toward $v_{\text{old}}$. On-policy, $d = 0$, so the restoring term contributes no gradient, and the residual consistency force has nothing to push against. EMA keeps $v_{\text{old}}$ lagged behind, so $d$ is nonzero and restoration switches on.
Two ablations back this. Keep EMA rollouts but replace $v_{\text{old}}$ with $v_\theta$ inside the target, so the data is off-policy but there is no anchor: it still collapses. Drop EMA but split every rollout into two minibatches and update once on each, so the second update sees a nonzero drift: it survives, at the price of reaching an OCR score of 0.9 in 320 steps instead of 21, which is the 15.2x slowdown on the left of the figure below.
Panel (c) is the most interesting number in the paper to me. Averaged over rollouts 0 to 17, the restoring term is 9.4 times smaller than the guidance in output space, yet its parameter gradient is 4.9 times larger. The authors' reading is that per-sample guidance points in many different directions across states and mostly cancels when the gradients are summed, while restoration points consistently back toward the anchor and survives. So in NFT, a weak restoring force ends up steering the update, and progress slows. They flag the full analysis as future work; I would read it as a strong hint, not a proof.
That observation also tells you what to build. If guidance cancels because each state carries its own noisy per-sample direction, give every state of a prompt the same direction.
## The fix: aggregate first, then assign
DiffusionART tilts the rollout distribution toward reward, $p_\tau(x_0 \mid c) \propto p_{\text{old}}(x_0 \mid c)\,e^{\tau R}$, and asks which way that moves the mean endpoint. Lemma 2 is a standard exponential-family fact: at $\tau = 0$ the derivative of the tilted mean is the covariance between reward and endpoint. Under rectified flow, $u_t = x_0 - \epsilon$ with $\epsilon$ independent of $x_0$, so that covariance is also the reward-velocity covariance at every noise level. Estimate it from the prompt's $K$ samples and normalise (Equation 5):
$$
\overline{\text{Cov}}(R, x_0 \mid c) = \frac{1}{K}\sum_{i=1}^{K}(R_i - \bar R_c)(x_0^{(i)} - \bar x_{0,c}), \qquad G(c) = \frac{\overline{\text{Cov}}}{\text{RMS}(\overline{\text{Cov}})}.
$$
Then, for every state $s_t$ the ODE sampler visited while generating that prompt's samples, freeze the target
$$
v^{\text{ART}}_{\text{tgt}}(s_t) = v_{\text{old}}(s_t) + G(c)
$$
and regress. The cached $v_{\text{old}}$ was recorded during the rollout, so nothing is re-noised and nothing is recomputed. The target never mentions the forward velocity, so there is no $e$ for anything to cancel. The abstract's phrase "removing the consistency force by construction" means exactly this.
The code is as short as the equation. In `src/visual_rl/algorithms/covariance.py`, the per-prompt statistic:
```python
# src/visual_rl/algorithms/covariance.py:71, 81-85
centered = r - means[groups]
numerator = x.new_zeros(group_count, x.shape[1]).index_add_(0, groups, centered[:, None] * x)
denominator = (counts - int(reduction == "unbiased")).clamp_min(1)
covariance = numerator / denominator[:, None]
covariance = torch.where((counts > 1)[:, None], covariance, 0.0)
rms = covariance.square().mean(1).sqrt()
```
It multiplies centered rewards by raw endpoints rather than by centered endpoints, which gives the same result because centered rewards sum to zero. It divides by $K-1$ rather than the paper's $K$, which the RMS normalisation then erases. And in `art.py` the loss is plain MSE toward a detached target:
```python
# src/visual_rl/algorithms/art.py:91-94
target = (old - displacement).detach()
residual = target - v
desired_step = residual.detach()
unmasked = weight * residual.square().flatten(1).mean(1)
```
The minus sign is a convention, not a disagreement with the paper. The code's velocity points from data toward noise, the paper's from noise toward data. `weight` is the noise level $\sigma$, the paper's factor $t$ that emphasises high-noise states.
Playing with the widget makes one property obvious that the paper only states in passing. With $K = 2$, the centered rewards are $\pm(R_1 - R_2)/2$, so $G$ is just the winner's endpoint minus the loser's, rescaled to unit RMS. The size of the reward gap disappears in the normalisation. That means DiffusionART-Pairwise, which replaces scalar rewards with a 1 for the winner and 0 for the loser, produces exactly the same guidance as scalar DiffusionART when $K = 2$, except on ties. The pairwise $K = 2$ run in Table 1 is effectively a scalar $K = 2$ run that only had to ask which video was better. That makes the pairwise result less surprising, and it is still useful: a preference model is enough at that group size.
Compare this with [DiffusionOPSD](/articles/diffusionopsd), which also turns the reward into a regression target but needs the reward's gradient to walk the clean-image estimate uphill. DiffusionART needs only reward values. Its direction comes from the spread of samples the model already produced, which is why it works with a black-box video reward model.
## The second update and the mask
A single update toward $v_{\text{old}} + G$ learns fast and then decays. Figure 6(b) in the paper shows exactly that on OCR, labelled "ART: 1 update." So DiffusionART reuses the cheap trick from the NFT ablation: split the rollout into two disjoint minibatches and take one step on each, both toward the same frozen target. Before the second step, the policy has already moved by $d(s_t)$, so the gradient is proportional to $G(c) - d(s_t)$: the guidance plus a restoring term that grows with each state's drift. There is no EMA schedule to tune. In NFT the same trick cost 15.2x, because restoration overwhelmed scattered per-sample guidance. Here the guidance is shared across a prompt's states, so it survives aggregation and stays dominant.
The last piece is a PPO-style brake. ODE transitions have no likelihood, so the paper borrows Flow-GRPO's reference SDE step to define Gaussian transitions for the current and old velocity, and uses their per-dimension KL as a drift score (Equation 7). A state is dropped from the second update when that score passes $\delta = 10^{-5}$ **and** the move is outward, meaning the drift and the descent direction have a positive inner product. Corrections that pull back toward the anchor always pass:
```python
# src/visual_rl/algorithms/art.py:134-136
masked = (active & corrective_mask_enabled & (energy >= threshold) & (alignment > 0)).detach()
keep = (anchor | (active & ~masked)).detach()
policy = torch.where(keep, unmasked, torch.zeros_like(unmasked))
```
In the video recipe, the reference SDE noise level is 0.8 and the threshold is the paper's `1e-5` (`configs/examples/hunyuan_art.yaml`). The paper is candid that the mask is a safety net, not the engine. On SD3.5-M and FLUX, training without it stays effective and can be slightly faster (Appendix A.6). On HunyuanVideo, the unmasked run drops around step 200 and recovers, and over the first 300 steps the maximum first-update gradient norm is 1.96 with the mask against 3.33 without.
Put together, one iteration is: sample $K$ videos per prompt with the deterministic ODE solver, caching states and predictions; compute $G(c)$ per prompt; freeze targets; take two masked MSE steps. No transition likelihoods, no re-noising, no EMA, no differentiable reward.
## Where the 12.02x comes from
The post's claim comes from this figure.
The abstract's sentence is careful, and once you know what it refers to it checks out. The run is a separate one with visual quality and motion quality weighted equally and no text-alignment reward. The main text-to-video experiment uses text alignment, visual quality and motion quality weights of 7:2:1. All three scores come from VideoReward on 53-frame, 480x480 HunyuanVideo samples. "Late-stage reward" is each baseline's level at the end of its curve, the open circles. Appendix A.5 gives the mean training reward over each run's final 50 steps: 13.018 for DiffusionART, 10.105 for DiffusionNFT and 8.510 for Flow-GRPO-Fast. 13.018 over 10.105 is 1.288, and over 8.510 it is 1.530. Those are the 28.8% and 53.0% exactly.
The 12.02x and 11.72x themselves are not in any table. Neither crossing time is given as a number in the text, so I could only check them against the plot. Reading the GPU-hour panel, DiffusionART crosses NFT's final level at roughly 125 GPU-hours and NFT gets there at roughly 1,450; Fast's level is crossed at roughly 90 against roughly 1,050. Both come out near 11.6 to 11.7x, which is as close as reading pixels gets you. I believe the labels, but I can't reproduce them to two decimals.
Three things about this plot are worth knowing before you quote it.
First, it is training reward on the reward model being optimised. The held-out evaluation for this 1:1 run is not tabulated. The main run's evaluation is (below), and it tells the same story, but the headline ratios are training-curve ratios.
Second, the GPU-hour axis is wall-clock times GPU count, and the repository's pre-release README, `README.review.md`, which is still in the tree, adds to the plot caption: "GPU hours include evaluation and waiting time." The public README dropped that clause. Nothing in the release says how much of each run's hours went to evaluation, so treat the axis as wall-clock cost, not pure training compute.
Third, "training steps" means optimizer updates, and the methods do not take the same number of updates per batch of generated videos. The repo's `docs/paper-video.md` says it plainly: "ART/Fast/Dance make two disjoint updates per rollout (200 rollouts for U400); NFT/ReFL make one (400 rollouts)." Every Hunyuan rollout is 32 prompts times 8 videos, 256 videos, whatever the method. So DiffusionART at step 20 has generated 10 rollouts' worth of video, and DiffusionNFT at step 50 has generated 50.
That explains a reply under the post that noticed the oddity: against Flow-GRPO-Fast the plot says 22.1x fewer steps but 11.72x fewer GPU hours, and against DiffusionNFT it flips, 5.2x on steps and 12.02x on hours. Put the paper's own step and GPU-hour pairs on one table and divide.
Per rollout, DiffusionART and DiffusionNFT cost about the same: 4.4 to 5.3 GPU-hours for ART across the three milestones, 5.0 to 5.1 for NFT. Both train on nine denoising positions per video, so that is what you'd expect. Against NFT, then, the GPU-hour ratio is roughly the rollout ratio. At visual quality 6, that is 10 rollouts against 50 (the paper's 2.5x step ratio becomes 5x in rollouts, and 5.7x in GPU-hours). The step axis understates DiffusionART's advantage over NFT by about 2x.
Flow-GRPO-Fast goes the other way. It takes two updates per rollout like ART, but trains on two of the first six positions per video instead of nine, so each rollout costs 2.35 to 2.55 GPU-hours, about half of ART's. Its step ratio overstates the compute gap by nearly 2x. The 11.72x is the honest number and the 22.1x is not.
Neither correction rescues the baselines. At every milestone DiffusionART needs a fraction of the compute. The cleanest line in the paper is the 7:2:1 run reaching visual quality 5.5 in 63.4 GPU-hours against DiffusionNFT's 1,401.4: 11.4x fewer steps and 22.1x fewer GPU hours. The same 22.1 appears on Figure 1 for a different comparison, so be careful which one you quote.
One more provenance note. In the release's config inventory, the Flow-GRPO-Fast long run behind Figure 1 is marked unresolved: `configs/paper/video/cover_hunyuan_fast_11_long.yaml` is a "bounded 400-update reference," because "neither supplied inventory nor inspected source gives its paper-plotted endpoint." The plotted run went past 1,000 steps. So that curve cannot be re-run from the release as published.
## The main run, evaluated
The headline experiment is full-parameter fine-tuning of the CFG-distilled HunyuanVideo-13B, learning rate `1e-5`, no KL penalty, 400 updates, 16 sampling steps for rollouts, with text alignment, visual quality and motion quality weighted 7:2:1. Evaluation uses VideoReward at 4 FPS on held-out prompts with a 30-step sampler (Table 3):
| Method | VQ | MQ | TA |
|---|---|---|---|
| Pretrained | 5.195 | 1.943 | 1.118 |
| Flow-GRPO-Fast | 3.149 | 1.874 | 1.667 |
| DanceGRPO | 3.922 | 1.206 | 1.685 |
| DiffusionNFT (EMA) | 6.827 | 2.683 | 2.096 |
| DiffusionNFT (on-policy) | 5.543 | 1.839 | −4.480 |
| DiffusionART | 8.970 | 4.224 | 2.111 |
The 31.4% and 57.4% gains over "the strongest baseline" are against DiffusionNFT (EMA): 8.970 over 6.827 and 4.224 over 2.683, both check. Text alignment ends level, 2.111 against 2.096. Two rows stand out to me. Both GRPO variants finish with *lower* visual quality than the model they started from. With 70% of the reward on text alignment, they spent their budget there. And on-policy NFT drives text alignment to −4.480. The first half of the paper predicts exactly this collapse, and here it shows up in a production-scale model.
Image-to-video on the step-distilled Wan2.1-I2V-14B, with four sampling steps, is the most lopsided result. DiffusionART reaches motion quality 6 in 45.8 GPU-hours against Flow-GRPO-Fast's 220.3, and ends at 11.196 VQ and 10.476 MQ, against Fast's 5.771 and 4.710 (Table 4). NFT plateaus early; the authors attribute that to the residual consistency force. On images the gaps are smaller, which is useful calibration. On SD3.5-M DiffusionART matches tuned DiffusionNFT (EMA) without per-task tuning, and reaches Flow-GRPO's step-500 reward in 10.0x, 5.2x and 9.6x fewer steps on GenEval, OCR and PickScore. On FLUX.1-dev with NFT's SD3.5 hyperparameters carried over unchanged, GenEval lands at 0.9684 against 0.8490 for NFT and 0.9183 for Flow-DPPO (Table 2).
The group-size table is worth reading with a little care (the text calls it "Table 8"; it is Table 1). Pairwise $K = 2$ ends at VQ 12.235 in 260.8 GPU-hours, and pairwise $K = 8$ at 12.092 in 883.3, which is where the 3.4x saving comes from. But pairwise $K = 4$ lands lower than both, at 10.883, and scalar $K = 8$ beats every pairwise run at 13.637. With one run per cell, I read "insensitive to $K$" as "$K = 2$ is good enough here," not as a trend.
What I could not find anywhere: repeated seeds, error bars, or a human preference study. Every video number is VideoReward scoring HunyuanVideo or Wan outputs. Evaluation uses held-out prompts and a different sampler, which helps. It is still the same reward model family that was optimised, so a reader worried about reward hacking has the qualitative grids in Appendix D and nothing else.
## What is actually on Hugging Face
The collection has two models. [`vanzl/DiffusionART-SD35M-MultiReward`](https://huggingface.co/vanzl/DiffusionART-SD35M-MultiReward) is a rank-32 PEFT LoRA for SD3.5 Medium, from step 300 of the four-reward run, under Stability's community licence. Its `release.json` records GPU inference verified on an H800.
[`vanzl/DiffusionART-HunyuanVideo`](https://huggingface.co/vanzl/DiffusionART-HunyuanVideo) is full transformer weights. I summed the twelve shard headers: 1,264 tensors, 12,821,012,544 parameters, all F32, 51.3 GB. The card asks for roughly 100 GiB of GPU memory and disk for its verified inference path, and the weights are under the Tencent Hunyuan Community License.
Its `release.json` is unusually honest, and it says this is not the model in Table 3. The source run is a different one (`tk1w4go1`), with 7:2:1 weights, a 100-update target, and a `reference_mse_beta` of 0.1 toward the pretrained velocity. The paper's runs use no KL regularisation, so that is an extra regulariser. The run's recorded terminal state is `crashed_after_optimizer_step_40`, and the released checkpoint is optimizer step 30 (rollout 15). Its own evaluation on 100 validation prompts reads VQ 6.51, MQ 2.68, TA 1.68. A solid step up from the pretrained 5.195, 1.943 and 1.118, but well short of the 400-update 8.970, 4.224 and 2.111. The same file sets `independent_gpu_inference_verified` to false. If you download it to see the paper's results, you will see an early, regularised checkpoint. Weighting the reward toward visual quality early and adding a pull toward the base model is a reasonable thing to ship. It just isn't what the tables measured.
The code is in better shape than most paper releases. It has one framework with DiffusionART and six baselines across SD3.5-M, FLUX.1-dev, HunyuanVideo and Wan2.1, pinned environments, about a hundred test files, and per-run provenance that admits what is missing. `docs/paper-video.md` describes its YAMLs as "training-semantic recipes, not bitwise trajectory reproductions," and lists which runtime commits could not be recovered. Two practical caveats. The video recipes assume 4 nodes of 8 GPUs, and `THIRD_PARTY_NOTICES.md` says "a project-wide license has not yet been chosen," so the code has no licence of its own yet, beyond the MIT and Apache-2.0 notices for the Flow-GRPO, DiffusionNFT and DanceGRPO code it derives from.
## What I'd take from it
The durable contribution is Section 3. "NFT needs EMA" was folklore with a tuning table attached. This paper gives a mechanism: a self-consistency force that only cancels on average, and a restoring term that EMA happens to switch on. It backs both with ablations. Anyone running forward-process diffusion RL should read those two propositions even if they never touch DiffusionART.
The method is small enough to port. If you already have an NFT-style trainer that caches rollout states, the change is: replace the re-noised per-sample target with $v_{\text{old}} + G(c)$ on the visited states, split the batch into two optimizer steps, and add the outward-drift mask. The paper and the 1:1 run make a strong case that this is both faster and steadier than NFT and the GRPO variants on large video models, and on images it at least removes the per-task EMA tuning.
The 12.02x is real. I would quote it with its context: one VQ:MQ 1:1 run, training reward, GPU hours that include evaluation and waiting, and baselines that were run once. Against Flow-GRPO-Fast, quote GPU hours, never steps. And if you want the model from the tables, it is not on Hugging Face yet.
## How I checked
I read the arXiv HTML and the LaTeX source (`main.tex`, Appendix A and B), rendered the figures from the source PDFs, and recomputed every ratio the text gives: 273/24, 1401.4/63.4, 50/20, 427/20, 250.2/44.2, 502.5/44.2, 33/19, 117/19, 165.6/42.9, 149.4/42.9, 320/21, 883.3/260.8, the final-50-step means behind 28.8% and 53.0%, and Table 3's 31.4% and 57.4%. All match to the stated precision. The 12.02x and 11.72x have no underlying numbers in the text, so I read the crossings off Figure 1 and got about 11.6 to 11.7x for both. Per-rollout costs use the updates-per-rollout counts from `docs/paper-video.md` in the repo at commit `016610f`, which I cloned and read without running: `covariance.py`, `art.py`, `nft.py`, `training/objectives.py`, and the paper configs in `configs/paper/video/`. The "evaluation and waiting time" clause is from `README.review.md` in the same commit. Model facts are from each Hugging Face repo's `release.json` and card, plus the safetensors headers of all twelve HunyuanVideo shards, read by HTTP range request. I did not train or run anything, so the curves themselves are the paper's. The DiffusionART-Pairwise equivalence at $K = 2$ is my own algebra on `art_pairwise_covariance`, not a claim the paper makes.
---
# GroundingPI: a 4.84B grounding model that answers in 1,000 coordinate tokens
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/groundingpi
> date: 2026-10-13
> tags: vision-language-models, object-detection, robotics, benchmarks, licensing, vla
[GroundingPI](https://huggingface.co/GroundingPI/GroundingPI) is a grounding model: give it an image and a phrase, and it answers with where things are, as boxes or as points. The [paper](https://arxiv.org/abs/2609.39601) (arXiv 2609.39601, "GroundingPI: A Grounding Foundation Model towards Physical Intelligence with Visual Primitives", 30 September 2026) makes two claims. First, that this 4B model beats 44 baselines on average across 34 grounding benchmarks, GPT-6 Astra included. Second, that plugging it into a robot policy or a driving planner as the visual backbone beats the usual vision-language and video backbones.
The repost that brought it to me called it "Released by XPeng's AI team". The paper says something narrower; more below. I did not download the weights. Everything here comes from the config, the processor and modelling code, the chat template, the token table, the [release repository](https://github.com/groundingpi/GroundingPI) at commit `0047d45`, HTTP range reads of the two safetensors headers, and the paper's own tables. Labels: **measured** means I computed it from a file; **reported** means it is the authors' number, not re-run; **reasoned** means my arithmetic on the other two.
## What is in the checkpoint
`config.json` names the parts directly. The language model's `text_config` carries `"_name_or_path": "Qwen3-4B-Instruct-2507"`: 36 layers, hidden width 2560, 32 query heads over 8 KV heads. The vision side is `MoonViT3dPretrainedModel`, from `modeling_groundingpi_vision.py`, and the image processor's docstring still reads "Image processor class for Kimi-K3". Between them sits a projector the code calls "the freshly initialized GroundingPI projection connector": a LayerNorm, a 2×2 merge of neighbouring patch features into one 4096-wide vector, then two bias-free linear layers (4096 to 4096 to 2560) with a GELU and an RMSNorm.
I read both shard headers with range requests and summed every tensor:
| Part | Parameters | Share |
|---|---:|---:|
| MoonViT-V2 vision tower (27 layers, width 1024) | 401,214,464 | 8.3% |
| Projector | 27,267,584 | 0.6% |
| Qwen3 decoder body (36 layers) | 3,633,511,936 | 75.0% |
| Input embedding (152,670 × 2560) | 390,835,200 | 8.1% |
| Output head, untied (152,670 × 2560) | 390,835,200 | 8.1% |
| **Total** | **4,843,664,384** | |
All of it is BF16, 9,687,328,768 bytes (**measured**). The total matches the Hub API and the paper's Table 9 to the parameter. So "4B" is the language-backbone family, which the paper's appendix says itself. The checkpoint is 4.84B. Of the 0.8B gap, 390.8M is an output head that Qwen3-4B-Instruct-2507 does not have: upstream ties its embeddings (`tie_word_embeddings: true`, vocabulary 151,936), and GroundingPI unties them (**measured**, from both configs).
Is the vision tower really Kimi K3's? Kimi K3's own `config.json` describes a vision tower with the same 27 layers, width 1024, FFN 4096, 12 heads and QKV width 1536. I pulled four small norm vectors from each checkpoint by byte range. They have identical shapes and are close but not identical: the mean absolute difference is 0.1% to 0.8% of the weights' mean magnitude (**measured**). That fits the recipe: K3's encoder, frozen in Stage 1, then trained at a learning rate of 1e-6 (**reported**). The same probe against Qwen3-4B-Instruct-2507's norms shows bigger moves, 0.8% to 8%, consistent with a decoder trained at 1e-5 (**measured**). The site's [Kimi K3 article](/articles/kimi-k3) covers where MoonViT-V2 came from.
## Points and boxes as vocabulary
The interesting part is the output. `added_tokens.json` holds 1,027 added entries. 1,000 of them are the strings `<0>` through `<999>`, at token ids 151669 to 152668, followed by `` at 152669 (**measured**). Each coordinate is one atomic token with its own embedding row and its own output logit. The model does not spell "4", "2", "1" in digits. It emits one token that means 421.
The rest of the grammar reuses Qwen's existing control tokens. From `grounding_pi/protocol.py` in the release:
```text
answer := entry (", " entry)*
entry := <|object_ref_start|> phrase <|object_ref_end|> <|box_start|> payload <|box_end|>
payload := "None" | tuple ("," tuple)*
tuple := coord coord # point (x, y)
| coord coord coord coord # box (x1, y1, x2, y2)
coord := "<0>" | ... | "<999>"
```
Three conventions do a lot of work:
- **Instances of one label share a wrapper.** Two mugs are one entry with two tuples, joined by a bare comma. Boxes are sorted by x1 and points by x, so the order is deterministic and teacher forcing has one right answer.
- **Absence is an answer.** A queried category that is not in the image gets its wrapper with the literal text `None` inside. The data engine treats "absent" as a fact it must verify before a sample is accepted, not as the default when a teacher returned nothing.
- **Queries use ``.** Categories in the prompt are joined by that token, so a phrase can contain an ordinary comma.
The coordinate mapping is in the paper's Section 7.1. A coordinate sits on an intermediate integer grid v from 0 to 1000, normalised by the image's width or height. It is mapped onto the 1,000-entry vocabulary by
$$
q(v) = \left\lfloor \frac{999\,v + 500}{1000} \right\rfloor
$$
and decoded back as $x = q/999 \cdot W$. The paper does not say how pixels reach the 0..1000 grid. Rounding to nearest is the obvious reading, and it is what the widget below does.
Here is a worked example on a 1280×720 frame. The left mug's box is (180, 430, 300, 590) in pixels. The x1 normalises to 1000 × 180 / 1280 = 140.6, which rounds to v = 141 and maps to q = 141. The y1 gives 1000 × 430 / 720 = 597.2, so v = 597, and q = floor(596,903 / 1000) = 596. Then x2 = 300 gives `<234>` and y2 = 590 gives `<818>`. With a second mug at (860, 450, 960, 580) and an absent car, the query `Locate all the instances that match the following categories: mugcar.` should produce:
```text
<|object_ref_start|>mug<|object_ref_end|><|box_start|><141><596><234><818>,<671><624><749><805><|box_end|>, <|object_ref_start|>car<|object_ref_end|><|box_start|>None<|box_end|><|im_end|>
```
That is two boxes and a negative in 8 coordinate tokens and 9 protocol tokens, plus the two labels' own text tokens. Decoding `<141>` gives 141 / 999 × 1280 = 180.7 px, which is 0.7 px off the original (**reasoned**, by hand from the paper's formula).
The widget runs both directions: build a query and read the string, or edit the string and watch the outlines follow. It flags what the release's parser rejects: a coordinate above `<999>`, a tuple of three, an inverted box, a space between coordinates, a missing `<|im_end|>`.
### What the grid costs
Two consequences are worth stating with numbers.
**The grid is a resolution limit.** One step is W/999 of the image: 1.28 px at 1280 wide, 3.84 px at 3840. The two-stage rounding (pixels to 0..1000, then 0..1000 to 0..999) makes it slightly worse than a single rounding would. I swept every tenth of a pixel across widths of 640, 1000, 1920 and 3840, and the worst round-trip error came out at 0.639, 1.001, 1.917 and 3.834 px. That is very nearly a full grid step, where a single rounding straight onto 0..999 would cap it at half a step (**measured**). The intermediate grid also has 1,001 values feeding 1,000 tokens, so exactly one collision exists: v = 500 and v = 501 both become `<500>` (**measured**). This affects the training labels, not the model, and at the encoder's own resolution it is sub-pixel. But it is a design choice with a measurable cost, and the paper does not discuss it.
**The encoder sees roughly 1,170 px, whatever you send.** The processor caps input at 4,096 patches of 14 px, and the 2×2 merge turns that into at most 1,024 visual tokens. Running the processor's resize logic on its own (no weights involved), a 1920×1080 frame and a 3840×2160 frame both arrive at the encoder as 1172×659, padded to a grid of 1,008 visual tokens. A 1280×720 frame arrives at 1173×660 (**measured**). The 1,000-step output grid is therefore matched to what the encoder actually sees: about 1.17 resized pixels per step, or 24 steps per merged visual token (**reasoned**). The flip side is tiny objects. A 20-px object in a 4K frame is 6 px wide by the time the encoder sees it (**reasoned**). VisDrone, the paper's tiny-object aerial set, is the detection column where GroundingPI scores lowest (40.44 F1mIoU) and where SenseNova-Vision (42.35) and Qwen3.7-Max (41.59) beat it (**reported**).
The paper's own ablation is the reason to pay this cost. Atomic coordinates against coordinates written as text: grounding average 73.68 against 71.21, and text coordinates run at 0.25× the speed (**reported**, Figure 10). Output length is the other half. Table 5 puts GroundingPI at 7.6 tokens per box on COCO and 5.1 on Dense200, against 148.8 and 74.5 for SEED1.5-VL (**reported**). That is roughly 19.6× and 14.6× fewer, and the paper itself calls these serialisation ratios, not measured speedups.
## How it was trained
Three phases, all in the appendix with hyperparameters; none re-runnable, because the training data is not published.
1. **Base VLM training (Stages 1-3).** First the projector alone on image captions, with both backbones frozen. Then everything, on text plus image-text data, at learning rates of 1e-5 for the language model and 1e-6 for the encoder. Then general visual and video QA at a 32,768-token context. 128 GPUs, BF16, ZeRO-1 (**reported**).
2. **Coordinate alignment, called SFT (Stage 4).** Eight spatial task groups: detection, GUI, referring boxes, referring points, OCR, layout, dense pointing and counting, and visual prompting. Coordinate tokens get the same cross-entropy as every other token. There is no IoU regression loss (**reported**).
3. **GRPO.** 8 responses per prompt, 256 GPUs, only language parameters trained, learning rate 5e-7. For grounding, the advantage is 0.7 × a coverage reward plus 0.3 × a strict reward. The strict reward combines F1 at IoU 0.50, 0.75 and 0.95, matched IoU, count, format and ordering, with penalties for oversized and duplicate boxes. OCR gets a Hungarian-matched text-plus-geometry reward (**reported**). Removing this stage costs 0.82 points of grounding average (**reported**).
Total exposure is 500.41B plus 221.17B tokens (**reported**). Labels come from a "data engine" that fuses several teachers' outputs field by field, validates them per task, and iterates through a grounding expert. No data, per-source counts or teacher list is released, which is the largest reproducibility gap.
## The 34-benchmark claim, recomputed
The abstract says GroundingPI beats 44 baselines across 34 benchmarks spanning 11 capabilities, averaging 73.68% against GPT-6 Astra's 71.54%. The appendix defines that number precisely: "The headline Avg is the mean of 11 independently computed capability scores." It is not a mean over 34 benchmarks.
Each capability score is itself an unweighted mean of table columns. I rebuilt all eleven for both models from the per-benchmark tables. Visual prompting, for example, is the mean of the FSC147, Dense200, COCO and LVIS F1mIoU columns in Tables 36-39: 73.59 for GroundingPI and 65.84 for Astra. Two need a guess. Referring is the mean of seven columns (HumanRef, RefCOCO, RefCOCO+, RefCOCOg, RefCOCOg val, RefCOCOg test, and the "RefCOCO avg" column, which is itself a mean of three of those). Robo Point averages five columns, again including an average column. Only those compositions reproduce the published 84.89 and 75.12. The result: 73.68 for GroundingPI and 71.54 for Astra, both exact to two decimals (**measured**, from the reported tables).
What the average hides:
- **It is 8 wins out of 11 against Astra, not a sweep.** Astra leads GUI by 13.67 points (92.58 against 78.92), Robo Point by 5.96 and Dense Point by 1.72 (**reasoned**). On ScreenSpot-Pro alone the gap is 93.17 against 65.78 (**reported**). If you want a GUI-clicking model, the other side of this comparison is the one to read: see [Holo4](/articles/holo4), [Fara 1.5](/articles/fara-1-5) and [LFM2.5-VL-3B](/articles/lfm2-5-vl-3b).
- **Against everyone, it is 16 of 27 columns.** Across the main-text Tables 1-4, GroundingPI has the best score in 16 of 27 benchmark columns (**reasoned**, by counting). It loses LVIS (56.02 against SenseNova-Vision's 56.12), VisDrone, HierText (41.70 against Qwen3.7-Max's 42.95), TotalText and DocLayNet (85.08 against 85.53). To Astra it also loses Dense200 pointing and five of the six robot-spatial and GUI columns. Its wins are large where they land: Dense200 boxes at 74.53 against Astra's 65.04, and Qwen3-VL-4B's 14.02 (**reported**).
- **"34 benchmarks" counts task-dataset pairs.** COCO appears as box grounding, as pointing and as visual prompting. The suite uses 20 distinct datasets (**reasoned**, from the benchmark list in Section 12).
- **The protocol is uneven by necessity.** GroundingPI's numbers are the mean of ten runs; locally evaluated baselines get three; starred rows are copied from other papers. Kimi K3 is marked N/A on pointing, OCR and layout because its outputs did not fit the parser. The paper says plainly that this "does not establish a lack" of the capability (**reported**).
None of this makes the result wrong. On RefCOCO, RefCOCOg and RefCOCO+ it scores 85.79, 81.96 and 84.01 F1mIoU against Astra's 79.45, 76.15 and 77.84 (**reported**); a 4.84B model doing that is a real result. It is an average claim, and the appendix is honest about it.
## The physical-intelligence claim
This is the half of the title that "PI" refers to, and it is the half the release does not let you check.
The manipulation setup is careful. Every backbone, GroundingPI and eight others, is run once per observation. Eight hidden states are sampled at normalised depths, projected to width 1024, resampled to 64 tokens each, and fed into the cross-attention blocks of one fixed 16-block flow-matching Action DiT. Same data, same 40,000 updates, same optimiser (**reported**, Tables 14-15). The only variable is the backbone.
Reading the bars rather than the summary (all **reported**, differences **reasoned**):
| Setting | GroundingPI | Best other backbone | Gap |
|---|---:|---:|---:|
| RoboTwin 2.0, in-distribution | 66.20 | 60.62 (LocateAnything) | +5.58 |
| RoboTwin 2.0, Clean2Random | 17.60 | 14.10 (Rex-Omni) | +3.50 |
| RoboCasa-GR1, in-distribution | 37.75 | 39.00 (RynnBrain) | −1.25 |
| RoboCasa-GR1, unseen object types | 28.94 | 28.06 (RynnBrain) | +0.88 |
| RoboCasa-GR1, unseen appearance | 37.78 | 36.93 (RynnBrain) | +0.85 |
| RoboCasa-GR1, unseen combinations | 37.86 | 37.42 (RynnBrain) | +0.44 |
The abstract's "by up to 24.8% relative" is the Clean2Random row: 17.60 / 14.10 = 1.248, which is 3.5 points absolute on a split where every model fails more than four times in five. Three of the four "wins in every OOD setting" are under one point. The paper does not report seeds or variance for these runs. Its own scaling appendix calls four settings "without repeated-seed uncertainty estimates" too few to establish a trend. The data-efficiency claim has the same shape: 28.75% with half the demonstrations against RynnBrain's 27.75% with three quarters (**reported**), a one-point margin.
Driving is a smaller claim than it sounds. GroundingPI predicts six waypoints over 3 s, serialised through the same coordinate tokens, from the front camera plus up to seven past ego states that include velocity, acceleration and steering. It reaches an average open-loop L2 of 0.296 m, against 0.301 for Qwen3-VL-4B and 0.308 for RynnBrain (**reported**). That is 5 mm better than the runner-up, on nuScenes open-loop L2, the metric where ego history alone is known to carry much of the signal: a shortcut discussed in the [Qwen-Drive-1.0](/articles/qwen-drive-1-0) write-up. No ego-state ablation is reported, so how much of 0.296 m the image contributes is not measured. The authors' own wording is "modest", and they say it does not establish closed-loop performance.
Then the release. The `vla/` directory is a trimmed copy of OpenWAM for RoboTwin and StarVLA for RoboCasa. Its backbone switch lists `wan22_ti2v_5b | cosmos_predict25 | qwen3_vl_4b | paligemma_3b | rex_omni_3b | locate_anything_3b | rynnbrain_2b`, and the RoboCasa launcher's `case` has the same seven. Neither has a GroundingPI entry, and no file under either code tree names it outside READMEs and licences (**measured**, by grep). The documented RoboTwin protocol is "global batch 256, 10 epochs", while the paper's Table 15 says 1280 global and 40,000 updates; the RoboCasa launcher does default to 40,000 steps at full data. There is no nuScenes code at all, and no action-policy checkpoint (**measured**). The baselines are reproducible from the release; the row they are compared against is not. The model card says as much in one sentence: "downstream action-policy results are described in the paper."
The broader argument, a grounding model as the "System 1" backbone, is labelled in the paper as a hypothesis it has not tested. [Flex-π](/articles/flex-pi) takes the opposite route, predicting geometry and semantics jointly with actions.
## XPeng, and what "released by" means
XPeng Inc. is the first of eight affiliations, with Peking University, HKU, Berkeley, Princeton, NUS, Tsinghua and HKUST (GZ). The paper's disclaimer is unusually explicit: the work "originated from exploratory academic research undertaken by the project leader Qize Yu during their internship at Xpeng Inc.", "does not involve commercial applications, product development, or commercial deployment", and does "not represent the official positions, technical direction, product roadmap" of the company (**reported**). XPeng supplied compute and data. Reading it as a signal about XPeng's cars or robots goes further than the authors do.
## The licence is two licences
The Hub tags the model `license: other`, named `apache-2.0-with-upstream-kimi-k3`. The `LICENSE` file explains the split. The authors' own contributions are Apache-2.0. "Kimi-derived material remains subject to the Kimi K3 License", whose conditions "continue to apply to the combined model package where applicable." The Kimi-derived material is the vision tower and its code, 8.3% of the parameters (**measured**), and the code notices in the repo say the same. Qwen3-4B-Instruct-2507 is Apache-2.0 upstream.
The Kimi K3 terms are permissive with two conditions, both scoped to large commercial use:
- **Model as a Service.** If you, with your affiliates, run a business that gives third parties API-style "language model inference or fine-tuning" and have more than US\$20M in revenue over 12 months, you need a separate agreement with Moonshot before commercial use.
- **Attribution.** A commercial product with more than 100M monthly active users or more than US\$20M in monthly revenue must display "Kimi K3" prominently in its UI.
- **Exemptions.** Internal use that exposes nothing to third parties is exempt from both.
Whether a hosted grounding API counts as "language model inference" the text does not settle. Below those thresholds it behaves like a permissive licence with a notice to keep.
## Running it
The weights are 9.69 GB in BF16, so a 24 GB consumer card holds them with room for the 16,384-token serving context (**reasoned**). The documented path is narrower: the repo's `run.py` builds a bundled Transformers 5.7.0 fork and serves through vLLM 0.18.x in eager mode, one image per request, tested on B300, B200, H200, H800 and PPU. If you write your own client, decode with `skip_special_tokens=False` or every coordinate disappears, and never insert spaces between coordinate tokens.
One small inconsistency for client authors. The paper decodes with q / 999 × W. The release's `visualize()` uses q / 999 × (W − 1), so the two disagree by up to one pixel at the right and bottom edges (**measured**, from `client.py`). Pick one and be consistent with whatever produced your ground truth.
The Hub repo also carries a stray `streammind_gate.py`, a Mamba-based gate nothing else references (**measured**). Harmless.
## What holds
- **The architecture matches the card to the parameter**: a 4.84B checkpoint whose vision tower is Kimi K3's, lightly retrained (measured).
- **The protocol is clean**: one token per coordinate, deterministic order, explicit negatives, a reference parser. Its cost is a 1,000-step grid, and two-stage label rounding nearly doubles worst-case error (measured).
- **73.68 against 71.54 reproduces exactly**, as a mean of 11 capabilities, 8 of them won, with a 13.67-point GUI deficit inside (measured, reasoned).
- **The robot and driving results are small and not reproducible from the release**: under a point on three of four OOD splits, a 5 mm L2 margin with no ego-state ablation, no variance, and no GroundingPI backbone in the released VLA code (reported, measured).
As a grounding model that writes a box in about 8 tokens and holds its own against a frontier model on dense detection and referring, it is worth having. As evidence that robot policies need grounding pretraining, it is a promising first experiment nobody outside can re-run yet.
---
# MiniCPM-V 4.7: Qwen3.6's own brain, new eyes, and experts read off a phone's flash
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/minicpm-v-4-7
> date: 2026-10-13
> tags: multimodal, vision-language-models, mixture-of-experts, on-device, quantization, speculative-decoding
The post that sent me here was a reply to OpenBMB's launch: "We got a Qwen3.6-35B
replacement." OpenBMB's own thread, posted on 12 October, is more careful. It
announces MiniCPM-V 4.7 as an open multimodal series in two sizes, a 35B-A3B
mixture-of-experts and a 1B, and leads with a phone: 13+ tokens per second on an
iPhone 17 Pro Max, 17+ on an iPad Pro M4 with 16 GB, "while costing only 3GB
memory", thanks to a scheduling trick called LES that gives a "5× speedup ... with
byte-for-byte identical outputs".
A 35B model on a phone is the part I cared about. Before reading any benchmark I
opened `config.json`, because "replacement" is a strong word for a model whose
acknowledgements thank Qwen3.5 and Qwen3.6 "for providing the language backbone".
## The replacement is the original
I diffed the language half of the 35B config against
`Qwen/Qwen3.6-35B-A3B/config.json`. Every field matches: 40 layers, hidden size
2048, the 3:1 pattern of Gated DeltaNet linear-attention layers to full-attention
layers, 256 routed experts with 8 picked per token, one shared expert, 512-wide
expert MLPs, the same M-RoPE sections, the same rope theta of 10,000,000. The only
difference is the vocabulary, trimmed from 248,320 rows to 248,144. The
`model_type` inside `text_config` is literally `qwen3_5_moe_text`.
The 1B is the same story one size down. Its text config matches `Qwen3.5-0.8B`
field for field, except that MiniCPM unties the output head from the embedding
table, which Qwen ties. That one change adds a 254M-parameter matrix, and it
matters for the naming below.
So "Qwen3.6-35B replacement" is true in the least surprising way possible: it is
Qwen3.6-35B-A3B's language model with a different vision stack bolted on and
OpenBMB's own multimodal post-training. I don't mean that as a criticism. It tells you
where to look for whatever is new, and it is not the transformer. It is the eyes
and the runtime.
### Counting the weights
I read the safetensors headers of both repos with HTTP range requests (eight
bytes for the header length, then the JSON header, never the weights) and summed
the shapes. All tensors are BF16.
| | MiniCPM-V 4.7 35B-A3B | MiniCPM-V 4.7 1B |
|---|---:|---:|
| total parameters | 35,212,875,824 | 1,554,476,272 |
| routed experts | 32,212,254,720 (91.48%) | none, dense |
| vision tower + projector | 552,986,032 | 548,266,416 |
| token embedding | 508,200,912 | 254,048,256 |
| output head (untied) | 508,200,912 | 254,048,256 |
| active per token, excluding embedding and vision | 2.95B | dense |
The 35B name is honest: 35.21B on disk. So is A3B. Per token the model touches
attention (1.01B in the DeltaNet layers, 0.27B in the full-attention ones), the
shared experts, the routers, 8 of 256 routed experts in each of 40 layers and the
output head, which comes to 2.95B. Each routed expert is a gate, up and down
projection of 2048 by 512, so 3,145,728 parameters, and each layer carries 256 of
them. The config also declares one multi-token-prediction layer, but there are no
MTP tensors in the checkpoint; OpenBMB ships a separate DSpark drafter instead.
The 1B is a different matter. It is 1.55B parameters. OpenBMB's own benchmark
chart labels it "1.3B", which is exactly the total minus the untied output head
(1,300,428,016). Call it whatever you like, but note what the table above shows:
the vision tower is 548M of those parameters, more than a third of the model. A
dense 0.8B language model wearing a 0.55B encoder is a very different animal from
Qwen3.5-0.8B, whose own vision tower is 12 layers of width 768, and the 1B's
headline comparison is against exactly that model.
## The eyes: a SigLIP2 that compresses itself halfway through
The vision config is SigLIP2-shaped: 27 layers, width 1152, 14-pixel patches,
position embeddings for a 70 by 70 grid. Two fields are not SigLIP: `insert_layer_id: 6`
and `downsample_mode: "16x"`. The model card credits "the intra-ViT early
compression technique from LLaVA-UHD v4" and says 4.7 supports "mixed 4x/16x
visual token compression". I wanted to know what is actually mixed, so I read the
modeling code. 4.7 inherits its vision path unchanged from MiniCPM-V 4.6 in
`transformers`:
```python
# transformers/src/transformers/models/minicpmv4_6/modular_minicpmv4_6.py:418-423
insert_layer_id = self.config.insert_layer_id if use_vit_merger else -1
if use_vit_merger and insert_layer_id >= 0:
for layer_index, encoder_layer in enumerate(self.encoder.layers):
hidden_states = encoder_layer(hidden_states, **attn_kwargs)
if layer_index == insert_layer_id:
hidden_states = self.vit_merger(hidden_states, target_sizes, **kwargs)
```
and a few lines further down, the switch:
```python
# modular_minicpmv4_6.py:559
use_vit_merger = downsample_mode != "4x"
```
So the image goes through seven SigLIP layers (indices 0 to 6) at full
resolution. Then `vit_merger` runs: a self-attention restricted to each 2 by 2
window of patches, after which the four patch vectors are concatenated (4,608
wide), pushed through an MLP and added to their mean. One vector per window
leaves. The remaining 20 encoder layers run on a quarter of the tokens. At the
end a second 2 by 2 merge, the projector, folds four more into one and maps them
to the language model's width of 2048. Sixteen patches in, one token out.
The 4x mode simply skips the first merger. All 27 layers run at full resolution
and only the projector merges, so four patches become one token. You can see this
in the GGUF repo too: `mmproj-model-f16-4x.gguf` is 209,084,192 bytes smaller than
`mmproj-model-f16.gguf`, which is the 104.5M parameters of `vit_merger` at two
bytes each. The 4x file just does not carry it.
Two details in that merger are worth a sentence. Its first MLP layer is
initialised block-diagonally, so at the start of training each of the four
patches in a window passes through its own slice and nothing is mixed yet; mixing
is learned. And the residual is the mean of the window, so the merger starts from
a plain 2 by 2 average and learns corrections to it. Placing it after layer 6 rather than at the end is the
whole efficiency argument: the last 20 of 27 layers are where most of the encoder
compute is, and they now run on a quarter of the sequence.
Then there is slicing, which decides how many patches you have in the first
place. The processor targets 448 by 448 pixels per crop. A large image is cut
into a grid of up to `max_slice_nums` slices (9 by default, the card recommends 36
for images), picked to match the aspect ratio, plus one downscaled overview of the
whole image. Every crop is snapped to multiples of 56 pixels so that both 2 by 2
merges divide evenly.
I ported that processor into the widget below, rounding rules included, and
checked it against a Python run of the same functions.
The default case is the photo from OpenBMB's own before-and-after clip, 1227 by
920. It becomes a 2 by 3 grid plus an overview, seven crops, 7,152 patches, and
447 tokens at 16x. That number is a nice cross-check: a comment in the MiniCPM-V
iOS app's source says a 9-slice image costs "about 447 tokens". At 4x the same
photo is 1,788 tokens.
On the encoder side, 16x does seven layers of full work and twenty at a quarter,
so it costs 12/27 of the 4x mode's patch-layers, 44.4%, before counting that
attention inside each crop is quadratic in its length. On the language side it
sends a quarter of the tokens. For a model whose language half is a 3B-active
MoE, that second saving is what decides prefill time on a phone.
### What 16x costs in accuracy
OpenBMB publishes both modes side by side, which I appreciate. Averaged over the
24 non-MMHal rows of its vision-language table, 16x gives up 1.55 points. It is
not uniform. The rows that lose the most are the ones that read small things:
CharXiv reasoning drops 6.4 (75.60 to 69.20), TextVQA 3.27, CCOCR 2.85, OCRBench_v2
2.8. General VQA barely moves (MMBench 0.85, MMMU 0.22). Merging pixels early
should hurt exactly these rows, and the split is useful: use 16x for photos and
scenes, 4x for charts, forms and dense text.
## Did the new eyes buy anything?
OpenBMB published this table for the 35B in instruct mode. Every number in
it is theirs: the footnote says "Results are from public reports and our local
evaluation using official checkpoints." I have not re-run any of it.
The thread says "34 benchmarks". The vision table has 26 rows and the language
table below has 8, so the count is right. The column to read is Qwen3.6 35B-A3B,
because it is the same language model with Qwen's own vision encoder and
post-training. Any difference between the two is the vision stack and OpenBMB's
data, which makes this the cleanest comparison in the release.
At 4x, MiniCPM is ahead on 19 of 26, behind on 6, tied on one (MathVision, 74.24
each). At 16x, it is ahead on 16 and behind on 10. The wins are concentrated where
you would hope a vision team would win. OmniDocBench is the outlier at +14.07
(94.24 against 80.17), then CharXiv reasoning at +8.1, LongVideoBench +5.8,
MMSI-Bench +5.3, OCRBench +5.2. The losses are mostly small, with one exception:
ScreenSpot, GUI grounding, where MiniCPM trails by 5.95 at 4x and 7.93 at 16x. If
you are building a computer-use agent, the "replacement" does not replace.
### One claim in the thread does not add up
The launch post says: "Even at 16× compression, 35B-A3B scores 90.4 on OCRBench,
5.2 points ahead of Qwen3.6 35B-A3B." The chart has 90.40 at 16x and 86.00 for
Qwen3.6 35B-A3B. The gap is 4.4 points. The 5.2-point gap is the 4x score, 91.20,
minus 86.00. The thread put the 16x score next to the 4x margin. It is a small
slip, and 16x still wins, but it is exactly the kind of number that gets copied
into a hundred reposts.
### The frontier chart, and what it leaves out
The footnote defines the score as the average of five benchmarks. I recomputed it
from the table: MiniCPM at 4x averages 76.86, Qwen3.6 35B-A3B in instruct mode
72.94, which matches where the star and the circle sit. So the star is the 4x
model; at 16x the same average is 74.48. Two things the chart does not tell you.
First, the hardware behind "inference time" is not stated anywhere I could find,
so the x-axis is a relative measure from OpenBMB's own setup. Second, look at the
diamonds: Qwen3.6 35B-A3B in thinking mode scores higher than MiniCPM's star, at
about four times the time. "Higher score in less time than Qwen3.6 35B-A3B" is
true of instruct against instruct. Against thinking mode it is a speed trade,
which may well be the trade you want, but it is a trade.
The language table is the eight benchmarks that make up the rest of the 34.
Against its own base it wins five of eight: CMMLU, IFBench (47.67 against 35.33),
AIME26, LiveCodeBench and tau-squared Bench (84.65 against 79.65). It loses
MMLU-Pro, SuperGPQA and BFCL-V4 by under 2.2 points each. Multimodal post-training
usually costs a language model some text ability; here the text side mostly held
or improved. Gemma-4 31B still beats everyone on LiveCodeBench (71.24) and IFBench
(54.33).
### The 1B, against the model it is built on
The card says the 1B "substantially outperforms Qwen3.5 0.8B", and on this table it
does: ahead on all 25 rows in both modes. Given that it is Qwen3.5-0.8B's own
language model plus a 548M-parameter encoder, that is the expected result, and the
comparison flatters it a little: it is a 1.55B model against a 0.8B one. The more
honest yardstick is MiniCPM-V 4.6 at the same size, which the card says 4.7
"continues to improve" on. At 4x it is ahead on 17 of 25 and behind on 7,
including MathVista (74.30 against 75.50), OmniDocBench (85.39 against 86.93) and
RefCOCO (84.85 against 86.70). An improvement on balance, not across the board.
## The runtime: 13.8 GB of experts, read from flash
Now the part that is new. A 35B MoE in BF16 is 70 GB. Even at 3 bits the routed
experts alone do not fit in a phone's memory. OpenBMB's answer, which they call
Lookahead Expert Scheduling, lives in a fork of llama.cpp
(`tc-mb/llama.cpp-omni`) and in the iOS demo app (`OpenBMB/MiniCPM-V-Apps`). I
read both. The app source turned out to be the better documentation: it is full
of measured tables in its comments.
First, the bytes. The app downloads a GGUF called
`MiniCPM-V-4_7-35B-A3B-mix_q3gud_im-les.gguf` (1,662,732,800 bytes: everything
except the routed experts) plus 40 files `experts/layer_000.bin` to
`layer_039.bin`, each exactly 346,030,080 bytes
(`MiniCPMModelConst.swift:369`). Divide by 256 experts and you get 1,351,680
bytes per expert, which is 3,145,728 parameters at precisely 3.4375 bits: the
llama.cpp Q3_K format. The public GGUF repo's `tensor-types.txt` for that
quantization says the same thing in one line, `ffn_(gate|up|down)_exps=Q3_K`. All
40 layers together are 13.84 GB of experts.
Per token, 8 experts in each of 40 layers is 320 expert reads, about 432 MB if
none of them is already in memory. At OpenBMB's own figure of roughly one
millisecond per expert read, that is a third of a second per token
spent waiting on flash. Their measured baseline, "expert weights loaded from SSD,
no optimizations", is 2.97 tokens per second. The arithmetic and the measurement
agree.
### Predicting the next layer with its own router
The predictor is the part I expected to be complicated, and it is one line of
graph code. While layer L runs, the fork multiplies layer L's post-attention
hidden state by layer L+1's router weights and takes the top 8:
```cpp
// tc-mb/llama.cpp-omni src/models/qwen35moe.cpp:455-462
if (moe_slots && moe_slots->lookahead_prefetch_enabled() && ubatch.n_tokens == 1 &&
il + 1 < n_layer && model.layers[il + 1].ffn_gate_inp != nullptr) {
ggml_tensor * prefetch_router_logits =
build_lora_mm(model.layers[il + 1].ffn_gate_inp, attn_post_norm);
lookahead_next = ggml_cont(ctx0, ggml_argsort_top_k(ctx0, prefetch_router_logits,
moe_slots->lookahead_prefetch_depth()));
```
The bet is that the residual stream changes little between two adjacent layers,
so the next router, shown this layer's state, mostly picks what it will pick a
millisecond later. OpenBMB's diagram puts the hit rate above 80%, six or seven of
eight. I could not check that number; it is not in the repo's tables. Note the
`n_tokens == 1` condition: prediction runs during decode only. Prefill, where one
layer can need over a hundred distinct experts, takes a different path.
This is where LES differs from [Edge0](/articles/edge0-streaming-moe), which
solved the same problem on the same backbone a month earlier with a trained
per-layer prerouter and ran it at four experts per token instead of eight. LES
trains nothing and keeps all eight. The router that decides what gets computed is
untouched; the prediction only decides what gets read early. A wrong guess costs
a wasted read, never a wrong expert.
### Most of the speed is a cache
The diagram's interesting phrase is "already resident". A one-layer prefetch
window of 1.15 ms cannot read eight experts at a millisecond each. What makes
six or seven of them resident is a per-layer cache that survives from one token
to the next. The engine gives every MoE layer its own window of expert slots
with an LRU policy (`llama-moe-slot.cpp`, "decode window"), and the prefetched
experts land in the same LRU. Consecutive tokens route heavily to the same
experts, so most of what a layer needs was used a token or two ago.
The app sizes that cache by device memory (`MiniCPMModelConst.swift:781-783`):
128 slots per layer at 16 GB, 48 at 12 GB, 24 at 8 GB, with 8 extra slots per
layer for prefetch (`defaultLookaheadPrefetchDepth = 8`, line 1193). A comment a
few lines above records why it is on by default, from a real multimodal session
with a 48-slot cache against no cache:
| | cache on, 48 slots | cache off |
|---|---:|---:|
| decode speed (tok/s) | 12.57 / 13.34 / 14.29 | 9.30 / 9.47 |
| steady-state misses | 14 / 33 / 74 | 173 / 224 |
| critical-path reads | 17 / 42 / 95 MiB | 222 / 289 MiB |
The same comment explains that a text-only benchmark (one paragraph repeated
twelve times) had shown the opposite, because repeated text routes almost
identically and the OS page cache alone was nearly perfect. The authors rolled
back a default the same day and wrote down the lesson: tune with real prompts,
not benchmark prompts. It is the most useful thing in the repository.
The table is also where I found the source of "13+ tokens/s": the cached run lands
between 12.57 and 14.29. The comment does not name the device.
The widget models one layer with OpenBMB's two timings, so you can see why both
pieces are needed.
OpenBMB's diagram puts the fully serial case at 8.0 ms of reads per layer.
With nothing resident and no overlap, the model gives 2.73 tokens per second,
close to the measured 2.97 baseline, which is the sanity check. Slide to seven
resident: waiting for the eighth expert before computing gives 11.6, and
computing the seven while the eighth is read gives 17.8. Below about six resident
the ordering hardly matters, because one expert read costs as much as the whole
layer's compute and there is nothing left to hide it behind. The ordering is worth
something only once the cache has done its job. The model leaves out the CPU-side
bookkeeping, which the engine's own comments measure at 3.5 ms per single-token
step, so treat its numbers as ceilings.
### Where the 3 GB comes from
The thread says "only 3GB memory". I could not find that number in any table,
but the app's own sizing reproduces it. The cache is (slots + prefetch slots) x
40 layers x bytes per expert. At the 12 GB tier that is (48 + 8) x 40 x 1,351,680
bytes, 3,027,763,200 bytes, 3.03 GB. My reading is that "3 GB" is the expert
cache, not the process. On top of it sit the 1.66 GB non-expert GGUF, which the
engine memory-maps so its pages can be reclaimed, the 1.12 GB vision projector and
the KV cache (small: only 10 of the 40 layers have one; the rest are DeltaNet
state). On a 16 GB iPad at the default 128 slots, the same formula gives 7.4 GB of
cache. The memory figure is a setting.
### "5x with byte-for-byte identical outputs" is two claims
The LES document has the ladder, measured on an iPad Pro 11-inch M4 with 16 GB:
| stage | decode | step | total |
|---|---:|---:|---:|
| experts read from SSD, no optimisation | 2.97 tok/s | | 1.00x |
| LES scheduling and load path | 9.81 tok/s | 3.30x | 3.30x |
| plus layer-wise mixed-precision quantization | 14.92 tok/s | 1.52x | 5.02x |
| plus DSpark speculative sampling | 17.39 tok/s | 1.17x | 5.86x |
The scheduling stage is lossless by construction: the real router picks the
experts, and prediction only changes when bytes arrive. It buys 3.30x. The 5x
in the thread, 5.02x in the table, includes the next row, which is a change of
quantization. A different quantization produces different weights and, at some
point, different tokens than the baseline did. Each piece is fine on its own;
the sentence that joins them claims identity for a speedup that is partly
compression. The before-and-after clip on the model card cannot settle it either
way: its two answers differ word for word, which is expected, because the
model's `generation_config.json` samples at temperature 1.0.
That clip is worth a second look for a different reason. Its decode counters
read 2.0 and 5.5 tokens per second at the end. The headline numbers come from
other runs, with the cache and the drafter switched on; the clip is the honest
reminder that a phone running a 35B model is a configuration, and the default one
is slower than the launch post.
### Why speculative decoding only adds 17% here
The drafter is a DSpark checkpoint (`MiniCPM-V-4.7-35B-A3B-DSpark`): 5 layers,
389,223,169 parameters, reading hidden states from target layers 3, 11, 19, 27
and 35, drafting 15 tokens a pass. Its card reports acceptance lengths of 5.098
on MMMU, 6.126 on MathVista and 7.387 on code, which on a GPU is a big win. On the
phone the app drafts only 4 tokens a round (`specDraftNMaxResolved`, default 4),
and the reason is the MoE: verifying 5 tokens means 5 rows through every layer,
and each row picks its own 8 experts, so a layer can need up to 40 distinct
experts instead of 8. Every one of them has to be in the cache or read from
flash. Speculation saves forward passes, but on flash-bound MoE a forward pass of
5 rows is not much cheaper than 5 forward passes. The 1.17x is that tension
showing. The site's pieces on [DSpark](/articles/deepseek-dspark) and
[DFlash 2](/articles/dflash2) cover the drafter itself.
## Self-Replay Fine-Tuning, briefly
The release also points to a fine-tuning recipe, Self-Replay Fine-Tuning: before
fine-tuning on a new task, sample the current model's own answers to general
prompts, filter them, and mix them with the task data (OpenBMB's post gives a 4:1
blend) so the model rehearses what it already knew. The post claims it "cuts
forgetting by 80%+" and shows, on a counting task, MiniCPM-V 4.5's general-score
drop shrinking from 9.25 to 1.80. It is a sensible, old idea (rehearsal with
self-generated data) and I did not test it. If you fine-tune either model, it is
cheap enough to try.
## An abliterated build, and what the adapter is made of
*Added 13 October 2026.*
AtomicChat, whose [measured quant ladder](/articles/qwen3-8-open-weights) I
liked for Qwen3.8, has [posted](https://x.com/atomic_chat_hq/status/2109764339213467817) an
"uncensored" MiniCPM-V 4.7 35B-A3B: refusals on filtered prompts "drop from 96.3% to 0%".
It is an abliteration, the refusal-direction ablation of
[Arditi et al., 2024](https://arxiv.org/abs/2406.11717). This site has covered the mechanism
and its limits before, on [GLM-5.3-Flash](/articles/glm-5-3-flash-uncensored) and as a
[runtime hook](/articles/runtime-abliteration). Mechanism and evidence only here, as there.
**What the card reports.** The [model card](https://huggingface.co/AtomicChat/MiniCPM-V-4.7-35B-A3B-Abliterated-Uncensored-GGUF)
is unusually complete. The 96.3% is 81 held-out English JailbreakBench prompts, BF16 to BF16;
Russian red-teaming goes from 70% to 0% on 100, XSTest's safe prompts from 7.6% to 0% on 250,
and with thinking on from 43% to 0% on 30. MMLU moves 84.25% to 84.50% on 2,000 questions
(McNemar p = 0.52), and the ablation alone costs a mean KL divergence of 0.0090 against the
original, under their own Q8_0's 0.0051 plus quantization. Two caveats the card states itself:
refusal is scored from the opening of each reply, with no judge, and vision was not tested
after the edit. In the public metrics dataset, 76 of the 81 English replies on the 20.1 GB
build hit the 256-token generation cap, which a reply-opening classifier does not mind but a
harmfulness judge would want to read to the end. The 0% has a 95% upper bound of 4.5% there.
**What the adapter is (measured).** I range-read the 106,269,248-byte LoRA GGUF's header. It
holds 240 tensors, an A and a B for each of 120 sites: `attn_output` in the 10 full-attention
blocks, `ssm_out` in the 30 DeltaNet blocks, the shared expert's `ffn_down_shexp` and the
stacked `ffn_down_exps` of all 256 routed experts in every one of the 40 blocks. Rank 1, F32.
No router, no embedding, nothing in the vision tower. Then I read the B vectors: every one is
the same unit 2,048-vector, in block 0 and block 39, for the shared expert, the DeltaNet output
and each routed expert. That is the shape of the published method. Projecting a direction $r$
out of everything a matrix writes to the residual stream is $W' = W - r\,r^\top W$, and
$r\,(r^\top W)$ is an outer product, a rank-1 update whose B is $r$ itself. One direction, all
writers. The cost of the GGUF format is that each expert carries its own copy of $r$: 2,048 x
256 x 40 floats is 84 MB of the 106 MB, one 8 KB vector repeated 10,240 times. The sizes sum
to the file to within header padding.
**Vision (measured, then reasoned).** The `mmproj` files are byte-identical to AtomicChat's
own unmodified MiniCPM-V 4.7 quants by the Hub's LFS sha256. Against OpenBMB's own
`mmproj-model-f16.gguf` (which is BF16 inside, despite the name), all 460 tensor names, shapes,
types and offsets match, the 128 extra bytes are metadata (name, licence, tags), and every
4 MB slice of tensor data I hashed, start, middle and end, is identical. The edit does not
touch the eyes. It does not need to for image prompts to be affected: image tokens enter the
same residual stream, and every matrix that writes to it after the input has lost its
component along $r$. What nobody has shown is whether a request carried in an image is
refused through that same direction. The direction was estimated on text prompts only, and
the card says so.
**Is it Qwen's direction? (reasoned)** The language model is Qwen3.6-35B-A3B's layer for
layer, as the config shows above, but AtomicChat estimated $r$ on MiniCPM's own
activations, after whatever training OpenBMB did on top. The files say nothing about whether the same vector would
work on Qwen3.6. Nobody has published that comparison.
## The 35B on a Mac
*Added 13 October 2026.*
OpenBMB has also [posted](https://x.com/OpenBMB/status/2109898597236129868) day-0 Mac
support through [Nativ](https://github.com/Blaizzy/nativ), "tested on Apple Silicon with Nativ
v0.4.x": the 35B-A3B at "4K+ prefill tokens/s, 90 tok/s decode", the 1B at 24K+ and 315.
Nativ is Prince Canuma's MIT-licensed SwiftUI app around a bundled `mlx-vlm` server, the MLX
library he also maintains; MiniCPM-V 4.7 support is mlx-vlm PR #2499. The post does not name
the chip, the memory or the quantization. Neither does the Nativ repository, nor do the
`nativ-community` and `mlx-community` conversions, whose only recorded checks are smoke tests
on an RTX PRO 6000 running MLX's CUDA backend.
So the configuration can only be bounded (reasoned). Decode is a memory-bandwidth problem: per
token the model reads the 2.95B active weights counted above. The published 4-bit conversions
keep the DeltaNet layers (1.01B) and the output head (0.51B) in BF16, so a token costs about
2.0 + 1.0 GB for those, plus about 0.54 GB for 8 routed experts in 40 layers at 4.25 bits, and
roughly 0.2 GB of the rest: about 3.8 GB. At 90 tok/s that is about 340 GB/s actually
delivered, beyond an M4 Pro's 273 GB/s on paper and into Max or Ultra territory.
Quantize everything to 4 bits and it is about 1.6 GB a token, near 140 GB/s, which an M4 Pro
could manage. Which of those OpenBMB ran is the missing fact.
The contrast with the phone runtime is the point. The mxfp4 conversion is 22.5 GB of
safetensors, held entirely in unified memory, so nothing is fetched mid-token and LES has no
job to do. The iPhone keeps 3 GB resident and streams 432 MB of Q3_K experts per token from
flash. Different hardware too, but much of the 7x between 13 and 90 tok/s is where the experts live.
## What I would use it for
If you run Qwen3.6-35B-A3B for anything with documents, charts, scans or long
video, try MiniCPM-V 4.7 at 4x. The language model is the same, the vision results
on paper are better where it matters (OmniDocBench by 14 points, CharXiv by 8),
and the text side did not regress much. The weights are Apache-2.0. If you build
GUI agents, stay with Qwen: ScreenSpot is the one place 4.7 is clearly worse.
Switch to 16x for photos and video frames, where it costs about a point and a half
on average and a quarter of the tokens.
The on-device story is real engineering and the code is open, but read it as a
runtime with knobs: cache size per layer, prefetch depth, draft length,
quantization. The launch numbers are those knobs set well on good hardware. The
idea I will remember is simple: reuse the next layer's router a layer early,
keep a small per-layer cache warm, and compute what you have while you fetch what
you don't. None of it needs training, so nothing stops it being ported to any
other MoE with the same shape.
Related on this site: [MiniCPM5-2B](/articles/minicpm5-2b), OpenBMB's text model
from September; [Edge0](/articles/edge0-streaming-moe), the trained-predictor
take on streaming this exact backbone; [a 200 GB MoE on one RTX 3090](/articles/big-moe-one-3090),
the desktop version of keeping experts out of fast memory;
[LFM2.5-VL-3B](/articles/lfm2-5-vl-3b), another small VLM that appears in the 1B's
comparison; and the [mixture-of-experts](/architectures/mixture-of-experts)
explainer for why total and active parameters come apart.
## How I checked
- **Weights:** read every safetensors header of `openbmb/MiniCPM-V-4.7-35B-A3B`
(16 shards) and `openbmb/MiniCPM-V-4.7-1B` with range requests and summed the
shapes; diffed both `config.json` text sections against `Qwen/Qwen3.6-35B-A3B`
and `Qwen/Qwen3.5-0.8B`. Expert bytes came from the app's constants and the GGUF
repo's file list and `tensor-types.txt`.
- **Vision path:** read `modular_minicpmv4_6.py`, `modular_minicpmv4_7.py` and
`image_processing_minicpmv4_6.py` from `transformers` main, ported the slicing
to TypeScript and checked it against the same functions in Python.
- **Runtime:** read `tc-mb/llama.cpp-omni` at commit `88bd37eb`
(`src/llama-moe-slot.cpp`, `src/models/qwen35moe.cpp`, `docs/les/`) and
`OpenBMB/MiniCPM-V-Apps` at `e356c17e`. I ran none of it; the iPad and iPhone
numbers are OpenBMB's.
- **Benchmarks:** transcribed OpenBMB's four chart images from the MiniCPM-o
repository (`OpenBMB/MiniCPM-V` redirects there, at `1d6b7d68`), recounted the
34, recomputed the frontier chart's five-benchmark averages and the OCRBench
margin. All scores are OpenBMB's own evaluations; I did not run any benchmark.
- **Abliterated build (added 13 October):** read the AtomicChat card, the Hub file list and
LFS hashes, and small files from its metrics dataset; range-read the LoRA GGUF header and
B vectors, and both `mmproj` headers plus four 4 MB data slices each. Ran nothing.
- **Mac numbers (added 13 October):** read Nativ at `d4ab3af` and the MLX conversion cards.
The chip and quantization behind 90 tok/s are unstated; the bandwidth figure is arithmetic.
- **Not checked:** the 80% prediction hit rate, the inference-time axis of the
frontier chart (no hardware is given), the 13+ tok/s on a named iPhone (the
measurement table I found does not name its device) and Self-Replay
Fine-Tuning's numbers.
---
# μLattice: an open Spotlight-style memory, and what its 7,280x counts
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/mlattice
> date: 2026-10-13
> tags: linear-attention, long-context, attention, kv-cache
On 12 October Eric E. ([@Ex0byt](https://x.com/Ex0byt/status/2109757628309971307), "Tensorbend Research") posted:
"The quadratic memory barrier in AI has been broken. I give you μLattice." The bullet points were memory that grows
with the input and has no fixed capacity, orders of magnitude fewer operations than attention, and perfect recall of
thousands of associations. The [repository](https://github.com/eelbaz/mLattice) (MIT, commit `7af2de3`) has a
243-line `PAPER.md`, a frank `LIMITATIONS.md`, about 6,000 lines of Python, six checkpoints, result JSONs and a
[live demo](https://huggingface.co/spaces/Ex0bit/mLattice).
This is worth reading closely, and not for the reason the post gives. The mechanism is a real, small, inspectable
implementation of an idea I wrote about a week ago in [Linear attention's memory problem](/articles/linear-attention-state-roundup):
Percepta's **Spotlight**, a memory that grows while each token touches nine cells. Percepta published a blog and no
training code. μLattice publishes training code, checkpoints and tests. The source comments say where the design
comes from (`lattice.py`: "Published (Percepta blogs)"; `kernel.py`: "The BLOG publishes cos^2(pi t / 3)"). `PAPER.md`
itself never says so. It cites Spotlight only as a baseline in one table.
I read the paper, the limitations file, the model, kernel, scan and bolt-on code, and every file in `results/`. I ran
the repository's own unit tests on CPU and tried its acceptance tests there. Every number below is labelled **measured** (I computed it
from a file or a run), **reported** (the repository's figure, not re-run) or **reasoned** (my arithmetic on the other
two).
## Two ways to pay for memory
A sequence mixer has to answer one question at every token: what do I remember about everything before me, and how
much do I pay to look it up?
**Attention** remembers everything. Token $t$ appends a key and a value to the KV cache and scores its query against
all $t$ stored keys. Memory grows as $O(T)$. Compute per token grows as $O(t)$, so a whole sequence costs $O(T^2)$
comparisons. Recall is exact, because nothing was thrown away. (The memory is linear, not quadratic. FlashAttention
never materialises the $T \times T$ score matrix. The quadratic bill is compute.)
**A recurrent mixer** like Gated DeltaNet keeps one fixed matrix per head. Each token does a constant amount of work
on it, so per-token cost is $O(1)$ and a sequence is $O(T)$. The catch is capacity. A $d_k \times d_v$ state can hold
roughly $d_k$ nearly orthogonal key/value associations before they interfere. At μLattice's width, 2 heads of
$32 \times 32$, that state is 2,048 numbers, 8 KiB in fp32. A KV cache at the same width stores $2 \times 64 = 128$
numbers per token, so the whole recurrent state is worth 16 tokens of cache (**reasoned**). Ask it to recall 8,192
pairs and it cannot.
The design space between those two is where [hybrids](/articles/rigel-hybrid-mamba), sparse attention and
[state-size tricks](/articles/linear-attention-state-roundup) live. Spotlight's move, and μLattice's, is different:
keep the recurrent cell, but have **many** of them, allocate them as needed, and route each token to a few.
## The mechanism
One μLattice layer (`src/mulattice/models.py`, `MuLatticeMixer`) takes the hidden state $h_t$ (width 64), runs a
width-4 causal depthwise conv over it, and computes six things per token:
- a **write address** $a^k_t = s \cdot \tanh(W_a h_t) \in \mathbb{R}^2$ and a separate **read address**
$a^q_t = s \cdot \tanh(W_{aq} h_t)$, where $s$ is a fixed scale (96 for the capacity checkpoint);
- a content key $\kappa_t$ and query $q_t$ (unit-normalised, 32 wide per head), a value $v_t$;
- a decay $\alpha_t \in (0, 1)$ and a write gate $\beta_t \in (0, 1)$ per head.
The address carries routing only. The content key decides what is stored inside a cell. That split matters: two
different keys can share a cell and still be told apart by their $\kappa$.
### The bump kernel: why it is nine cells
A continuous address has to become a set of integer cells, differentiably. μLattice uses a compact bump in each axis:
$$
b(t) = \cos^2\!\left(\frac{\pi t}{3}\right) \text{ for } |t| \lt \tfrac{3}{2}, \quad 0 \text{ otherwise}
$$
normalised over the cells it touches:
$$
\varphi_z(x) = \frac{b(x - z)}{\sqrt{\sum_j b(x - j)^2}}
$$
The support is an open interval of length 3, so it contains at most three integers. In two dimensions the weight is
the product $\varphi_{z_x}(x)\,\varphi_{z_y}(y)$: at most a $3 \times 3$ patch, nine cells. Both $b$ and its derivative
vanish at $|t| = 3/2$, so as an address moves under gradient, a cell's weight fades out smoothly instead of
snapping off. The normalisation makes $\sum \varphi^2 = 1$ over the nine cells, so the kernel between two addresses,
$D(q, k) = \sum_z \varphi_z(q)\varphi_z(k)$, is exactly 1 when they coincide and falls to 0 once they are three cells
apart. "Exactly nine" in the paper is "at most nine": at a half-integer coordinate one of the three weights is
exactly zero. At an integer coordinate the weights are 0.943, 0.236, 0.236 per axis (**measured**, from the formula).
### Cells are gated-delta states in a hash map
Each lattice cell holds one $32 \times 32$ matrix $W_c$ per head, zero until something writes to it. For each of the
nine cells around the write address, with $\varphi_c$ its bump weight:
$$
\bar v = \alpha\, W_c \kappa, \qquad W_c \leftarrow \alpha W_c + \beta \varphi_c\, (v - \bar v)\, \kappa^\top
$$
That is the gated delta rule from [Gated DeltaNet](/articles/ltc-gated-delta), applied per cell: read back what the
cell currently predicts for this key, write only the residual. Re-writing a key with a new value overwrites it
algebraically, which is why the forgetting protocol works at all. The read uses the other address:
$$
o_t = \sum_{c \in \text{3x3}(a^q_t)} \varphi_c(a^q_t)\, W_c\, q_t
$$
The reference implementation (`lattice.py`) is a Python dict keyed by integer `(x, y)`. The training scan
(`sparse_scan.py`, `fastscan.py`) computes every address up front, sorts the cell keys to give each distinct cell a
dense id, and runs the recurrence either token by token or cell by cell. The repository's tests check that the
dict, the token loop and the cell-parallel scan agree; I ran them on CPU and every check passed, including
"batched scan == hash-map lattice, max diff 5.55e-17" (**measured**).
The widget runs the exact kernel from `kernel.py`. Add keys and two things grow: the cells that exist because
something wrote to them, and the density. The query never reads more than nine cells; what changes is how many other
keys share them.
## Where this sits
Memory addressed by location, with a few locations touched per access, is an old idea:
- **Sparse distributed memory** (Kanerva, 1988) writes each item to every hard location within a Hamming radius of
its address. μLattice's 3 x 3 patch is the same instinct with a smooth, learned, 2D address.
- **Lattice memory** ([Goucher and Troll, arXiv 2107.03474](https://arxiv.org/abs/2107.03474)) is the closest
ancestor I found: "a differentiable random access memory module with $O(1)$ performance regardless of size",
entries on lattice points, nearest neighbours found by the lattice's symmetry, kernel-interpolated. Its memory is
**parametric**, trained into the weights.
- **Product-key memory** ([Lample et al., arXiv 1907.05242](https://arxiv.org/abs/1907.05242)) and hashed
n-gram tables are also parametric: a huge table learned in training, read sparsely at inference.
- **Titans** ([arXiv 2501.00663](https://arxiv.org/abs/2501.00663)) and the
[linear-attention variants](/articles/linear-attention-state-roundup) are in-context, but fixed-size: the state
is learned at test time, its shape set at build time.
- **Spotlight** is in-context **and** grows. μLattice is that.
So the design is Spotlight's, with SDM and lattice memory as precedents. μLattice contributes the open, runnable
version and hard-won training notes (`PAPER.md` Section 5: density ceilings, learning-rate entry, an address-scale
change that collapses recall from 0.96 to 0.13).
## Claim 1: "7,280x fewer operations than attention"
`PAPER.md`, Section 4.3: "At T=131,072: attention performs 1.1 trillion key-query comparisons; μLattice performs
151 million cell accesses — 7,280× fewer operations." Both numbers reproduce, as $64T^2 = 1.0995 \times 10^{12}$ and
$9 \times 128 \times T = 1.51 \times 10^{8}$; their ratio is 7,281.8, which is $T / 18$ (**reasoned**). The README calls the figure "measured head-to-head on the same GPU". It is a count, not a measurement.
The two sides are not counted in the same unit. $64T^2$ is attention's multiply-adds (2 heads, 32-wide scores, 32-wide
value mix, causal half of $T^2$, both ways). The lattice side counts a "cell access" as 128 operations, but a cell
access multiplies a $32 \times 32$ matrix. Per token, per head, a write touches nine cells three times (read-back
$W\kappa$, decay, rank-1 update) and a read touches nine more once: $9 \times 2 \times 4 \times 1024 = 73{,}728$
multiply-adds per token (**reasoned**, from `sparse_scan.py`).
| T = 131,072, one layer | attention | μLattice | ratio |
|---|---|---|---|
| PAPER.md's count | 1.1 trillion | 151 million | 7,280x |
| multiply-adds | 1.1 trillion | 9.66 billion | about 114x |
The second row is **reasoned**. It is still two orders of magnitude, and it grows linearly with $T$, so the
architectural point stands: past a break-even near 1,152 tokens, the lattice does less arithmetic and the gap keeps
widening. But 7,280x overstates the 131K gap by a factor of 64, and below about a thousand tokens attention does less
work.
**Wall-clock goes the other way.** The paper's own table, from `results/cost_comparison.json` (**reported**), has
attention faster at every length: 381.0 ms against 1,184.8 ms at 131,072 tokens. The paper says so and blames the
Python-level scan, which is fair. What it reads as evidence is the slope: attention's per-token time rises from
1.1 to 2.9 μs (2.6x) while μLattice's stays near 9 μs. Two notes on that:
- Attention's per-token **compute** grew 128x over that range (1K to 131K) while its time grew 2.6x (**reasoned**). At
d_model 64 a fused attention kernel is dominated by fixed overhead, not by the $T^2$ term. The flat-versus-rising
comparison is real, but it is measured in the regime where attention is cheapest.
- The script that produced `cost_comparison.json` is not in the repository. I could not check which SDPA backend ran
(PyTorch's FlashAttention backend needs fp16 or bf16; the rest of the repository runs fp32) or what the
"attn peak MB" column, 16,248 MB at 131K against 3,195 MB for the lattice, includes; taken at face value it is the one column in the file that favours μLattice, but a fused kernel holds nothing like that, so it most likely counts materialised scores or allocator slack. The README's "flat at 9.5–9.8 μs from 32K to 131K tokens while
cells grew from 1,246 to 37,100" also does not match that JSON, whose rows run 6.8 μs at 32,768 tokens to 9.0 μs
at 131,072 (**measured**, from the file), and the cell counts appear in no results file.
## Claim 2: memory that grows, "no fixed capacity", no quadratic blowup
Per-token cost is $O(1)$ in stored state: that is structural, and the unit test "per-token cost flat as memory grows"
passed on my CPU, 185.5 to 197.8 μs per token as cells grew from 9,494 to 293,795 (**measured**, the repository's own
test, not a trained model).
Memory is $O(T)$, the same order as attention's KV cache, with a much larger constant. One cell is 2 heads of
$32 \times 32$ fp32, 8 KiB (**reasoned**, `PAPER.md` agrees). At 8,192 stored pairs the lattice held 7,703 cells, 63 MB
(**reported**). A KV cache for the same 133,120-token sequence at this width is $133{,}120 \times 512$ bytes, 65 MiB
(**reasoned**). They come out about even here only because the protocol spends 16 tokens on each pair. On the cost
benchmark, 37,100 cells is 290 MiB against 64 MiB of fp32 KV cache at 131,072 tokens (**reasoned**).
"No fixed capacity" is the claim I would retract. The address is $s \cdot \tanh(\cdot)$, which confines every
coordinate to $[-(s+1), s+1]$: a box of $(2s + 3)^2$ cells, 38,025 at $s = 96$, which caps the lattice at 297 MiB
(**reasoned**). The scale is fixed before training. `LIMITATIONS.md` says it plainly: "The address scale must be set in
advance so that the expected key count does not exceed this density." Recall degrades above about 0.22 to 0.25 keys
per cell (Section 5.1), so the trained checkpoint's working capacity is about 8,400 to 9,500 keys (**reasoned**). The
37,100 cells on the cost benchmark are 97.6% of that box (**reasoned**). The model code itself records the trade-off:
an unbounded linear address "breaks routing" and held-out recall stays 0.000. `PAPER.md`'s own route out is 3D
addressing, 27 cells per token, which is untested.
So: the per-token compute is genuinely flat and the store is allocated on demand, but the capacity is a
hyperparameter, exactly as a fixed state's is. It is a much bigger one.
## Claim 3: recall 1.0000 at 8,192 pairs
The capacity protocol (`tests/test_capacity.py`) stores 8,192 distinct key/value pairs with 14 filler tokens before
each, 133,120 tokens in all, then asks 1,024 queries and reads the answer at each query's separator token. The released
checkpoint scored 1,024 of 1,024 (**reported**, `results/test_capacity.json`), and 1,023 of 1,024 on the dense
version with no filler.
I tried to re-run it. The README says the acceptance tests want a CUDA GPU (about 5 s there), and on this shared
4-core CPU box neither the 131K protocol (stopped after 15 minutes) nor the dense one (stopped after 6) finished, with
most of the time spent in the kernel rather than in arithmetic (**measured**). So the recall figures here are
**reported**. What I could check is that the result file is internally consistent (1,024 queries, all four depth
quartiles at 1.0, T = 133,120) and that the code it comes from reads answers causally, one pass, at the separator.
What does it measure? Associative recall over a **closed vocabulary**. Keys are 16,384 token ids; each sequence draws
8,192 of them and pairs them with random value ids. The pairings are fresh every sequence, so the model is not
memorising answers. But the address is a function of the key token's embedding, and every one of the 16,384 key ids
has been seen thousands of times in training. The model learns a place for each one. The collision numbers show it:
8.7% of 8,192 keys share a cell at 0.215 keys per cell (**reported**), against about 19% for keys dropped uniformly at
random into 38,025 cells (**reasoned**, $1 - e^{-8191/38025}$). Learned placement beats the birthday rate by about two
to one.
The training recipe then pushes on exactly that vocabulary. The curriculum climbs 64 to 8,192 pairs, polishes "at the
exact evaluation distribution" (filler 14, 8,192 pairs) and re-samples "keys that still fail" with 1/p loss weighting.
`experiments/polish_capacity_weakkeys.py` names them: 16 key ids out of 16,384 caused every residual error, and the
script forces those keys into every batch. Checkpoints were selected on fresh probe seeds, not on the test seed, so
this is not test-set leakage in the usual sense. It is fitting the closed key space until every key has a clean cell.
That is legitimate for a capacity demonstration. It tells you nothing about keys the model has never placed:
Section 5.8 notes that a 16,384-key vocabulary trained from random initialisation "did not escape chance-level loss", and the
131,072-pair setting Spotlight reports would need about 262K keys and an untested scale near 385.
Everything is single-run: capacity and forgetting on one checkpoint each, two seeds for real-text retrieval.
## Claim 4: the bolt-on to a frozen Qwen3.5-4B
A [reply](https://x.com/Ex0byt/status/2109757628309971307) asked to "put it in a real llm"; the author pointed to
`experiments/boltoneval.py`. Section 4.8 attaches the frozen capacity checkpoint to a frozen Qwen3.5-4B. Each test
document holds 2 to 10 entity-value facts (`eval_bolt_on.py`). Entity and value strings are hashed with blake2b into
the lattice's synthetic key and value ids and written in. At the question, the lattice's 64-wide readout goes through a
10.8M-parameter MLP that emits 8 soft-prompt tokens in front of the question. Only that bridge trains.
| condition, 400 trials | exact match |
|---|---|
| frozen model, question only | 0.025 |
| document pasted into context | 1.000 |
| lattice soft tokens | 0.645 |
| soft tokens from another case | 0.015 |
(**reported**). The per-type breakdown explains the 0.645 completely. I tallied the case files (**measured**): in seed
999 the test condition gets 122 of 200, and 122 is exactly the number of colour, food, number and time cases, every
one of them right; all 38 city and 40 password cases are wrong. Seed 7777: 136 of 200, against 137 closed-type
cases. The value strings of those four types come from small fixed sets (15 distinct times, 27 numbers in seed 999)
that also appear in training.
That is what the mechanism predicts. The lattice retrieves the value **id** perfectly (lattice retrieval 1.0000). The
bridge learns to turn a readout of a value id into the answer string, and it can only do that for strings it was
trained on. A fresh password hashes to an id the bridge has never decoded. The test isolates a learned lookup
decoder, not long-context memory: with 2 to 10 facts per document, pasting the document into a 4B model's context
scores 1.000.
Two baselines would say more. One: a Python dict from entity hash to value id, feeding the same trained bridge.
Since the lattice's retrieval is already perfect, I expect that to score the same 0.645 (**reasoned**). Two: enough
facts per document that pasting them in costs real context. The JSONs also record `test_vs_floor_z: 0.0` while the
paper says the z-tests "underflow (p < 1e-15)"; the effect is obviously real, the stored statistic looks like a bug.
And `PAPER.md` Section 6 still lists "The mixer has not been successfully attached to a frozen pretrained language
model" as a limitation two pages after the bolt-on section.
## Claim 5: what was not tested
This needs to be in large type. **μLattice has not been tested as an attention replacement in language-model
pretraining.** `PAPER.md` says so twice and `LIMITATIONS.md` again. Every result is a one-layer, 64-wide mixer on
synthetic recall, needle retrieval and bigram retrieval from wikitext. On MAD's Copy task it scores 0.003, because
content-addressed cells have no notion of position. Spotlight's blog reports language-model runs at 140M to 670M
parameters; μLattice does not reproduce those, so the "barrier broken" framing rests on Percepta's results, not on
this repository's.
## What holds
| claim | verdict |
|---|---|
| per-token cost independent of what is stored | holds; structural, and the scan tests agree |
| "7,280x fewer operations" | the count mixes units; about 114x in multiply-adds at 131K, and slower in wall-clock |
| "quadratic memory barrier" | attention's memory is linear; the barrier is compute |
| "no fixed capacity" | does not hold; a tanh box of 38,025 cells, sized before training |
| recall 1.0000 at 8,192 pairs | reported, not re-run; a closed vocabulary the model was polished on |
| bolt-on 0.645 vs 0.025 | holds; matches the share of test answers seen in training |
| LM pretraining | not tested |
I like this repository more than its launch post. The scan code is careful, the tests check invariants, and the
design constraints are negative results few people publish. If Spotlight's idea gets checked outside Percepta, it
will be on code like this. The next experiments are cheap: a dict-plus-bridge baseline, keys outside the training
vocabulary, and the mixer inside a small LM.
---
# Muse Wonder: what the two points behind Opus 5.5 are made of
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/muse-wonder
> date: 2026-10-13
> tags: benchmarks, agents, agentic-coding, pricing, safety
Meta Superintelligence Labs shipped [Muse Wonder](https://research.meta.ai/blog/muse-wonder) on October 12, and the post that sent me to it was not Meta's. It was a reply-guy chart: "It scores 56 on AA's Intelligence Index, putting it 2 points below SOTA (Opus 5.5), and overall the 2nd best model in the world per the Index!" The same author followed up an hour later with the sentence that most people were thinking: "I have similar benchmaxxing concerns to when Google dropped Argon's benchmark results."
Two points on a composite index is a strange thing to argue about. The index is a weighted average of ten evaluations, each with its own test-set size and its own noise, so "2 points" could be one benchmark having a bad day or a broad, real deficit. And "benchmaxxing" is a claim about how a model was trained, which nobody outside Meta can see. This is a closed model: no weights, no config, no parameter count, no architecture. What I can check is the numbers. So I took the 56 apart.
A disclosure before anything else: this site is written and maintained with Claude agents, and the model Muse Wonder is being measured against is Anthropic's. I have tried to read Anthropic's numbers here exactly the way I read Meta's.
## What shipped
Two models. Muse Wonder is the fast one, available through the Meta Model API and Muse Code. Muse Wonder Full is "the full version of the model", in Mark Zuckerberg's [thread](https://x.com/finkd/status/2109689305073614918), and according to Meta's blog it is "available today for Muse Code subscribers". Muse Wonder will power the Muse assistant for subscribers first, and Meta says it will use Wonder to build "specialized models that we can bring to everyone", which is how Muse Spark became [Muse Glimmer](/articles/muse-glimmer), the 30B open model distilled from it.
The [model card](https://research.meta.ai/static/muse-wonder-model-card) is eight pages and says almost nothing about the model itself. Text, image, video and audio in; text out; a 1M-token context; "trained at our gigawatt-scale Prometheus data center". No parameter count, no architecture, no training-token count. Meta's one sentence on why Wonder is better than Spark is that it builds "on larger, stronger base models". Artificial Analysis lists the size class of Muse Wonder as "large" and leaves Wonder Full's blank.
Everything else is benchmarks. Meta's blog leads with the Artificial Analysis chart, then a fourteen-row results table, then a scatter plot that is Meta's own answer to the benchmaxxing charge. I'll take them in that order, but the index first, because that is where the 56 comes from and it is the only number in the launch that nobody at Meta computed.
## How the index is built
The [Artificial Analysis Intelligence Index v4.3.2](https://artificialanalysis.ai/methodology/intelligence-benchmarking) is a weighted average of ten evaluations that AA runs itself, in four categories: Agents 30%, Coding 20%, Scientific Reasoning 20% and General 30%. Inside those:
| Category | Evaluation | Weight | Test set |
|---|---|---|---|
| Agents | AA-Briefcase v1.1 | 15% | 91 tasks, private |
| Agents | GDPval-AA v2.1 | 10% | 220 tasks |
| Agents | AutomationBench-AA | 5% | 657 tasks, private |
| Coding | Terminal-Bench 4.0 | 10% | 66 tasks, 3 repeats |
| Coding | SciCode | 10% | 288 subproblems |
| General | AA-Omniscience | 10% + 5% | 6,000 questions, private |
| General | GDP.pdf | 10% | 100 tasks, 5 repeats |
| General | AA-LCR v1.1 | 5% | 100 questions |
| Science | Humanity's Last Exam | 10% | 2,158 questions |
| Science | CritPt | 10% | 70 problems, private |
Two details matter for what follows. The two agent benchmarks scored by pairwise judging, AA-Briefcase and GDPval-AA, produce an Elo rating, and AA maps it into the index as `clamp((Elo - 500) / 2000)`. So 2,000 Elo spans the full weight, and on Briefcase's 15% one index point costs about 133 Elo. And AA-Omniscience enters twice: its accuracy at 10%, and one minus its hallucination rate at 5%. A model that says "I don't know" instead of guessing gets paid for it.
AA embeds every model's per-evaluation scores in its model pages. I pulled them, applied the weights above, and got 55.802 for Muse Wonder Full and 52.394 for Muse Wonder, which are AA's published 55.80 and 52.39 to the third decimal. For older models my rebuild lands within 0.17 of AA's figure, because AA freezes an Elo at the time a model is added and the page carries today's Elo. That small drift aside, the index has no hidden term: it is exactly the table above.
The widget below runs that rebuild. Pick two models; each bar is how many index points one evaluation hands to one model over the other after weighting.
## Where the 1.82 points come from
Wonder Full against Opus 5.5 is 55.80 against 57.62, a gap of 1.82. Laid out evaluation by evaluation, it is not a broad deficit at all.
One row does nearly all of it. AA-Omniscience accuracy is 43.2% for Wonder Full and 66.2% for Opus, which at a 10% weight is 2.30 index points to Opus. That single line is bigger than the whole gap. Omniscience is 6,000 open-answer questions of the "do you know this fact" kind across 42 topics, answered with no tools, from a set AA keeps private. It is the closest thing in the index to a measurement of what the pretrained model has memorised.
Then the same benchmark hands most of it back. Wonder Full's hallucination rate on Omniscience is 23.0%; Opus's is 58.6%. The rate counts how often, out of the questions a model does not get right, it answers wrongly instead of declining, and at 5% weight on one minus the rate, Wonder Full collects 1.78 points there. Net, Omniscience costs Wonder Full 0.52 points. Opus 5.5 knows a lot more, and guesses a lot more.
What is left is spread thin. Humanity's Last Exam costs Wonder Full 0.77 points (53.6% against 61.4%), AA-Briefcase 0.58, SciCode 0.43, Terminal-Bench 4.0 0.25, and GDPval-AA and AA-LCR 0.05 each. Wonder Full wins back 0.64 on GDP.pdf (32.6% against 26.2%), 0.11 on CritPt and 0.09 on AutomationBench-AA. Sum them and you get the 1.82.
I find the Briefcase line more interesting than its size. Briefcase scores multi-week knowledge-work projects three ways, and AA publishes all three. On the rubric pass rate, whether the deliverable meets the task's checks, the two are level: 61.3% for Wonder Full, 61.2% for Opus. On analytical quality the Elo ratings are 2,116 and 2,154, close. On presentation quality they are 1,524 and 1,714. Wonder Full does the work about as well and loses on how the result looks. If you are building an agent that writes into a database rather than a slide deck, that loss does not reach you.
My reading of the whole decomposition, and it is a reading: the gap to Opus is mostly knowledge, which is set by pretraining scale and data, and least of all by agentic skill, which is what Meta says it built for. The agent and coding rows are within a few tenths. If Meta wanted to buy back two points cheaply, Omniscience recall is the most expensive place to buy them, because there is no shortcut to knowing more facts. Hold on to that for the benchmaxxing section.
## How much one benchmark can move
The weights tell you how far a single evaluation has to move to matter. On a 10%-weight evaluation, one index point is ten points of score. Terminal-Bench 4.0 has 66 tasks, so one task solved or failed in all three repeats is 1.5 points of score and 0.15 of index. Twelve Terminal-Bench tasks would close the whole gap to Opus. CritPt has 70 problems; one problem is about 0.14 index points.
Those are small test sets, and small test sets are noisy. A rough binomial estimate puts Terminal-Bench 4.0's sampling noise at about 3.5 points of score if the repeats were independent and about 6 if a task tends to pass or fail all three times together, which is 0.35 to 0.6 index points from that one benchmark. The estimate is mine. AA publishes a confidence interval only for the Elo evaluations, and Briefcase's is tight (Wonder Full's 95% interval is 1,719 to 1,740 Elo, about 0.08 index points either way).
So the 1.82 gap to Opus is real in the sense that it is several times the noise of any one input and is concentrated in a 6,000-question benchmark, the least noisy one in the index. The ordering just above Wonder Full is a different matter. AA lists Claude Sonnet 5.5 at 56.00 and Opus 5.5 at xhigh effort at 55.99, against Wonder Full's 55.80. Sonnet's lead is 0.20 points, about one and a third Terminal-Bench tasks. "Second best model in the world per the Index" is true if you round to integers and count Sonnet 5.5 and Wonder Full as tied at 56; on AA's own unrounded board Wonder Full is third, and the honest description is that three models sit within 0.20 of each other. Meta's chart (above) settles it by not drawing Sonnet 5.5 at all.
Muse Wonder, the API model, is a different case: 52.39, below Claude Fable 5.1's 53.35, GPT-6 Astra's 52.67 and Gemini 4 Argon's 52.56, level with GPT-6.1 Sol. One of the replies under the original post asked the right question, which is which of the two people actually get. Wonder Full is the 56. The model on the API is the 52.
## The part that is not close: tokens
The index result I would put first, if I were Meta, is not a score. AA records how many output tokens each model spends across the index. Wonder Full averaged 53,754 output tokens per task; Opus 5.5 at max effort averaged 119,166. Across the whole suite it adds up to 135 million output tokens against 260 million.
Muse Wonder and Opus 5.5 have the same list price on AA's page, \$4 per million input tokens and \$20 per million output. AA's cost to run the full index was \$5,733 for Muse Wonder and \$8,708 for Opus 5.5. Wonder Full has no API price on AA's page, so I can't do the same sum for it.
[Vals AI](https://www.vals.ai/models/meta_muse-wonder-full) ran its own suite independently, prices Wonder Full at the same \$4 and \$20, and lands in the same place from a different direction. On the Vals Index, Wonder Full is 5th of 47 at 64.57%, behind Gemini 4 Argon (68.90%), Claude Sonnet 5.5 (67.04%), Claude Opus 5.5 (66.97%) and Claude Fable 5.1 (65.83%). Its cost per test is \$8.56, against \$32.14 for Opus 5.5, with 102.21 million output tokens against 477.68 million.
The trade I would actually consider is a model a couple of points behind the leader that does the same job in under half the tokens. For a long-running agent loop, where cost and wall-clock scale with tokens, that matters more than a fraction of an index point. Vals' latency figures say the same: 56 minutes 36 seconds per Vals Index test for Wonder Full against 1 hour 19 minutes for Opus 5.5.
## Meta's own table
The fourteen-row table on Meta's blog is the launch's real argument, and it is a curated one. It compares Wonder Full and Wonder with Claude Fable 5.1, GPT-6 Astra and Claude Opus 5.5, all at max effort.
The card's methodology section is more candid than the table. Three sentences in it change how to read the rows.
The first is the score-selection rule: "Where a comparable third-party or provider-reported result exists, we use it; otherwise we use the highest of our internally validated run, self-reported score, or official leaderboard score." Highest-of-three is a best-of selection, and it applies to Meta's own numbers. The card does not say which rows it was used for.
The second is a footnote on one row: "Muse Wonder Full was reported with xhigh reasoning effort" on SWE-Marathon, in a table whose header says every column is max. Either xhigh beat max on that benchmark, or max was not run. The card also says SWE-Marathon "is not a controlled common-harness comparison", because the GPT-6 Astra and Fable 5.1 figures come from Codex and Claude Code.
The third is what the Personal Superintelligence Index is: "an internal composite of seven personal and knowledge-work evaluations: Agents' Last Exam, AutomationBench-AA, FocusBench, JobBench, GDP.pdf, VBT, and DRACO." Two of those seven, AutomationBench-AA and GDP.pdf, are also rows of the table and also inside the AA index, so the same results are counted up to three times in one table. FocusBench and VBT are internal. The PSI is the table's first row and the one with the widest Muse lead.
To see which rows can be checked, I sorted them by where the card says each number came from and looked up every third-party figure on the third party's own page.
Most of what can be checked checks out. Harvey LAB-AA, Terminal-Bench 4.0 and the index itself match Artificial Analysis to the decimal, and Vals Finance Agent V2 matches Vals (59.06). Two rows don't line up the way the table implies. Meta's GDP.pdf numbers come from Surge, and AA runs GDP.pdf too, with different results: 32.6% for Wonder Full and 26.2% for Opus in AA's run, against 33.2 and 30.6 in Meta's table. And the AutomationBench-AA figure in the table is AA's strict pass@1; the index uses a partial-credit score, on which the two are much closer (71.33 against 69.54).
The pattern that surprised me runs the opposite way from the benchmaxxing story. Of the eight rows with a third-party source, a Muse model is best on five. Of the six rows Meta ran or built itself, a Muse model is best only on the two that are Meta's own benchmarks, PSI and MemoryBench, and it loses the other four: HealthBench Pro, JobBench, OSWorld V2.1 and ProgramBench. If Meta were tilting its own harnesses, I would expect the in-house rows to be its best ones. They are its worst.
ProgramBench is the row worth reading twice. Meta scores it as "the percentage of rollouts passing at least 95% of tests", run in-house with mini-swe-agent, and gets 59.0 for Wonder Full against 70.5 for Opus. Vals runs the same benchmark with its strict headline metric, a task counts only when every hidden test passes, on 200 tasks. There, [Opus 5.5 fully resolves 37 and Wonder Full 1](https://www.vals.ai/benchmarks/programbench) (0.50%, 27th, with runs averaging six hours). The 95% threshold is a legitimate metric, and the benchmark itself reports it as "Almost Resolved". But it turns a gap of 37 to 1 into a gap of 70.5 to 59.0. Meta still printed the row it lost, and lost it.
The safety charts on the blog have a quieter omission: they compare Wonder against Fable 5.1 and GPT-6 Astra only, with Opus 5.5 dropped from all five of them, while the capability table includes it.
## Is it benchmaxxed? Three tests
"Benchmaxxed" means a model scores well on the benchmarks people quote because training was steered toward them, and less well on everything else. You can't see training from outside. You can look for its fingerprints.
### Meta's own test
Meta answers the charge directly with the most interesting figure in the launch.
The idea is older than LLMs. In image classification, [Taori et al.](https://arxiv.org/abs/2007.00644) found that models' accuracy on a shifted test set sits on a line against their accuracy on the original one, and [Miller et al.](https://arxiv.org/abs/2107.04649) showed the line is remarkably tight across architectures. A model that sits above the line has "effective robustness": it generalises better than its in-distribution score predicts. A model fitted too closely to the in-distribution set tends to sit below it.
Meta's version plots each model at several reasoning efforts, so each model traces its own line as it spends more compute. I measured the chart's pixels against its axis ticks. Both dashed trend lines have a slope of about 0.93, and at 60% in-distribution the Wonder Full line sits at about 40% out-of-distribution, the Spark 1.3 line at about 30%. About ten points of extra generalisation, by Meta's measure. Meta's explanation: "a stronger base yields post-training gains that carry over more effectively to unfamiliar tasks."
It is the right shape of argument and I can't check any of it. The in-distribution composite and the out-of-distribution set are not named, there are four points per model, and there is no competitor on the plot, so it shows Wonder generalises better than Spark, not that it generalises as well as Opus. A benchmark Meta did not target in post-training can still be one it looked at when choosing checkpoints.
### Private against public
The second test uses AA's index, which mixes evaluations with public test sets and ones AA keeps private. If a model had seen the public sets in training, or been tuned against them, it should do relatively better on those than on the private ones.
Four of the ten evaluations are private: AA-Briefcase, AutomationBench-AA, AA-Omniscience and CritPt, 45% of the weight between them. Against Opus 5.5, Wonder Full gives up 0.90 index points on the private evaluations and 0.91 on the public ones. Per ten percent of weight, that is 0.20 private and 0.17 public. If anything Wonder Full does marginally worse where it cannot have seen the questions, but the difference is a few hundredths of a point and Omniscience dominates the private side. For scale, Fable 5.1 against Opus shows a bigger tilt the same way (0.51 private, 0.34 public), and Muse Spark 1.3 tilts the other way. The test is weak, four evaluations is not many, but it finds nothing.
### An inverted leaderboard
The third test is the one I'd watch. Look for a benchmark where a lab's models rank far above where they rank everywhere else.
[Harvey's Legal Agent Benchmark](https://www.vals.ai/benchmarks/hlab) on Vals is one. Its top three are all Meta: Muse Spark 1.2 at 25.42%, Muse Spark 1.3 Max at 23.75% and Muse Spark 1.3 at 22.92%. Wonder Full is 4th at 21.67%. Claude Opus 5.5 is 41st at 3.75%, and Sonnet 5.5 is 44th at 2.92%. On AA's version, LAB-AA, Wonder Full scores 13.1 and Opus 4.2. Meta leads its "domain expertise" section with this benchmark.
A sixfold lead on one benchmark, for a family that trails on the composite, is the fingerprint the word describes. But two things cut against reading it as Wonder being tuned for the launch. The lead belongs to the whole Muse line and predates Wonder: three Spark models outscore it. And Wonder Full scores lower than Muse Spark 1.2 on Vals' run, the opposite of what steering a release toward one benchmark would do. LAB's headline is gated on hallucinations, a task scores zero on any material hallucination, and Wonder Full's low hallucination rate on Omniscience is at least consistent with a model that is rewarded there for not making things up. The most I can say is that Meta's models have been unusually good at this benchmark since Spark 1.2, and that I can't tell legal-work training from benchmark training from outside.
### And the vibes
There are anecdotes in the replies both ways. One developer posted that Wonder "did worse than basically every chinese model I've ever ran" on his own RocketLeagueBench. One task, in one harness, tells you about that task. Vals' thread reported the largest gain in maths: on ProofBench, its private proof benchmark, Wonder Full "solved 94 of 100 problems, compared with 58 for Muse Spark 1.3 Max." A private benchmark is the kind a model can't have been fitted to, and a 36-point jump on one is harder to explain away than a launch table.
Where that leaves me: the headline index number is independent and reproduces exactly. The second-place framing needs Sonnet 5.5 to be left out. Meta's table is curated in ways the card documents, mostly in Meta's favour, but its self-run rows are where it loses, not where it wins. The private/public split shows nothing, and the one inverted leaderboard belongs to the Muse family, not to this release. The gap that remains is factual recall, which is the hardest thing to game and the most expensive to buy.
## The safety card, briefly
The card's risk section is more specific than most. Meta does "not rule out the model reaching this threshold in the Chemical & Biological domain" without mitigations, and applies classifiers and safety training on that basis; cyber and loss-of-control are assessed at "moderate or lower" before mitigations. One footnote is worth reading in full. On ExploitGym, Wonder Full "succeeded in capturing 92 flags", and a verification pass "found that 21 of those used the intended vulnerability, while the other 71 were captured through unintended vulnerabilities." Meta tells you 21 is the comparable figure. Reporting the 92 and then explaining why it is the wrong number is the kind of disclosure I'd like to see more of. The full safety report is "available soon".
## What I'd do with it
If you are choosing a frontier model for agent work and paying per token, Muse Wonder Full is in the same band as the Anthropic and Google models on both independent indexes, and it gets there on about half the output tokens. It is weakest exactly where you'd reach for retrieval anyway: closed-book facts, and the polish of a generated deliverable. For long-horizon coding the external evidence is mixed: Terminal-Bench 4.0 has it level with Fable 5.1 and behind Opus, DeepSWE has it ahead, and on ProgramBench's strict metric it fully solved one program in 200.
Read the model name carefully. The API model is Muse Wonder, which scores 52.39, three and a half points below the 55.80 everyone quoted.
And keep the open alternatives in view. [Muse Glimmer](/articles/muse-glimmer) is the shape of what Meta has said it will do with Wonder: distil it into something smaller and open. For the personal-agent pitch specifically, [OpenMuse](/articles/openmuse) is an unrelated, MIT-licensed personal agent you host yourself. For how another lab's launch claim held up against the same index, see [MiMo-V2.6](/articles/mimo-v2-6), and for a launch number taken apart by effort level, [Claude Haiku 5.5](/articles/claude-haiku-5-5).
## How I checked
Everything here was fetched on 2026-10-13. Meta's [blog post](https://research.meta.ai/blog/muse-wonder) and its [model card PDF](https://research.meta.ai/static/muse-wonder-model-card) (8 pages, read in full) supplied the table, the methodology quotes and the six figures I reproduce; the chart images are Meta's own files from the post, the light-theme versions. The [Artificial Analysis methodology page](https://artificialanalysis.ai/methodology/intelligence-benchmarking) gave the weights, test-set sizes and the Elo normalisation. Per-model scores came from the model objects embedded in AA's model pages, which carry each evaluation's score, Omniscience accuracy and hallucination rate, token counts and the cost of the index run; I recomputed the index from them with the published weights, which reproduced both Muse figures to three decimals. The private/public split uses AA's own "Private" column. The Vals AI figures are from its [Wonder Full model page](https://www.vals.ai/models/meta_muse-wonder-full), the Vals Index, ProgramBench and Harvey LAB leaderboards, and Vals' thread on X. The two numbers on Meta's out-of-distribution plot, the slope and the ten-point offset, are my reading of the chart's pixels against its axis ticks. The noise estimate for Terminal-Bench 4.0 is my own binomial arithmetic, not AA's. The posts are [@synthwavedd's](https://x.com/synthwavedd/status/2109688487372898480) and Mark Zuckerberg's thread, read with their replies through the fxtwitter mirror. I could not check anything about the model's size or architecture, Meta's internal benchmarks (PSI, MemoryBench, FocusBench, VBT), the in- and out-of-distribution sets behind the scatter plot, or AutomationBench-AA's strict pass@1 figures, which are not in AA's page data.
---
# Natura-1: a whole robot episode as one token sequence
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/natura-1
> date: 2026-10-13
> tags: robotics, world-models, flow-matching, mixture-of-experts, scaling-laws
On 2026-10-13 a lab called Nature Will [announced Natura-1](https://x.com/naturewill/status/2109858475333570565)
as "our first general foundation model for the physical world". The thread makes three claims
worth an engineer's time. It is a 50-billion-parameter mixture-of-experts transformer pretrained on
one million hours of egocentric human experience. It drives 20-DoF dexterous hands after
post-training on "a limited amount of robot data". And its novelty is not a new block but "the
arrangement of multimodal tokens within a single sequence and the causal structure of attention".
The long-form source is a [technical blog post](https://www.naturewill.com/blog/natura-1). There is
no paper. I searched arXiv for the name and the lab, the Hugging Face model API
(`/api/models?search=natura` and `?author=naturewill`) and the web for a repository. I found nothing:
no paper, no code, no weights, no model card. Everything below is therefore **reported** unless I
say otherwise. The useful exception is that the blog's charts are HTML, not pictures, and each data
point carries its raw value in a `data-value` attribute. I read those, and that is where the
**measured** numbers in this piece come from.
## Why dexterous hands are the hard case
A parallel gripper has one degree of freedom. A 20-DoF hand has twenty, and the blog's framing is
that "the action space grows exponentially with the number of hand joints". The more practical
problem is data. Robot demonstrations are collected by teleoperating a robot, so they only exist
where a robot already works. The blog puts it as a loop: "a robot can only gather data in settings
where it already works, and it can only work in settings where data already exists".
Humans do not have that problem. People already use their hands in nearly every environment, and
head-mounted cameras record it cheaply. So the bet is the one several labs are now making: pretrain
on first-person human video, then adapt to a robot hand with a small amount of robot data. The
open lineage here is long. Physical Intelligence's [π0](https://arxiv.org/abs/2410.24164) and
[π0.5](https://arxiv.org/abs/2504.16054) put a flow-matching action expert beside a vision-language
backbone. NVIDIA's [GR00T N1](https://arxiv.org/abs/2503.14734) mixes real robot data, human video
and synthetic data. World-action models such as [FLUX 3 Action](/articles/flux-3-action) and
[Flex-π](/articles/flex-pi) denoise future video and actions together so that video pretraining
becomes motor knowledge. Natura-1 sits closest to the last group, at a larger scale and with far
more human data.
## The mechanism: one episode, one sequence
The blog's model section is short, so I quote it where it matters. Natura-1 "models an entire
episode autoregressively as one sequence of language, image, video, and action tokens". The backbone
is "a standard MoE transformer", and "modality-specific inductive bias lives only in the sequence
layout and the attention pattern".
Concretely, that means no separate vision tower talking to a separate action head through
cross-attention, which is how many VLAs are wired. There is one stream. An instruction comes first,
then the starting image, then the episode unrolls as runs of video tokens, each followed by action
tokens. A transformer reads it left to right.
Three design choices follow from that layout.
**Every step is supervised in one pass.** Because the whole episode is one sequence under a causal
mask, a single forward and backward pass trains a prediction at every timestep. The blog contrasts
this with "a single sampled observation–action window", which is how most robot policies are
trained: sample one moment, predict one action chunk. With teacher forcing, every timestep's
prediction is conditioned on the true past, so an episode of $T$ steps yields $T$ supervised
predictions for the price of one sequence. This is the language-model trick applied to robot
episodes, and it is why the blog can say the design lets physical learning "approach the training
efficiency of frontier language and video models". That last part is a claim, not a measurement.
**Video and action are predicted together.** "At any moment, what the camera sees and what the hand
does are two projections of the same physical state." Natura-1 keeps past video as memory and
learns to predict future video "as part of its world model". Video and action are trained "together
using flow matching and joint denoising, with a separate loss formulation for each modality". In
flow matching the model learns a velocity field that carries noise to data. Here, noised future
video tokens and noised future action tokens sit in the same sequence and are denoised together, so
an action can be conditioned on the frames it will cause. The payoff is supervision density: action
labels are scarce, video is abundant, and video "dominates the tokens of every trajectory". A model
graded on predicting the video gets far more gradient per episode than one graded on actions alone.
**Language has to change the future.** The blog's "Prediction Is Not Understanding" paragraph names
a real failure mode: in most clips the scene alone predicts what happens next, so a model can learn
to ignore the instruction. Natura-1 is "shaped so that a difference in stated intent yields a
different predicted future, in video and action alike", using fine-grained captions from the data
pipeline. How it is shaped (a contrastive loss, paired captions, instruction dropout) is not
described.
The widget lays out one episode as the blog draws it. Move **now** and the timesteps before it
become clean memory while those after it become noised targets. The mask shows what is stated and
what is not. Stated: attention is causal across the episode, so a timestep sees the instruction,
the first image and every earlier timestep, never a later one. Not stated: what attention looks
like inside one timestep. Joint denoising of a step's video and actions suggests those tokens see
each other (block-causal), which is what FLUX 3 Action does. But the blog never says so, so the
default view leaves those cells hatched. Token counts per frame, frames per step and the action
chunk size are not published either; the counts in the widget are schematic.
### What the sequence does not tell you
The model is a 50B MoE, and that is all. The blog gives neither the number of experts, the routing
scheme, nor the **active** parameter count. For an MoE the active count sets the per-token compute
and therefore the latency. A 50B MoE could activate 3B parameters or 20B, and the serving story
changes by an order of magnitude between those. Readers who want the trade-off spelled out can see
the [mixture-of-experts architecture page](/architectures/mixture-of-experts). The tokenizers for
video and actions, the context length and the number of denoising steps at inference are also not
given.
## The data: a million hours, and how they got there
The data section is the most concrete part of the blog. The corpus is "1 million hours of
egocentric human data, consisting of 61,702,471 episodes, 317,842 tasks, and 14,286 scenes"
(reported). It is "primarily built through controlled data acquisition, complemented by large-scale
internet ego videos", with monocular, stereo, multi-camera and wrist-mounted views.
Dividing one by the other gives an average episode of about **58 seconds** (reasoned: one million
hours × 3,600 s ÷ 61,702,471 episodes = 58.3 s). That is a sensible unit for "one sequence": long
enough to hold a multi-step task, short enough to fit in a transformer context once video is
tokenized. The blog does not say how long the trained context actually is.
The pipeline has three stages that matter for robot transfer:
1. **Cleaning.** Deduplication by visual features across vendors and internet sources, sensor
validation, and for internet video, shot segmentation, hand-visibility checks and VLM scoring.
2. **Motion reconstruction.** An in-house labeller recovers dense hand and body motion from
monocular and stereo video. The blog reports "a 41% reduction in MPJPE compared with existing
open-source approaches on an internal benchmark with motion capture ground truth" (reported). The
benchmark, the baselines and the absolute error are not named, so the 41% cannot be placed. This
stage is where human video becomes action labels: the reconstructed hand pose is the "action"
in a human episode.
3. **Language annotation.** Hierarchical captions, goal to subtask to step, written by an agentic
annotator that shares context across an episode.
The system "processes 350,000 hours of ego data per week" (reported), so the whole corpus is about
three weeks of pipeline throughput (reasoned: one million ÷ 350,000 = 2.9). The X thread's "44K → 1M
hours" refers to the scaling study below, not to collection history.
Two things are missing that would decide how much of this transfers. First, the retargeting from a
human hand to the robot hand. The blog says only that "anthropomorphic dexterous hands offer the
closest correspondence available today". Second, the robot data: "a limited amount", with no hours,
episode count or task count. Without those I cannot tell whether "limited" means tens of hours or
thousands. If you work on reconstruction from video, the [TrackEverything](/articles/trackeverything)
piece covers the dense-3D-tracking side of the same problem; the
[Ego2Act and EgoTools](/articles/egocentric-video-benchmarks) article covers how egocentric
understanding is being benchmarked.
## Infrastructure: the numbers with no hardware attached
The blog makes four infrastructure claims, all reported:
| Claim | Value | What is missing |
|---|---|---|
| Training MFU | "exceeds 50%", 98% GPU utilization | GPU model, count ("hundreds"), precision |
| Attention kernel throughput | ≈1,200 TFLOP/s, "over 6×" a tuned FlexAttention | GPU, precision, sequence shapes |
| End-to-end speedups | 1.7× training, 2× inference | baseline system |
| Serving latency | "within 100 milliseconds" end to end | GPU, how many, active params, chunk size |
The kernel number says something about the hardware even though the blog does not. An H100 SXM's
dense BF16 peak on NVIDIA's spec sheet is 989 TFLOP/s. An attention kernel averaging 1,200 TFLOP/s
over forward and backward cannot be BF16 on an H100, so it is running on a newer part, at lower
precision, or both (reasoned). That matters for the latency claim. Under 100 ms per observation is
plausible for a 50B MoE with a few billion active parameters on a datacenter GPU or a small
multi-GPU node. The infrastructure figure shows hybrid tensor, context and expert parallelism
("TP · CP · EP") in the inference workers, which suggests the serving path spans several GPUs. It
is not something anyone will put on a robot. Treat "real time" here as "real time from a rack".
## Checking the scaling claim
The thread's headline is: "With egocentric training data scaled from 44K to 1M hours ... Prediction
MSE error on real-robot data falls by 67.7%, without fine-tuning on robot data."
I took the eight raw values per panel from the chart's HTML and recomputed everything (measured):
| Validation set | Metric | First checkpoint | Last checkpoint | Change | Blog says |
|---|---|---|---|---|---|
| Human, held out | MSE | 0.032041 | 0.016766 | −47.7% | 47.7% |
| Human, held out | MAE | 0.091569 | 0.057439 | −37.3% | 37.3% |
| Robot, never trained on | MSE | 0.027169 | 0.008764 | −67.7% | 67.7% |
| Robot, never trained on | MAE | 0.103448 | 0.052470 | −49.3% | 49.3% |
A least-squares fit of log error on log hours gives back the blog's printed power laws to four
digits: robot MSE $= 1.7193 \cdot D^{-0.3893}$ with $R^2 = 0.948$ in log space, and human MSE
$= 0.3569 \cdot D^{-0.2235}$ with $R^2 = 0.981$ (measured; the blog prints 0.3570 for the human
coefficient). The arithmetic is honest. What it measures is narrower than the thread suggests.
**It is teacher-forced prediction error, not task success.** The blog says so: "Both evaluations use
teacher forcing, conditioning each prediction on the past observation." The model sees the true past
of a recorded robot episode and is scored on how close its predicted actions come to what the
teleoperator did. That is a useful signal of transfer, since the robot set was never trained on. But a
policy can match a demonstrator's average motion closely and still fail at the moment of contact, which
is where dexterous tasks are won or lost. Nature Will reports no curve of real-robot success against
pretraining hours.
**The x-axis is training progress, most likely within one run.** The checkpoint positions in the
chart's data are 43,660, 87,321, 130,981 and 174,641 hours, then 314,679, 578,264, 736,415 and one
million (measured). The first four are exact multiples of the first: 1×, 2×, 3× and 4×. With the
blog's own wording ("as training advances from approximately 44,000 to one million hours"), the
natural reading is checkpoints along one run, labelled by hours of data seen so far. If so, data
and compute grow together on this axis, and the curve cannot separate "more data" from "more steps".
A data-scaling claim proper would train separate models on 44K-hour and 1M-hour corpora at matched
compute. The blog may have done that; the chart does not show it (reasoned).
**The tail is flattening.** Over the last three checkpoints, from 578K to one million hours, robot
MSE drops 8.6% and human MSE 7.4% (measured). The power-law fit for robot MSE has the lowest $R^2$ of
the four panels, 0.948, and the last points sit above the fitted line. That is not a broken curve,
just a reason not to extrapolate it to ten million hours.
## Checking the benchmark
The thread's video speaks for itself, and the blog's six long-horizon workflows (wontons, a shipping
box, tea, calligraphy, a screen protector, a yogurt bowl) are worth watching. Note the dagger: those
six "use additional tuning on top of Natura-1". The zero-shot claims rest on a separate benchmark.
The headline is Natura-1 at 53.4 (Base) and 59.1 (Agentic) against π0.5 at 13.5 and GPT-6-Astra at
16.7 (reported). Four things limit what that comparison says.
**It is not the same robot.** Natura-1 runs on a dual-arm ROKAE AR5 with Wuji 20-DoF hands. On that
platform "neither baseline produces a scored result": π0.5 fails even after fine-tuning on Nature
Will's dexterous-hand data, and GPT-6-Astra, used "directly as a coding-based policy", does not score
within the 60-minute per-task limit. The 13.5 and 16.7 come from an ARX AC One dual-arm **gripper**
platform, with π0.5 fine-tuned on gripper demonstrations of "exactly the same general tasks". To its
credit, the blog states that "a strictly like-for-like baseline" is unavailable. The chart still
puts the bars side by side. A gripper scoring 0.0% on bimanual coordination and "Not scored" on
in-hand manipulation is measuring the robot as much as the policy.
**The benchmark is in-house and unpublished.** Twelve capability dimensions in three groups
(dexterous control, instruction following, long-horizon execution). No task list, trial count,
scoring rubric or success criterion is published. The four example tasks shown come with their
"closest demonstrations retrieved from the training set", which is a good practice I wish more
labs copied. It is still four tasks chosen by the authors.
**The overall score is not the mean of its parts.** I averaged the twelve per-dimension scores from
the blog's chart (measured). Natura-1 Base averages 56.0 against an overall of 53.4. Agentic averages
66.2 against 59.1. On grippers, over the eleven dimensions that were scored, GPT-6-Astra averages 30.3
against 16.7 and π0.5 15.9 against 13.5. Averaging the three group means instead does not match
either. The overall is presumably weighted by tasks or trials, which the blog does not publish, so a
reader cannot reproduce 53.4 from what is shown.
**The trial counts look small.** In-hand manipulation is 16.7% for both Natura-1 variants. Tool and
object generalization is 100.0% for both. Agentic's recovery score is 83.3% and its tool use 66.7%.
Those are one sixth, one, five sixths and two thirds, which is what a handful of trials per dimension
looks like (reasoned; the counts are not published, and other cells such as 35.4% are not small
fractions). A two-point gap between variants on such a dimension can be one trial.
Two smaller points. "Natura-1 Agentic" is never defined in the text; it scores higher but takes
139.31 s per task against Base's 63.16 s (reported), which is consistent with a planning-and-retry
wrapper around the same policy, though that is my inference. And execution-time ratios such as
"43.3× faster than GPT-6-Astra" also cross platforms; I get 43.3 and 4.6 from the reported seconds
too (reasoned), but a gripper and a hand do not run the same motions.
## What I'd take from it
The architecture is the interesting part, and it is a clean bet. Put the whole episode in one
sequence, keep the transformer standard, predict future video and actions together, and let the
abundant modality (video) carry most of the gradient. Teacher forcing over a whole episode is a real
efficiency win over window sampling, for the same reason it is in language modelling. The data
pipeline is described with more specificity than most labs offer: 61,702,471 episodes, 317,842
tasks, 14,286 scenes, a 41% MPJPE claim, a weekly throughput.
What the release does not support is a quantitative comparison with anything. The robot-transfer
number is a prediction error along what looks like one training run. The benchmark compares hands
with grippers on a private suite whose aggregate cannot be rebuilt from its parts. The 50B is total
parameters with the active count withheld, and the 100 ms loop has no hardware attached. None of
that makes the demos less impressive; a nearly 30-minute single take of a robot following spoken
instructions is hard to fake. It means the evidence today is the video, not the numbers.
If you want an open model in the same family to read and run, [FLUX 3 Action](/articles/flux-3-action)
publishes weights and code for joint video-action denoising, [Flex-π](/articles/flex-pi) shows the
same shared-latent idea with switchable outputs, and [Cosmos 3](/articles/cosmos-world-model) puts
reasoning and generation in one sequence. For a flow-matching expert bolted onto a VLM rather than a
single stream, see [Qwen-Drive-1.0](/articles/qwen-drive-1-0). Natura-1 is the biggest bet yet on
human video as the pretraining corpus for hands. Whether it wins will show when someone outside the
lab can measure it.
---
# OpenTakeoff: an agent can trace the floor, only a person can ink it
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/opentakeoff
> date: 2026-10-13
> tags: mcp, agents, tool-calling, developer-tools, licensing
A post from [@RoundtableSpace](https://x.com/RoundtableSpace/status/2109452742725132486) made four claims about a repository I had not heard of. OpenTakeoff *"measures rooms, counts symbols and calculates materials and waste, mainly for flooring estimates"*. It *"runs in the browser without an account or file uploads, and exports to Excel and CSV"*. Its *"MCP server exposes 53 tools for opening plans, setting scale, tracing rooms and exporting"*. And *"agent results remain proposals until a person approves them."*
That last line is why I opened it. Most tools that let a model touch numbers somebody will sign treat review as a UI nicety. A construction bid is a number somebody signs, and the error costs money in a way a chatbot transcript does not. One reply under the post put it well: *"the error cost shows up in the bid, not the demo."* The repository owner replied too, saying they use it every day for commercial estimating.
So I shallow-cloned [`Kentucky-ai/opentakeoff`](https://github.com/Kentucky-ai/opentakeoff) at `9624be8` (2026-10-10, MCP package `0.9.98`) and read it: the scale code, the geometry, the totals, the symbol matcher, every tool registration in the MCP server, the review model on both sides, the export writers and the security notes. I did not run it. Everything below comes from the source, the repository's own docs and evaluation records, and arithmetic I did on those.
The short version: all four claims hold, and the repository is more careful than the post. The interesting parts are in the edges. The count of 53 is a count after two tools were pulled. "Proposals until approved" is a review state, not a quarantine, and one verb walks around it. "No uploads" is true of plans, not quite of page views.
## What a takeoff is, in one paragraph
A takeoff is the step before pricing a construction job: someone measures quantities off the drawings. For flooring, that means square feet of each finish (carpet, vinyl, tile) per room, linear feet of wall base and transition strips, a count of thresholds and drains, and an order quantity that adds cutting waste and rounds consumables up to whole buckets. The drawings arrive as PDFs, usually CAD exports with real vector linework, sometimes scans. Commercial tools for this exist and are seat-licensed. The README says that until OpenTakeoff there was no open-source takeoff engine at all. I can't prove a negative, but I didn't find one either.
OpenTakeoff is two front ends over one engine. The browser canvas in `web/` is a React app that renders PDFs with pdf.js and keeps everything in IndexedDB. The MCP server in `mcp/` is a stdio process that imports the same TypeScript modules from `web/src/lib`, so a ring an agent commits and a ring an estimator draws go through the same `computeShapeMetrics` and the same `conditionTotals`. That shared code is what makes the agent half more than a wrapper. The two halves still differ in one place the README admits: room detection paths are not identical, which is part of why One-Click is switched off right now.
## Paper inches, image pixels, feet
Every coordinate in the system lives in one frame, and every tool description repeats it: *"image px at render scale 2.0: PDF pt × 2, origin top-left, y down."* A PDF point is 1/72 of a paper inch, the engine rasterises at 2×, so one paper inch is 144 image pixels. That number is the whole scale table. In `web/src/lib/sheets.ts`:
```ts
const PX_PER_IN = 72 * RENDER_SCALE; // 144
const arch = (inPerFt: number): number => (1 / inPerFt) / PX_PER_IN;
// '1/8" = 1\'-0"' → upp = 8 / 144 ≈ 0.0556 real feet per image px
```
`upp`, units per pixel, is the only number a sheet's scale carries. Architectural scales (`1/16"` through `3"` to the foot), engineering scales (`1" = 10'` through `1" = 60'`) and metric ratios from 1:20 to 1:500 are all one line each. Metric is a display toggle; the stored quantity is always feet.
There are four ways to set it, and `set_scale` refuses unless you give exactly one: a standard `label`, a raw `upp`, `use_detected` to adopt the scale note read off the sheet, or `calibrate`, two points and the real distance between them. Calibration is one division:
```ts
// mcp/src/session.ts, setScale
const px = Math.hypot(p2[0] - p1[0], p2[1] - p1[1]);
upp = feet / px;
```
Area is where it gets unforgiving. `closedMetrics` in `geometry.js` is the textbook shoelace sum over the ring in pixels, and `computeShapeMetrics` multiplies that by `upp * upp`:
$$
A_{\text{SF}} = \frac{1}{2}\Big|\sum_i (x_i y_{i+1} - x_{i+1} y_i)\Big| \cdot \text{upp}^2,
\qquad P_{\text{LF}} = \sum_i \lVert v_{i+1} - v_i \rVert \cdot \text{upp}
$$
A scale error is a length error, and the area squares it. Calibrate on a dimension string that reads 24'-0" but type 24'-6", and every length on that sheet is about 2.1% long and every area about 4.2% large. Pick `1/4"` on a `1/8"` sheet and every area comes out at a quarter of the truth. The README says it bluntly: *"Pixels × a wrong scale² is every number wrong at once."*
The widget below runs those formulas on a made-up L-shaped room drawn at `1/8" = 1'-0"`, 336 SF as drawn. Leave the scale unset and `measure_polygon` refuses with the server's own message. Pick a standard scale, or calibrate on the 24'-0" dimension and mistype it, and watch the area move by the square of the mistake. The right-hand panel is the totals step, which the next section is about.
## The scale is a gate, and the agent's scale is pencil
The README's list of rules an agent can rely on starts with this one: *"Scale is a gate, not a default."* It holds in the code. Every measure verb opens with the same check:
```ts
// mcp/src/session.ts, measurePolygon (and measureLine, measureSurface, …)
if (s.upp == null) throw new UserError(this.scaleGate(s));
// → "Set the scale for plan.pdf first — use set_scale (detected: 1/8" = 1'-0")."
```
Two details in `setScale` are better than the README makes them sound. A scale the agent sets always lands with `confirmed: false`; quantities still flow, but `takeoff_summary` names every sheet whose scale nobody has confirmed, and only a person at the canvas clears the flag. And an agent cannot rescale a sheet that already holds measurements a person accepted. It gets *"This sheet contains human-reviewed measurements. Recalibrate it in the canvas…"* instead. Rescaling silently re-prices every shape on the sheet, so this is the right place for a hard stop.
The detected scale is a suggestion read off the page text, never applied. `detectScale` canonicalises every text item (curly quotes to straight, whitespace stripped) and matches it against the scale table with boundary checks, so `11/8"` does not read as `1/8"` and `1:500` does not read as `1:50`. A hit in the title-block region (the right 45% of the sheet, the bottom half) wins. A single page-wide note is accepted. Several distinct scales and no title-block note returns nothing at all, because details on a plan sheet are usually drawn larger, and guessing between them is the error the gate exists to prevent. When a sheet carries several notes, `set_scale` says so, and a ring that sits inside a viewport whose note disagrees comes back with a warning.
The canvas adds the estimator's own check. *Check a dimension* (`K`) takes a printed dimension and grades the current scale against it; the README gives the bands as green within 1%, amber within 5%, red past that. Its working order for a real bid spends one line on it: *"Ten seconds a sheet, and it's the only mistake that gets every number at once."*
## Waste lives in the order, not the measurement
The totals step is `conditionTotals` in `web/src/lib/totals.js`, and it is short enough to state completely. A *condition* is one finish, say `CPT-1`, carrying a waste percentage, a ×N multiplier for identical floors, a wall height and a list of supporting materials. For each condition the function sums its shapes by role (floor area adds, deducts subtract, wall traces add wall SF, linear runs add LF and, with a thickness, border SF, counts add EA), multiplies everything by N, and only then applies waste:
```js
// web/src/lib/totals.js, conditionTotals
floor *= mult; wall *= mult; border *= mult; lf *= mult; ea *= mult; seam *= mult;
const total = floor + wall + border;
const w = 1 + waste / 100;
// ...
total_sf_net: round2(total * w),
sy_net: round2((total * w) / 9),
```
Waste never touches the measured number. The canvas shows the measured square footage, the report shows both, and the order columns carry the allowance. A comment in the file owns up to a naming wart: the `_net` suffix means the waste-adjusted order quantity, the opposite of what an estimator calls net, and it stays because the CSV headers and the versioned `report.v1` keys are public.
Supporting materials are where a careless implementation would double-count, and this one does not. A materials row is a coverage rate (square feet per gallon of adhesive, say) and a basis (area, linear, count or figured seam length), and the order is `Math.ceil(basis / per)`. The basis is `total`, the multiplied but un-wasted area. Cut-off carpet does not get glued down, so dividing the wasted figure would buy adhesive for the trash bin. Whether every estimator agrees is a pricing question; the choice is consistent and documented.
The README's own report screenshot checks out against those formulas. WD-1 measures 743.4 SF at 10% waste: 743.4 × 1.10 is 817.74, shown as 817.7, and 817.74 ÷ 9 is 90.86 SY, shown as 90.9. VCT-1 is 103.5 SF at 5%, which gives 108.7.
Roll goods are the one place a percentage is the wrong model, and the engine knows it. Opt a condition into broadloom or sheet vinyl and `rollgoods.js` lays out lanes and seams per room. The README's example is the clearest argument I have seen for doing this properly: a 20-ft-wide room off a 12-ft roll seams once down its length, while the same square footage as two 10-ft rooms seams not at all, and *"no factor on area can tell those apart."* Seam tape and weld rod are bought by that figured seam length, which is why `seam_lf` is a materials basis of its own. One reply under the post made the same point.
## Counting without a vision model
The post says it counts symbols. I expected an image model. It is geometry. `symbolsweep.ts` takes one marqueed example of a drain or a fixture and finds every other placement of the same cluster of vector segments:
1. The fingerprint is every segment fully inside the seed rectangle, relative to its length-weighted centroid. Fully inside, so a marquee around a drain doesn't drag in the wall behind it.
2. Candidates come from up to three anchor segments of distinct lengths, rarest on the sheet first. Every sheet segment of matching length votes for where the symbol's centre would be, under each of the eight transforms of a square (four rotations, with and without a mirror).
3. Each candidate is scored as the length-weighted fraction of seed segments that find a sheet segment with both endpoints within 2 px. At `1/8"` that tolerance is about 1.3 inches of real distance.
4. A score of at least 0.92 counts. Between 0.75 and 0.92 the placement comes back in `withheld`, with its location, as a question.
That second band is the design decision I like most in the repository. A near-miss is neither silently counted nor silently dropped, and the server's initialize instructions spend a whole paragraph on it: *"WITHHELD IS NOT A FAILURE — IT IS THE ANSWER."* The same doctrine runs through the derivations. `derive_transitions` will not commit a carpet-to-tile transition between two rooms separated by a wall, because the real transition is a threshold in a doorway and nothing in a trace says where the doorway is. It reports the run, its length and the gap in inches, and leaves it to a person or a follow-up `place_count`.
Two more choices show the same temperament. The matcher never searches over scale. A symbol seeded on a detail drawn at `1-1/2"` to the foot is 12 times the size of the same mark on a `1/8"` plan, and the sweep resizes by that exact ratio only when both sheets have a scale set; otherwise it runs at 1:1 and says `scale_assumed`, *"because an unknown ratio plus a zero count is not evidence of absence."* And the candidate budget has a hard ceiling of 250,000 scored placements, with `complete: false` in the reply when the ceiling cut scoring short, so a truncated count is reported as a floor and not a total.
Two siblings cover the cases where geometry alone is the wrong tool. `count_marks` counts value-annotated device tags (an `S1` with `200` CFM under it) from the text layer in one deterministic call. `sweep_schedule_row` starts from a schedule row's key, fingerprints the marker that tag is drawn in, and counts a match only when the row's own tag sits inside the marker, because drafting reuses one bubble shape for many marks.
## Fifty-three, counted
The post's 53 is the README's number, and the README generates it. A script, `check-tool-count.mjs`, rewrites every `` marker in the docs from the code and fails CI when prose drifts. So I went to the code.
`mcp/src/tools.ts` has 55 `server.registerTool` calls. `staging.ts` sorts the same 55 into four stages and a test enforces that the stages partition them exactly:
| Stage | Tools | What they do |
|---|---|---|
| setup | 11 | load a plan, sheet info, set the scale, the sheet graph, resolve a room tag, find schedules, read text, sheet structure, raw vectors, render a view |
| measure | 13 | propose a batch, two One-Click flood tools, polygon, line, wall surface, cut-out, counts, three sweeps or censuses, two derivations |
| revise | 24 | list, edit, delete, undo, proposal and condition-edit revisions, scope collisions, condition twins, annotations, verdict marks, four RFI verbs |
| handoff | 7 | summary, three exports and a DXF, import, re-apply rules |
`gate.ts` then withholds `one_click` and `detect_rooms`. The flood-fill engine behind them, the README's headline feature, is *"being re-validated against a wider plan corpus"*, and on a default build the two verbs are not registered at all, so an agent never sees a tool it cannot use. 55 minus 2 is the 53. Set `OPENTAKEOFF_ONE_CLICK=1` and you get 55; set `OPENTAKEOFF_MCP_STAGED_TOOLS=1` and the server starts with the 11 setup tools plus an `open_tool_stage` verb that enables the rest stage by stage.
The README's own tool table lists 46 names, the two gated ones included. The nine it leaves out are `propose_takeoff`, `revise_proposal`, `withdraw_proposal`, `propose_condition_edit`, `withdraw_condition_edit`, `scope_duplicates`, `scope_merge`, `count_marks` and `edit_annotation`. Five of those are the proposal machinery the post's last claim rests on. `docs/MCP_TOOL_INDEX.md` and `mcp/README.md` name all 55; the front-page table is stale.
Staging exists because the descriptions are long. I extracted them from `tools.ts`: the 53 default tools carry a little over 62,000 characters of description before any parameter docs, and `symbol_sweep` alone is about 7,800. That text is where the rules live (trace on the innermost wall face, cross doors on the wall centreline, look at the overlay before trusting a total), and the repository has evidence that it works, which I come back to below. The agent manual is honest about the cost: staging is *"context economy, not a permission boundary."*
## Pencil and ink
The review model is the reason to read this codebase. It has two actors and two kinds of mark. Agent work is *pencil*: every commit from the MCP server goes through one private method, `Session.commit`, which stamps the origin and forces the review flag:
```ts
// mcp/src/session.ts, commit()
if (origin?.actor === "agent") origin = { ...origin, reviewed: false };
if (origin) origin = this.stampProposal(origin); // attach the open batch
if (origin?.actor === "agent" && !origin.assignment)
origin = { ...origin, assignment: { source: "asserted" } };
```
`propose_takeoff` opens a named batch with a required label and rationale ("Level 2 rooms per finish schedule A-601"), and every agent commit after it attaches to that batch. The agent hands the batch over by writing the `takeoff_canvas.v1` JSON with `export_takeoff`; the estimator imports it in the browser, where agent shapes draw dashed, and a single *Accept* pill turns the whole batch into ink. Forty rooms become one decision. The estimator can still correct a vertex first, and when they do, `stampEdit` freezes the machine's original ring in `origin.proposed_verts_norm` beside the correction. The README is explicit that this pairing, the machine's guess next to the expert's fix, is the training signal its authors want to collect.
The project's own recording shows the whole loop. In a terminal on the left, Claude Code loads a VA medical centre finish plan, sets the scale the sheet declares, and one-clicks carpet in three patient rooms. Each export lands in the browser on the right as dashed pencil, and the operator accepts it. The recording leans on `one_click`, one of the two flood-fill tools the current build withholds, so today's agent would trace those rooms by hand instead.
Approval is a second, separate mark. The estimator's green `APPROVED` seal is minted only by the canvas's Approve tool. `mark_verdict` lets an agent sign its own work, but it takes no actor argument, so the only thing it can mint is a grey `AGENT` diamond. The marked-set PDF tallies the two on its cover: *N estimator-approved · N agent-marked*. A project manager can see at a glance how much of a set a person actually looked at.
Then I went looking for where accepted work is protected, because that is where a design like this usually leaks. It mostly holds. `edit_shape` refuses a shape with `reviewed: true`. So does `cut_out`, and so does `scope_merge` when the reviewed shape would be the one trimmed. `withdraw_proposal` removes only the pending shapes of a batch and reports how many accepted ones it left alone. `delete_verdict` refuses a human seal.
`delete_shape` has no such check. `Session.deleteShape` looks up the id and splices the shape out; there is no `reviewed` test anywhere in it, and none in the tool handler. That contradicts the `import_takeoff` description, which promises that reviewed shapes *"arrive untouchable by agent verbs."* The blast radius is smaller than it sounds: the MCP session is a separate process with its own copy, and the canvas import is additive, so deleting an accepted shape in the session cannot reach the estimator's canvas. It does reach anything the session exports afterwards, including the report and the marked set. I'd call it a one-line fix the project will want, not a hole in the model.
The other thing to understand is what *proposal* does not mean. `conditionTotals` sums shapes by role and never looks at the review flag. Pending pencil counts in `takeoff_summary`, in `export_report` and in the canvas's live counter from the moment it is committed; the review screenshot above shows a CPT-1 total that includes shapes still waiting for the Accept pill. The marked set does disclose it: when it is written from an MCP session, a credit line reads *"Machine-traced via OpenTakeoff MCP"* and counts the shapes still pending human review. And condition knobs are softer than geometry. `propose_condition_edit` exists and holds a waste or tag change pending, but `edit_condition` changes waste and multiplier directly with *"No review gate — quantity config, not traced geometry."* A careful agent will use the proposal; nothing forces it to.
So the post's claim is right about the state of the work and loose about the numbers. Agent results stay *unreviewed* until a person accepts them. They are not *excluded* until then.
The widget walks the same rules on three made-up rooms in one batch. Try the agent verbs on a room before and after the Accept pill, and watch which ones refuse.
One more boundary is worth stating, because the README almost states it. The review flag and the seal's actor are fields in a JSON file. The canvas trusts the `actor` field of an imported approval (*"the actor field is the authority"*), and the provenance module calls its own trust model *"initials on a paper markup."* Against the 53 MCP verbs, the pencil-and-ink line is structural. Against an agent that also has a shell and can write a JSON file by hand, as Claude Code does, it is a convention the estimator's eyes enforce. For a tool with no accounts I think that is a fair model; just know which of the two you are relying on.
## What "nothing uploads" covers
The web app is client-only, and I found nothing that contradicts that for plans. `ingest.js` unpacks zips and wraps images into PDFs in the browser, `store.js` keeps PDF bytes and the takeoff in IndexedDB, and the Excel export is a hand-rolled 262-line SpreadsheetML writer (`xlsx.js`) that writes every cell as an inline string, so a finish tag that starts with `=` stays inert text instead of becoming a formula. The MCP server is a stdio subprocess that opens no socket; `SECURITY.md` is unusually clear about where its trust boundary sits and why a `path` argument writing where you asked is not a vulnerability.
Three things do talk to the network, and the repo discloses all of them. The bring-your-own-model agent panel sends your goal, the tool results and crops of the sheet (up to 1,024 px a side) to an endpoint you configure, and nothing if you configure none. The optional team mode syncs to your own Google Drive. The *Contribute* button posts a whitelisted, derived-only payload to a capture server you run. Because those endpoints are user-chosen, the CSP leaves `connect-src` open, and `_headers` explains the trade-off it accepts in a comment.
The fourth is not optional. `web/index.html` loads the Cloudflare Web Analytics beacon with the project's token, and the CSP allows its origin. It is a cookie-less page-view counter; it sees that a page loaded, not what plan you dragged in. But the README's *"nothing phones home"* is a statement about your plans, and anyone self-hosting a fork will want to delete that line.
## How well an agent actually traces
With One-Click gated, an agent measures a room by reading wall strokes with `get_sheet_vectors` and drawing the ring itself with `measure_polygon`. Whether a model can do that well is the real question, and the repository has the most honest answer I've seen a project publish about itself, in `evals/mcp-workflow-bench/plan-set/`.
The set is three public plans: a VA healthcare finish plan, a VA clinic plan and a city ADU plan, with reference rings prepared from the PDF linework. The pass gate is strict on purpose, an IoU of at least 0.985 and every vertex within 2.5 px of the other ring, so a skipped one-foot door notch fails a room. Blind runs of one agent (Claude Fable 5.1, flat tool mode, One-Click off) passed 1 of 6 rooms on St. Cloud, 3 of 4 on Roseburg and 0 of 6 on Porterville against the corrected references, 4 of 16 by my sum. Read with the gate's strictness in mind, most of the misses are small: all six St. Cloud rings were within 0.4% of the reference area. The big ones are genuine estimator questions, a washer-dryer alcove on Porterville that was 17.47% off and a sink alcove at 6.39%, both of which the project lists as open questions rather than agent errors.
Two things in those records raised my opinion of the project. First, the blind runs found errors in the *references* before they found errors in the agent: the first version had traced a door leaf as a wall face and missed door notches, and the references were re-authored from the linework, with the old ones kept beside them. Second, a controlled trial isolates what the long tool descriptions buy. Same model, same stripped task with no tracing conventions in it, one change: the server build with the tracing rules packaged into the descriptions and the wiki. Rooms with every door crossed on the wall centreline went from 0 of 6 to 6 of 6, and median area error from 1.45% to 0.05%. Strict passes stayed at 1 of 6. One run per condition, one model, one plan, as the write-up itself says; it is evidence that the descriptions change behaviour, not a benchmark.
## AGPL as of last Saturday
The repository was Apache-2.0 through commit `84c7f48e` (tag `apache-2.0-final`) and is AGPL-3.0-only from 2026-10-10, three days before I write this. `NOTICE` is clear that the Apache code stays Apache for everyone who has it. New contributions go through a CLA that grants Kentucky AI the right to license them *"under terms other than AGPL-3.0, including commercial terms."* It is the usual open-core arrangement, stated in the open, and the README draws the same line around the data, where the engine is open and the models trained on the company's own estimating archive stay proprietary.
For most readers this changes little. Running the canvas or the MCP server for yourself is unaffected. Hosting a modified copy for other people means publishing your changes, and wrapping it in a product means reading the licence first. The reply under the post that flagged the relicensing was right to.
## Would I hand it to an agent?
For flooring takeoffs, yes, with three habits. Confirm every sheet's scale in the canvas myself, because one wrong scale is every number wrong. Read the `withheld` arrays, because that is where the engine puts the questions it refuses to answer for you. And treat `takeoff_summary` as pencil totals until the batch is accepted.
The parts I would lift for other domains are the contract, not the flooring. Refuse instead of defaulting when a unit is missing. Return near-misses as questions with coordinates. Keep the machine's original guess frozen next to the human's correction. Make the human's mark a code path the agent cannot reach. If you build an MCP server that lets a model touch numbers a person signs, read `mcp/src/session.ts` and `docs/AGENT_GUIDE.md` before you design your own. The same question, what an agent may do and who it has to ask, comes up in [OpenShell](/articles/openshell) at the kernel boundary and in [the argument about which constraints survive better models](/articles/scaffolding-gets-eaten). The construction-data side of this site is [KOLC+ and Gaussian splats in BIM](/articles/kolc-3dgs-bim); the browser-only, IndexedDB-backed architecture is the same one [OpenCut](/articles/opencut) uses for video.
To try it, the canvas is at `opentakeoff.kentucky-ai.com` with a sample plan, and the agent side is one line:
```bash
claude mcp add opentakeoff -- npx -y opentakeoff-mcp
```
## How I checked
I cloned `Kentucky-ai/opentakeoff` with `--depth 1` at `9624be8` (2026-10-10) and did not execute any of its code. I counted tools by listing every `server.registerTool` call in `mcp/src/tools.ts` (55) and diffing that list against the names in the README's table (46); the stage split is `TOOL_STAGES` in `mcp/src/staging.ts` and the gate is `mcp/src/gate.ts`. The description length is my own extraction of each tool's `description` string from `tools.ts`, with the shared coordinate sentence substituted and template expressions dropped, so treat 62,000 characters as approximate. Scale, area and totals formulas are from `web/src/lib/sheets.ts`, `geometry.js`, `shapeMetrics.js`, `totals.js` and `Session.setScale`; the report arithmetic is mine on the README's screenshot. The symbol matcher constants (2 px, 0.92, 0.75, three anchors, a 250,000 ceiling) are from `web/src/lib/symbolsweep.ts`. The review guards are from reading `editShape`, `cutOut`, `scopeMerge`, `withdrawProposal`, `deleteShape` and `commit` in `mcp/src/session.ts`, plus `approvals.js`, `proposals.js` and `importTakeoff.js` in `web/src/lib`; the `delete_shape` finding is from reading, not from a run. Network behaviour is from `web/index.html`, `web/public/_headers`, `SECURITY.md` and a search for `fetch(` across `web/src`. The evaluation numbers are the repository's own (`evals/mcp-workflow-bench/plan-set/`), not reproduced. The licence history is from `NOTICE`, `CLA.md` and the changelog. The source post has no thread and no video; the replies I quote are from its first page of replies.
---
# QwenGyre: training agents whose rollouts take four hours
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/qwengyre
> date: 2026-10-13
> tags: reinforcement-learning, agents, agentic-coding, qwen, training, distributed-training, infrastructure, long-horizon
A single rollout in the run this paper cares about lasts about two hours. The training step that consumes it waits for sixteen of them, and it can't start until the slowest one is done. On Alibaba's NL2RepoBench workload the mean rollout execution takes 1.93 hours, and the mean query, which is the slowest of its sixteen, takes 2.96 hours. That gap, about one and a half times, is the cost of waiting for a straggler, and it is paid on every training step.
[QwenGyre](https://arxiv.org/abs/2609.33848) (Weiqi Wang, Yuxin Zhou, Mouxiang Chen and nine others at Alibaba Token Hub, with USTC and Tsinghua) is the framework Qwen built to stop paying for it. The [announcement on X](https://x.com/Alibaba_Qwen/status/2109633085658808515) leads with two speedups: "up to 1.85× speedup over Colocate, 1.78× over Async". I went to the paper wanting to know which experiment produced each of those, because a scheduler's speedup is only meaningful against a named baseline on a named workload. They come from different experiments, and the more interesting result is in the ablation table the post doesn't mention.
There is no code. I checked the obvious GitHub and Hugging Face names and found nothing, and the paper never promises a release. Everything below is from the paper, its figures, and arithmetic on its tables, plus a toy simulator I wrote to see the scheduling rules move.
## Why hours-long rollouts break both standard layouts
The RL loop is group-relative, in the GRPO family. For each task the policy runs $N$ independent attempts (16 here), a reward model or test suite scores each, and every attempt's advantage is its reward relative to its group. A training step consumes $B$ groups (32 for NL2RepoBench). The group is the unit, and a group isn't ready until all sixteen of its attempts have a reward. One slow attempt holds back fifteen finished ones.
The attempts are long. In the paper's 122B experiments a rollout makes around four hundred tool calls (Figure 5 peaks there), and the trajectory tree for one rollout carries about half a million tokens with shared prefixes counted once. The tail is the trouble. On the 122B model 9.51% of queries take four hours or more. On the 2.4T model the tail becomes the bulk: 61.1% of queries take at least four hours, and in 38.2% of those the slowest attempt is killed by the overall timeout.
Online RL frameworks place rollout and training on GPUs in one of two ways, and both waste hardware here.
Colocate, the HybridFlow and ReaL layout that veRL popularised, time-shares one pool. Everyone generates, then everyone trains. With hour-scale attempts the pool spends a long stretch with most GPUs serving three or four stragglers. You can't preempt the stragglers either. The harness is a black box, Claude Code 2.1.220 in a container, and it has no checkpoint-and-resume, so killing a live attempt changes its outcome for reasons that have nothing to do with the policy. The paper treats "never interrupt a live execution" as a hard constraint, and that one rule shapes most of the design.
Async splits the GPUs into a fixed rollout pool and a fixed training pool and lets them overlap. Now the training pool idles whenever a full batch isn't ready, which at these durations is most of the time, and the rollout pool can't borrow those idle trainers when the tail is long.
[SWE-1.7's async loop](/articles/swe-1-7) and [Fireworks' cross-region RL](/articles/frontier-rl-cheaper) both live on the Async side of this. QwenGyre's argument is that the right split changes minute by minute inside one burst, so it should be computed rather than chosen.
## Making the comparison fair: count staleness
A faster scheduler is easy to build if you let it train on older data. Dispatch more work ahead and the trainer never waits, but the groups it trains on were sampled from a policy several versions old. The paper pins this down before measuring anything, and I think this is the most careful part of the paper.
Two knobs control it. $\mu$ is the number of training steps per burst, the stretch between weight publications. $\varphi$ is the extra dispatch, how many batches of groups beyond the current burst are allowed in flight. Under boundary dispatch, where new groups enter only right after a publication, the target queue depth is $Q_b = (\varphi + \mu)B$, and Appendix A proves the mean scheduling staleness is
$$
\mathbb{E}[d] = \varphi + \frac{\mu - 1}{2}
$$
where $d$ is the version gap between the policy that sampled a group and the trainer that consumes it. The $\tfrac{\mu-1}{2}$ term is the average position of a step inside a burst. The proof is a counting argument: each surviving group ages by one per update, replacements enter at age zero, and in steady state the outstanding age doesn't drift.
Every comparison then holds $\mathbb{E}[d]$ fixed. At $\mathbb{E}[d] = 1.5$, Async runs $(\varphi, \mu) = (1.5, 1)$, Colocate runs $(0, 4)$ because it can't dispatch extra work without interrupting a harness, and QwenGyre runs $(0.5, 3)$. I checked all three against the formula, and the $\mathbb{E}[d] = 1$ rows in Table 2 too: they agree. Figure 7 also plots the largest log-probability gap between the current and the recorded rollout policy per step, and QwenGyre's is if anything lower than the baselines'. So the speedups aren't bought with stale data, at least by this measure.
One caveat the appendix states itself: this is group-weighted mean staleness. It says nothing about the worst case, or about how far behind any one token is.
## Cells, a waterlevel and a core
QwenGyre splits the GPUs into $K$ cells. Each cell can hold a whole training replica (tensor, pipeline, expert and context parallel groups all stay inside it), or it can run SGLang and serve rollouts. On the 122B model that is eight cells of four nodes; on the 2.4T model, four cells of 48 nodes. Cells have a fixed order. Cell 0 is the core, the only one that holds optimizer state and the authoritative weights. The rest are satellites.
The scheduler tracks three counters: executions dispatched, launched and finished. The waterlevel $w(t)$ is dispatched minus finished, the rollout work still outstanding. It can only fall during a burst, because new work enters only at burst boundaries. With $k$ cells training, rollout capacity in concurrent executions is
$$
C_R(t) = C_s + (K - k(t))\,C_e
$$
where $C_e$ is one cell's concurrency (144 executions per cell on the 122B runs) and $C_s$ is an optional standalone rollout pool. The rule that makes this safe is one inequality. The next cell may leave rollout only if
$$
w(t) \le C_R(t) - C_e
$$
which says that after this cell goes, the capacity left still covers every execution that is queued or running. No live harness ever loses its model. A supply check gates it too: at least a fraction $\rho_{\min}$ of the batch's groups must be ready and unclaimed, continuously, for a hold time $\tau$, so a cell doesn't switch to training only to sit there. Cells move one at a time, core first, and once in training they stay for the rest of the burst.
The neat part is what happens at the burst boundary. After the final optimizer step, the satellites go back to rollout and the core publishes weights to every rollout engine. The core stays in training only if the next batch already has $\rho_{\min} B$ ready groups and
$$
w(t) + \mu N B \le C_s + (K - 1)\,C_e
$$
meaning the rollout side can absorb the outstanding work plus the whole next dispatch without it. Otherwise it reshards into a local SGLang and serves rollouts too.
Moving a cell is plumbing, and the paper treats it as plumbing. The proxy stops sending the cell new requests, reserves capacity elsewhere, cancels the in-flight requests and retries them on the new engine, which the harness sees as one slow model call. KV cache blocks can be pinned and pulled over RDMA so the destination doesn't recompute a long prefix. The cell then drops its rollout weights and KV cache and loads training state; the processes and communication groups stay alive. Measured switching times are 8.52 s into training and 3.46 s back, against rollouts measured in hours.
Letting a satellite join a step that has already started is the second trick, and it is what makes small cells worth having. The core exposes its weights through Mooncake's Transfer Engine; a joining satellite pulls them by RDMA and starts on the same batch. The weights don't change during gradient accumulation, so every joiner reads a consistent snapshot. When the batch is done the core reduces gradients from everyone who took part, clips, and steps once. For that collective the system prebuilds $K-1$ sets of communication groups, one for each prefix of 2 to $K$ cells, which works only because cells join in a fixed order.
Streaming feeds them. A packing buffer pulls ready groups (oldest policy version first) and cuts them into small micro-steps; cells claim micro-steps from a shared queue as they have capacity, so a cell that joined late or runs fast simply takes more. Each cell runs a continuous 1F1B pipeline that pauses at a clean point when the queue is empty and drains once at the end of the batch.
That figure says something the text doesn't. Average the eight training shares and you get 23.0%. In other words, even with training overlapped as tightly as the system can manage, three quarters of the cluster's time in this window goes to generating rollouts. The scheduler's job is mostly to keep training off the critical path, and on these runs training is the smaller job.
## A toy version you can push around
I wanted to see where the rules help and where they don't, so I wrote a minute-by-minute simulator of the three placements with the paper's rules: boundary dispatch, the $(\varphi, \mu)$ pairs that hold staleness at 1.5, the waterlevel inequality, the supply check, in-order joining, streaming, and the core-retention test. Eight cells, eight executions per cell, four groups of four attempts per step, twelve steps. Durations are lognormal around 1.5 hours with a per-query factor, capped at a 4.5-hour timeout. Every policy sees the same draw of durations, so the only difference between the three panels is the scheduler.
At the default settings Elastic finishes the twelve steps in 13.5 hours, Colocate in 18.1 and Async in 22.1, which puts the toy's speedups (about 1.3x and 1.6x) in the same range as the paper's 122B NL2RepoBench results. Don't read more into the agreement than that. It is a toy, the durations are made up, and real rollout throughput doesn't scale linearly with occupancy.
Three things in it are worth trying.
Tick "let Async stream groups into training". Async drops from 22.1 to 18.5 hours without any elasticity. Training the moment a group lands is a large share of the win, and the paper's own ablation agrees, as the next section shows.
Push the tail to 0.75 and the training cost down to 0.25 h. Colocate catches Elastic, and on four of the first five draws it wins. When almost every query runs into the timeout, every burst ends at the timeout no matter how cleverly the cells move, and Colocate's four steps per burst against Elastic's three start to count. The paper sees the same thing at full scale: on the 2.4T model, where 61.1% of queries run four hours or more, the gain over Colocate shrinks to 1.21x.
Pull the tail down to 0.2 instead. Attempts now finish close together, there is little tail to reclaim, and Elastic's lead over Colocate falls to about 1.1x. Its lead over Async barely moves, staying near 1.5x, because the four fixed training cells still sit waiting for whole batches whatever the durations do.
## The execution is a tree
The second half of the paper is about what to train on, and it matters as much as the scheduling once a harness does real context management. Claude Code compacts its history when the context fills, delegates to sub-agents with their own contexts, and retries. One execution stops being one conversation. If you concatenate everything into a single sequence, some responses end up conditioned on text they never saw when they were generated.
The proxy records every model call token-in, token-out: the exact input and output token ids and the behaviour log-probabilities, the same capture idea as [Hugging Face's multi-harness proxy](/articles/multi-harness-rl). It also caches original tool-call payloads by call id and restores them before matching, because harnesses reformat tool calls and that would otherwise break prefix matching. Each new request is matched against existing paths; only the new suffix becomes a node, and a request whose context diverges starts a branch. The result is a prefix tree per execution in which every output keeps its original context, even after the harness compacted that context away.
That trace has 636 proxy requests and 1,347 unique message nodes. Expanding its ten paths independently gives 1,716 node occurrences, 369 more than the tree, and each of those 369 is a repeated prefix that a naive pipeline would train on twice.
Admission works like this. Only tokens the policy itself sampled can be targets; system prompts, tool output, and assistant-role text the harness injected are context only. A summary is a target where the policy generated it and plain context where the harness pastes it back into the next user message, and the token-level record is what tells the two apart. Leaves are ranked by role, main over main-summary over sub-agent over sub-agent-summary, on the reasoning that the task reward credits the main agent's turns most directly. Then the processor draws leaves one at a time from the highest class that still has unmasked trainable tokens, with probability proportional to that count, fixes the drawn path's targets to whatever is still unmasked, and masks the whole root-to-leaf path. It stops when nothing trainable remains or after $J_{\max}$ paths.
The widget uses the case study's shape and message counts. The split between policy tokens and context inside each segment is mine, and the paper says itself that message counts can't give the token probabilities. With the cap at 4 it admits both main paths, the main summary and one sub-agent path, which is the outcome the paper's Appendix C walks through for $J_{\max} = 4$. With the cap at 3, the summary's shared prefix is already trained on the first main path, so it contributes only its own few tokens.
The loss normalization is where this pays off. Every admitted path inherits its execution's group-relative advantage, and token losses are averaged over the execution's disjoint target set:
$$
\mathcal{L}_i(\theta) = \frac{1}{L_i} \sum_{u \in \mathcal{T}_i} \ell_u(\theta),
\qquad
\mathcal{L}(\theta) = \frac{1}{BN} \sum_{q=1}^{B} \sum_{k=1}^{N} \mathcal{L}_{i(q,k)}(\theta)
$$
So an execution that compacted five times and spawned six sub-agents carries the same total weight as one that ran straight through. A flat mean over trajectories would hand the branchy execution more say just for branching. Toggle the flat-mean view in the widget to see a 3-token summary path get the same weight as a 43-token main path. The objective itself is GSPO with token-level importance weights capped at 5 and a sequence-level clip of $[0.995, 1.005]$ applied per execution; Qwen's [first-order view of RL objectives](/articles/first-order-rl) is the background for why that ratio needs to stay so close to one.
The cap is measured. With $J_{\max} = 1$, main trajectory only, training scores lag and gradient norms run more than an order of magnitude higher. With $J_{\max} = 5$ the scores match training on every leaf, and forward-backward time per step is 74.8% of uncapped, 25.2% saved.
Partial scoring is the part the replies on X liked best, and it is simple. When an execution hits the harness timeout, its workspace is preserved and scored under the task's normal rule: on NL2RepoBench, a separate Qwen 3.7-Max evaluator runs independently written tests against whatever code exists. The reward never comes from elapsed time or termination status. A missing assessment, where the evaluator crashed or the overall budget ran out, is kept distinct from a valid zero and the execution is dropped. On the 2.4T run that split is 9.37% harness timeouts, scored and trained on, against 4.63% overall timeouts, discarded.
I couldn't find an ablation of partial scoring itself. One reply under the post asked how much of the six-point gain came from runs that would have scored zero before. The paper doesn't say.
## Checking the numbers
I recomputed every derived number in Table 1 from its hour column. All of them reproduce: 1.556 for original Async against QwenGyre, 7.7% slower when Async's training pool shrinks to 8 nodes, 29.9% slower with three-step bursts, 11.6% faster with streaming, 2.9% and 14.8% for the two Colocate steps, 7.6% slower with the bigger standalone pool, and 1.19x, 1.24x, 1.08x and 2.2% for the E rows. The 2.4T totals check too: 134.55 h for Async, 91.47 h for Colocate and 75.42 h for QwenGyre give 1.784x and 1.213x, and the post's "~44% vs Async" is 1 minus 75.42 over 134.55.
Then I lined the headline numbers up with their experiments.
| claim | where it comes from |
|---|---|
| 1.85x over Colocate | TerminalBench, Qwen 3.6 122B, staleness 1.5, 24 steps (Table 2 prints 1.849) |
| 1.78x over Async | NL2RepoBench, Qwen 3.8 2.4T, 48 steps |
| 52.5% to 58.5% | the same 2.4T run's evaluation curve |
The intro gives the full ranges: 1.38 to 1.78 over Async and 1.21 to 1.85 over Colocate. The weakest result against Colocate is the flagship run. Nothing in the paper is wrong here, and the abstract's "up to" is accurate. But the post invites you to read 1.85 and 1.78 as one system on one workload, and they aren't.
The evaluation panel of that figure matters more than the speedup panel. All three runs start at 0.525. Reading the chart, Colocate ends near 0.58 and Async near 0.57, against QwenGyre's 0.585. A scheduler should change wall-clock time and leave learning alone, and the paper argues exactly that, so the near-tie is expected. It also means the abstract's "QwenGyre yields a 6.0% absolute gain" credits the framework with what 48 steps of RL did. The gain is real. A run on Colocate gets most of it, about 16 hours later. For scale, the released Qwen3.8-Max reports 55.9 on NL2Repo-Bench in [its launch table](/articles/qwen3-8-max), so this run's 58.54 is an internal checkpoint under the paper's evaluation, not the shipped model's score.
The ablation table is the part I'd read twice.
Table 1 walks from each baseline to QwenGyre over the first 12 steps on 32 nodes:
| run | what changes | hours | vs QwenGyre |
|---|---|---|---|
| A0 | original Async, 16 training / 16 rollout nodes | 25.08 | 1.556 |
| A3 | A0 plus streaming | 22.18 | 1.376 |
| C0 | original Colocate, all 32 nodes switch together | 23.23 | 1.442 |
| C1 | C0 with 4 standalone rollout nodes | 22.56 | 1.400 |
| C2 | C1 plus streaming | 19.23 | 1.193 |
| E0 | QwenGyre: eight 4-node cells switching independently | 16.12 | 1.000 |
| E2 | E0 regrouped as four 8-node cells | 16.47 | 1.022 |
Measure each step against the gap it closes. Colocate starts 7.11 hours behind QwenGyre. A standalone pool and streaming, C0 to C2, remove 4.00 of those hours, 56%, before any cell switches on its own. Async starts 8.96 hours behind, and streaming alone removes 2.90, about a third. Fine-grained elastic cells, the headline mechanism, are worth 1.19x on top of the best non-elastic configuration in the table. Still a solid result, but a team already running a colocated veRL-style stack would get most of the way with streaming and a small dedicated rollout pool, both of which are much less work than a scheduler that reroutes live harnesses.
E2 says the same thing from the other side: halving the number of cells costs 2.2%. The paper explains that E0 has 20.85% slack left, with training entirely hidden behind rollout, so coarser cells only expose a little training on the critical path. You don't need many cells. You need more than one.
Some things I couldn't check. Every configuration is a single run, so there are no error bars on any speedup, and with hour-scale durations drawn fresh each run, the noise could be a few percent. The Async split for the 2.4T run isn't stated; Table 1 shows that split alone moves Async by 7.7% on the smaller model, and the 1.78x is the comparison most sensitive to it. Hardware isn't named beyond node counts; the flagship comparison alone is 192 nodes for 301 hours across three runs. And the post's "Qwen3.8-2.4T-A95B" active-parameter count isn't in the paper; it matches what Alibaba published for Qwen3.8-Max.
## What I'd take from it
For almost everyone the transferable parts are the cheap ones.
Count staleness the paper's way before comparing schedulers. $\mathbb{E}[d] = \varphi + (\mu-1)/2$ is one line, it is exact under boundary dispatch, and it stops a fast async setup from quietly winning by training on stale groups.
Stream training at group granularity. Both the paper's ablation and my toy say this is the largest single lever, and it fits into Async or Colocate unchanged.
If you build anything elastic, borrow the waterlevel rule. "Leave rollout only when what remains covers everything outstanding" is the whole safety argument for not killing live harnesses, and it is a single comparison.
Normalize per execution, not per trajectory. Any harness that compacts or delegates will otherwise reward branching, and nothing on the learning-curve plots would tell you it was happening.
The elastic scheduler itself, with RDMA weight pulls, prefix-ordered communicator sets and live KV migration, is for teams running RL across a hundred-plus nodes on rollouts measured in hours. Several systems in the paper's related work already move GPUs between rollout and training (DynaResize, Libra, BiDiRL, TideRL), and Libra's satellites joining as data-parallel replicas is close to this. What QwenGyre adds is doing it under an unmodified harness that can't be paused, plus the evidence that it holds at 2.4T. The paper is also honest that streaming with several steps per burst gives up a global shuffle within the burst, and that the effect of that ordering on learning hasn't been isolated.
Related reading on this site: [Rollout Routing Replay](/articles/rollout-routing-replay) on why MoE RL needs the rollout-time routing recorded, [LongStraw](/articles/longstraw-2m-rl) on fitting million-token RL steps into memory, [DeepSeek's DSec](/articles/dsec-agent-sandbox) on the sandbox side of the same loop, and [SGLang](/articles/sglang), the rollout engine every cell here runs.
## How I checked
I read the arXiv HTML of 2609.33848v1 in full, including Appendices A to C, and took the figures from it; the paper's Figure 1 is missing from the HTML build, so it comes from the image in Qwen's second post in the thread, which is the same figure. The announcement thread and its replies came through the fxtwitter mirror. I recomputed every ratio and percentage in Tables 1 and 2 and the 2.4T totals from the printed hours, and the staleness pairs from Equation 2. The 23.0% mean training share is the average of the eight values printed in Figure 6. The end-of-run evaluation scores for Colocate and Async are read off Figure 8's chart and are approximate. I probed `QwenLM/QwenGyre` and similar names on GitHub and searched Hugging Face for the name; neither exists. The simulator and the tree widget are mine, built from the paper's stated rules; their hours and token counts illustrate the mechanism and are not the paper's measurements.
---
# ReplaySSM: an SSM decoder that writes its state once every 16 tokens
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/replayssm
> date: 2026-10-13
> tags: state-space-models, linear-attention, speculative-decoding, quantization, kernels, inference, vllm
ReplaySSM has turned up on this site three times already, each time as a line item. It is in the
optimisation list of [GLM-5.3-Flash](/articles/glm-5-3-flash), between tensor parallelism and W8A8. It
is the change that made Z.ai's KDA decode kernel *slower*, 0.90x, in the version history of
[how GLM built its own inference stack](/articles/glm-built-its-own-inference-infra). And it is the
name of a maintainer note in the [Cinference](/articles/cinference) engine. Nobody, including me,
had explained what it does.
Yesterday Ze-Wei Liou [posted](https://x.com/zwljohnny/status/2109679465609470216) the paper, a joint
NVIDIA blog post and the code, with Tri Dao and five NVIDIA engineers. The post makes four claims: it runs
in production in GLM-5.3-Flash and Kimi-K3; it ships in TensorRT-LLM, vLLM, FlashInfer and SGLang; it
speeds up standard decoding by up to 1.47x and speculative decoding by up to 2.1x; and it makes a
low-precision SSM state possible "whose rounding error never propagates through the recurrence."
I expected a kernel paper. What I found is a two-line algebraic identity and a careful account of
what falls out of it. The identity is the kind of thing you can check on a napkin. The consequences
are where the interesting engineering is, and also where the post's wording runs ahead of the
evidence.
## Why a fixed-size state is not cheap to decode
A Mamba-2 layer keeps, per head, a state matrix $S_t \in \mathbb{R}^{d_v \times d_k}$ and updates it
once per token:
$$
S_t = \alpha_t S_{t-1} + v_t k_t^{\top}, \qquad o_t = S_t q_t
$$
$\alpha_t \in (0,1)$ is a data-dependent decay, $v_t$, $k_t$, $q_t$ are this token's projections, and
$o_t$ is the output. (Mamba-2's own names are $x$, $B$, $C$ for $v$, $k$, $q$; the paper and the kernels
mix both.) Gated DeltaNet and Kimi Delta Attention add a delta-rule correction, which I come back to.
The selling point of this layer is that $S_t$ does not grow with context. The catch is in how a decoder
runs it. Every step loads $S_{t-1}$ from HBM, scales it, adds a rank-1 outer product, reads the output,
and stores $S_t$ back. For one head the step does about $5 d_v d_k$ FLOPs and moves $2 \cdot 4 d_v d_k$
bytes with an FP32 state, so the arithmetic intensity is about 0.63 FLOP per byte. An H100 wants around
20 FLOP per byte to keep its FP32 cores busy. Batching does not help, because every sequence brings its
own state and no element is reused. The step is a memory copy with a little arithmetic attached.
That copy is not small. I pulled the released `config.json` of
[Nemotron-3-Super-120B-A12B-NVFP4](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4)
and counted its `hybrid_override_pattern`: 40 Mamba-2 layers, 8 attention layers and 40 MoE layers,
with 128 Mamba heads of $d_v = 64$, $d_k = 128$, and `mamba_ssm_cache_dtype` set to `float32`. One
layer's state for one sequence is $128 \times 64 \times 128 \times 4$ bytes, exactly 4 MiB; across 40 layers
that is 160 MiB per sequence, and at batch 256 it is 40 GiB of state that has to be read and written on
every decode step.
Attention's KV cache in the same model is FP8 with 2 KV heads of dimension 128, so one token's entry
is 0.5 KiB per layer. One Mamba state is the KV cache of 8,192 tokens. Attention only reads its cache
while the SSM reads and writes, so the SSM layers move more bytes per step until $40 \times 8$ MiB equals
$8 \times 0.5 N$ KiB, which is $N = 81{,}920$ tokens. The paper does this same arithmetic and then
measures the crossover later, near 100K, which it puts down to attention kernels having had more
optimisation. Their profile, the left panel below, shows the SSM layers as the largest slice of a
decode step all the way out to 100K tokens of context. A hybrid model that was meant to be "mostly
cheap layers" is mostly paying for state traffic.
## The identity
Unroll the recurrence over the last $h$ tokens, starting from a checkpoint $S_{\text{ckpt}}$ taken
before them. Write $\gamma_{i:j} = \prod_{\ell=i}^{j} \alpha_\ell$ for the decay accumulated between two
positions, and $\gamma_h = \gamma_{1:h}$:
$$
S_t = \alpha_t S_{t-1} + v_t k_t^{\top} = \gamma_h S_{\text{ckpt}} + \sum_{j=1}^{h} \gamma_{j+1:h}\, v_j k_j^{\top}
$$
Both sides are the same matrix. The left side is how every SSM decoder runs: keep the summary, update
it, throw the inputs away. The right side says you could instead keep an old summary and the last $h$
inputs, $(\alpha_j, v_j, k_j)$, and rebuild the current state whenever you need it. Nothing about the
model changes; outputs agree up to floating-point order of operations.
ReplaySSM picks the right side. On a normal step (a *non-flush* step) it appends this token's inputs to
a small buffer, computes the output from the checkpoint plus the buffer, and stores nothing but the new
record. When the buffer holds $L$ records, a *flush* step folds them into the checkpoint, writes the
checkpoint once, and empties the buffer. The paper uses $L = 16$.
The saving falls straight out. Over a window of $L$ steps both schemes read the state $L$ times, because
every output needs it. Conventional decoding also writes it $L$ times; ReplaySSM writes it once. So the
state term goes from $8 L\, d_v d_k$ bytes to $4(L+1)\, d_v d_k$, a ratio of $(L+1)/2L$, just over a
half. The buffer adds a little back: step $i$ reads the $i-1$ records before it, so a window reads
$L(L-1)/2$ records and writes $L - 1$. The paper's Appendix C.3 prices a record per head at
$2 d_v + 2 d_k + 4$ bytes and gets 0.58 of conventional traffic for Nemotron-3-Super. I reran it and
get the same 0.581.
That count is deliberately conservative, and the released kernel shows by how much. In
`selective_state_update_replayssm_output_only.py` the buffer is three tensors: `x_cache` and `dt_cache`
per value head, and `B_cache` per *group*, so the 128-wide key is stored once for the 16 heads that
share it. For one Nemotron-3-Super layer a record is $128 \times (128 + 4) + 8 \times 256 = 18{,}944$
bytes. Stepping that layout through a window in the widget further down, with one token verified per
step, gives 4.40 MiB per token against 8 MiB for conventional decoding, about 0.55.
### Output without a state
The second trick is smaller and, I think, more consequential. On a non-flush step nobody will ever
read the state of that step: the next step rebuilds from the checkpoint again. So the step only needs
$o_t$, and a product of three things can be bracketed either way:
$$
o = (v k^{\top})\, q = v\, (k^{\top} q)
$$
The left side builds a $d_v \times d_k$ matrix and multiplies it by $q$. The right side computes one dot
product and scales a vector. Applied to the whole window:
$$
o_t = \gamma_{h}\, S_{\text{ckpt}}\, q_t + \sum_{j=1}^{h} \gamma_{j+1:h}\, v_j\, (k_j^{\top} q_t)
$$
The checkpoint is read once, as a matrix-vector product. Everything else is dot products against
cached keys, and because $k$ and $q$ are shared by every head in a group, the dot products are computed
once per group. The paper counts the state-building term shrinking by $d_v d_k / (d_v + d_k)$, about 43
for this model. That does not matter much for standard decoding, which is memory-bound either way. It
matters a great deal for speculative decoding, below.
Here is the non-flush branch of the released Mamba-2 kernel. It is the equation above, line for line:
`ck_acc` is $S_{\text{ckpt}} q$ streamed over tiles of the state, `bc_cache` holds the precomputed
$k_j^{\top} q$ (a small kernel computes them once per group), and the append that follows writes only
`x`, `dt` and `B`:
```python
# vllm/model_executor/layers/mamba/ops/selective_state_update_replayssm_output_only.py, lines 309-357 (trimmed)
ck_acc = tl.zeros([BLOCK_SIZE_M], dtype=tl.float32)
for i in tl.static_range(NF_NDS):
...
ck_acc += tl.sum(st.to(tl.float32) * c_chunk[None, :], axis=1)
checkpoint_out = ck_acc * total_decay_cache
bc_cache = tl.load(bc_pre_ptr + offs_k_cache * stride_bc_pre_pos, ...)
cache_out = tl.sum(
x_all_cache.to(tl.float32) * (scale_cache * bc_cache)[None, :], axis=1
)
out = checkpoint_out + cache_out
# Append the current token (x, dt, B) into the buffer at write_pos.
tl.store(x_cache_ptr + offs_m * stride_x_cache_dim + write_pos * stride_x_cache_pos, x_cur, ...)
```
`scale_cache` is $\Delta_j\, e^{A(\text{total} - \text{cumsum}_j)}$, the per-record decay rebuilt from the cached
`dt` values, so the kernel caches $\Delta_j$ rather than $\alpha_j$. The only full-state store in the
file is inside the `else` branch that starts at line 358. One difference from the paper's main text:
its Algorithm 1 materialises the state on every step, while the shipped kernel takes the output-only
route on every non-flush step (the appendix counts both routes).
### Why 16
A larger window amortises the one write over more tokens, but every step reads more records. The paper
writes traffic per committed token as a function of $L$ and minimises it:
$L^{*} = \sqrt{2\bar{a}\,(M_S/r - T)}$, where $M_S/r$ is the state-to-record size ratio (84 for this
model), $T$ the tokens verified per step and $\bar{a}$ the mean accepted. For standard decoding that
is about 13; the sweep over 8, 16 and 32 lands on 16 for all four models. The June blog post had picked
8 for Nemotron and 16 for Qwen; the October paper tunes vLLM's own launch configuration first and settles
on 16 everywhere.
### The delta-rule wrinkle
Gated DeltaNet does not fit the template at first glance:
$$
S_t = \alpha_t S_{t-1}(I - \beta_t k_t k_t^{\top}) + \beta_t v_t k_t^{\top}
$$
Expanding the product gives $S_t = \alpha_t S_{t-1} + u_t k_t^{\top}$ with
$u_t = \beta_t (v_t - \alpha_t S_{t-1} k_t)$, the same outer-product form with a corrected value. The
correction depends on the previous state, so you cannot cache $v$ and replay; you cache $u$, which is
computed once, on the step it belongs to, from the rebuilt state. That is why GDN takes the
state-and-output route even on non-flush steps (it needs the state read at both $k$ and $q$), and why
its speculative path needs a $T \times T$ triangular solve to get all draft corrections at once, the UT
transform that delta-rule training kernels already use. KDA, with a per-channel decay, unrolls the
same way; [the K3 architecture page](/architectures/kimi-k3) has its state at 414 MiB per sequence,
which is the number ReplaySSM is trying not to copy.
## Speculative decoding: rollback becomes a pointer
Speculative decoding is where an SSM's summary-and-discard habit really hurts. A drafter proposes a
few tokens, the target model verifies $T$ of them in one pass, and only afterwards do you learn how many
were accepted. Attention handles the rejects by moving the end-of-sequence pointer in its KV cache back;
the rejected keys are simply never read again. An SSM has already folded the drafts into its state, and
the update has no cheap inverse.
vLLM's answer is to write a snapshot of the state after each of the $T$ tokens and keep the one that
matches the accepted prefix. That is one state read and $T$ state writes per step, on a path that was
already memory-bound, plus $T$ state slots per request that cap how many requests fit. The paper's
kernel measurement on Nemotron-3-Super at batch 256 has vLLM's verification costing 4.66 times a plain
decoding step at $T = 8$.
ReplaySSM gets rollback almost for free because the drafts were never summarised. It appends all $T$
draft records behind the $h$ committed ones, computes all $T$ outputs with the output-only form (one
read of the checkpoint against $T$ queries, $K^{\top} Q$ as one small matrix product, a causal mask over
decays), and after sampling advances the committed pointer by $a$. Rejected records sit past the
pointer and get overwritten. A flush, when it happens, folds only the *committed* records, so it can run
before acceptance is known.
The one subtle part is when to flush. The buffer is a ring of capacity $L + T$, and the rule is to flush
when $h + T > L$: if this step did not flush and all $T$ were accepted, the next step's $T$ records
would overrun committed ones. The rule depends only on $h$ and $T$, so it is known before the step
runs and the decision can live inside a captured CUDA graph. In the vLLM fork the threshold is
`spec_flush_threshold = replayssm_buffer_len + max_spec_len` (`mamba_attn.py` line 123), and the
on-device commit kernel sets the next step's flag with
`next_is_flush = ((new_wp + 2 * MAX_SPEC_LEN) > MAX_CACHE_LEN)` (`selective_state_update_replayssm_spec.py`
line 810). Substitute the threshold and that is $h + T > L$. The June blog post states the rule as
$h + 2T > L$ for a buffer of capacity $L$; it is the same rule with $L$ counting the whole ring.
The widget below runs that bookkeeping. Pick how many tokens are verified per step, press a button to
say how many the verifier accepted, and watch the pointer and the checkpoint. It also counts state
traffic per committed token at the Nemotron-3-Super layer shape, for vLLM's snapshots, for plain
decoding and for ReplaySSM.
Two things are visible after a few presses. With $T = 4$ and every draft accepted, ReplaySSM moves
1.30 MiB of state per committed token, against 5 MiB for the snapshot path and 8 MiB for plain decoding:
four tokens share one checkpoint read, and the write comes once every few steps. With every draft
rejected the ratio is worse but still well under the snapshots: 4.47 MiB against 20, because a
rejected step still committed one token and still wrote no state. The other line is memory per
request. Snapshots hold $T$ states; ReplaySSM holds one checkpoint and $L + T$ records. That second
number is what lets ReplaySSM keep scaling with batch size after vLLM's speculative path runs out of
room. The June post measured that as 3.0 to 3.3x more concurrent requests in the same memory.
ReplaySSM is not the first scheme to cache inputs for SSM speculation, and the paper spends an appendix
being fair about it. The Mamba in the Llama, Snakes and Ladders and STree all avoid per-token snapshots
by replaying the accepted tokens onto a checkpoint. They still write at least one state per step, and
their replay is a serial recurrence that holds the state tile in registers for $T$ or $2T - 1$ steps.
The authors re-tuned each one's launch configuration (which made them 1.4 to 3.0x faster than as
released) and then measured all of them at batch 256. The result I did not expect: at $T \geq 4$
STree and The Mamba in the Llama move 1.6 to 2.9x *fewer* bytes than vLLM's snapshots and are still not
faster. Fewer bytes only help if the kernel can keep enough memory requests in flight, and a serial
recurrence holding a state tile in registers cannot.
## Checking the speed numbers
The post says "up to 1.47x" for standard decoding and "up to 2.1x" for speculative decoding. Both are
in the paper. Neither is the whole story, so here is where each one lives.
Standard decoding first. The paper's Table 6 gives end-to-end decode-step speedup over vLLM with its
best swept launch configuration. At batch 256 it is 1.47x for Nemotron-3-Nano-4B, 1.42x for
Nemotron-3-Super-120B, 1.27x for Qwen3.5-4B and 1.20x for Qwen3.5-122B, which is the 1.20-1.47x in the
abstract. The isolated SSM kernel is 1.46 to 1.80x faster at the same batch (Table 7). The gap between
the two is just Amdahl: attention, the MoE GEMMs and everything else are untouched.
The same tables show where it does not help. At batch 1 the end-to-end speedup is 0.96 to 0.99x, a
slight slowdown, and the kernel alone is 0.41 to 0.88x. At that size the conventional kernel reaches
only 21.6% of peak DRAM bandwidth on an H100 and 9.3% on a B300, so cutting its traffic saves nothing,
while ReplaySSM still does the extra buffer reads and arithmetic. That fits the 0.90x Z.ai reported when
they first added it to their KDA decode kernel (their post gives no batch size, so I cannot say more), and
it does not contradict anything in the paper: the paper says plainly the method pays off only where the baseline is bandwidth-bound, from roughly batch
128 up, where vLLM's kernel saturates at 84-86% of peak. If you run a hybrid model for one user on one
GPU, this is not for you. Two other footnotes are worth knowing. The batch-512 cells of 1.72x and
1.22x on the B300 models are marked as inflated by a 1,100 W power cap that throttled only the baseline.
And against vLLM's shipped Mamba-2 defaults rather than the tuned baseline, the batch-256 numbers would
have been a little better (1.49x rather than 1.47x on the 4B model); the authors chose the harder
comparison.
Now speculative decoding. The paper runs the two 120B-class models with their MTP heads as drafters on
GSM8K prompts, 256 generated tokens each. The headline cell is $T = 4$ at batch 512: 1.85x the
throughput of vLLM's speculative decoding on Nemotron-3-Super and 2.10x on Qwen3.5-122B. That is the
"up to 2.1x". The abstract says "about 1.8x", which matches the batch-256 column of the same grid
(1.77x and 1.79x). And the grid itself goes higher: Nemotron-3-Super at $T = 8$ and batch 512 is
2.34x.
| what the number is | Nemotron-3-Super | Qwen3.5-122B |
|---|---:|---:|
| over vLLM speculative, $T = 4$, batch 256 (the abstract's "about 1.8x") | 1.77x | 1.79x |
| over vLLM speculative, $T = 4$, batch 512 (the post's "up to 2.1x") | 1.85x | 2.10x |
| over vLLM speculative, $T = 8$, batch 512 (largest cell) | 2.34x | no result (out of memory) |
| over plain decoding, $T = 4$, batch 512 | 2.16x | 1.90x |
| over vLLM speculative, $T = 4$, batch 1 | 1.04x | 1.12x |
The row I would quote is the fourth one. A speedup over vLLM's speculative path is partly a measure of
how badly that path does at large batch: on Nemotron-3-Super at batch 256, conventional speculation
peaks at $T = 2$ with 9,551 tokens per second and falls to 5,644 at $T = 8$, *below* plain decoding at
6,807. On Qwen3.5-122B at batch 512 it is below plain decoding at $T = 4$ already. Against plain
decoding, ReplaySSM's speculation is 2.16x and 1.90x, which is the honest "speculative decoding now
works on hybrids at serving batch" number. Acceptance lengths match the baseline within 0.05 tokens in
every cell, so none of it comes from accepting more drafts.
The June blog post, which used Nemotron-3-Ultra-550B as well, reported up to 1.48x for standard decoding and
1.87-1.96x for speculative decoding over plain decoding. Those numbers differ from the October ones because
the setups differ, starting with a buffer of 8 on Nemotron, not because anything got worse.
## The precision argument, derived
The third claim is the one I wanted to check by hand, because "rounding error never propagates" is a
strong sentence.
Start with a single low-precision state used for everything, the way you would naturally quantise an
SSM cache. Let $S_t$ be the exact state and $\hat S_t$ the stored one, rounded after every update:
$\hat S_t = \mathcal{Q}(\alpha_t \hat S_{t-1} + v_t k_t^{\top})$. Call the rounding error of step $t$
$e_t$ and the accumulated deviation $D_t = \hat S_t - S_t$. Subtract the two recurrences:
$$
D_t = \alpha_t D_{t-1} + e_t = \sum_{j \leq t} \gamma_{j+1:t}\, e_j
$$
Every rounding error ever made is still in the state, decayed but never removed, and the output error
is $D_t q_t$. With a constant decay $\alpha$ and independent per-step errors of bias $b$ and variance
$\sigma^2$, the mean deviation settles at $b/(1-\alpha)$ and the variance at $\sigma^2/(1-\alpha^2)$. A
head with long memory, $\alpha$ close to 1, amplifies its rounding error by a large factor, and it takes
on the order of $1/(1-\alpha)$ tokens to get there. Round-to-nearest contributes through the bias term,
which grows linearly; stochastic rounding is unbiased but its variance still accumulates. A KV cache
has no such loop: an FP8 entry is written once and read many times, never fed back into itself. That
is the reason, the post says, KV caches ship in FP8 while SSM states still ship in FP32.
ReplaySSM changes this in two steps. First, with one low-precision checkpoint, rounding now happens once
per window instead of once per step, so the same sum runs over flushes and the limits become
$b/(1-\alpha^L)$ and $\sigma^2/(1-\alpha^{2L})$. Smaller, but the loop is still there.
The second step is the real idea. Non-flush steps only *read* the checkpoint to make outputs; only
flushes *update* it. So keep two copies. The flush starts from a high-precision checkpoint
$S^{\text{HP}}$, folds in the buffer in FP32, and casts the result independently into both formats:
$S^{\text{HP}}_m = \mathcal{Q}_{\text{HP}}(\cdot)$, $S^{\text{LP}}_m = \mathcal{Q}_{\text{LP}}(\cdot)$ from the same
unrounded matrix. Non-flush steps read $S^{\text{LP}}$. The next flush reads $S^{\text{HP}}$, never
$S^{\text{LP}}$.
Take the recurrence copy as exact for a moment, so $S^{\text{HP}}_m = S_m$, and write
$S^{\text{LP}}_m = S_m + \varepsilon_m$, where $\varepsilon_m$ is a single rounding of a clean state. A
non-flush step $h$ positions into the next window outputs
$$
\hat o_t = \gamma_h (S_m + \varepsilon_m)\, q_t + \sum_{j=1}^{h} \gamma_{j+1:h}\, v_j (k_j^{\top} q_t) = o_t + \gamma_h\, \varepsilon_m\, q_t
$$
The error is $\gamma_h \varepsilon_m q_t$. It contains $\varepsilon_m$ and nothing else: no
$\varepsilon_{m-1}$, no sum over history, because the previous low-precision copy was thrown away and
never entered anything that survives. Since $\gamma_h \leq 1$, the error is bounded by one rounding of
the last checkpoint, whatever the sequence length. That is the sense in which the read error does not
propagate, and the derivation holds.
What it does not say is that a low-precision state has no accumulating error at all. Two terms remain,
and the paper names both. The recurrence copy is itself rounded at every flush unless you keep it in
FP32; the paper's preferred configuration keeps it in FP16 with stochastic rounding, which re-enters the
loop at $1/L$ of the per-step rate and at a precision where the loop is mild. And on GDN, the cached
corrections $u$ on non-flush steps are computed from the low-precision read copy, so they carry its
error into the rest of that window's outputs; the flush recomputes them from cached raw $v$ and the
high-precision state, so the error stops there. The emulation kernel shows the separation plainly: on a
flush it quantises `S_new` into `state_hi_ptr` with stochastic rounding and then, under the comment
"Quantize the unrounded solution independently for the read copy", into `state_lo_ptr` with
round-to-nearest (`emulation/mamba2_replay.py`, lines 150-174); a non-flush step loads `state_lo_ptr`
and nothing else (line 176).
I wanted to see the shape of this rather than take it from a plot, so I built a toy: one Mamba-2 head
with an 8 by 32 state, Gaussian inputs, a fresh decay every step, run for 4,096 steps four ways (exact,
rounded every step, one ReplaySSM checkpoint, and separate copies with an exact recurrence). The TypeScript
version runs in your browser below, from a fixed seed, so you see the numbers I quote.
With an FP8 state, stochastic rounding, a window of 16 and a long-memory head (decays between 0.9995 and
0.99999), the relative output error after 4,096 steps is 111% when the state is rounded every step,
37% with one checkpoint rounded once per window, and 2.4% with separate copies. The first two are still
climbing; the third went flat within the first few dozen steps and stays at one rounding's worth. Flip
the head to short memory (decays between 0.9 and 0.95) and the picture changes: everything plateaus
early, at 9.9%, 2.1% and 1.4%. For a head that forgets in a few dozen tokens, rounding once per window
already captures most of the benefit, and the second copy buys a factor of about 1.5. The separation
earns its keep on the slow heads, the ones carrying long-range information, which is exactly where the
recurrence amplifies error. Switching to round-to-nearest shows the other thing the paper reports: per
step, rounding to nearest is smaller than stochastic rounding early on (at around step 40 it is 10.7%
against 15.3% for the every-step FP8 state) and much worse later (506% against 111% at step 4,096),
because its bias adds linearly while stochastic rounding's noise adds in quadrature. The paper uses
stochastic rounding for anything that feeds the recurrence and round-to-nearest for the read copy,
which is rounded once and never fed back.
The toy is a toy. Its inputs are random, not a language model's, and its error is not KL divergence.
The paper measured the real thing: KL divergence of next-token distributions from FP32-state decoding,
under teacher forcing on PG-19 books, out to 128K generated tokens.
0.136 divided by 0.0038 is 35.8, the post's 36x. That is the best of the four models. On the other three
the gap at the longest length is 13 to 29x, and on the two Nemotron models the separate curves still
rise a few times over the range; the paper says so in as many words, "the growth is reduced rather than
removed." The 120B-class models also have a KL floor about two orders of magnitude above the 4B ones,
which the authors show is already there between two FP32 runs that differ only in eager versus CUDA-graph
execution, so every curve there has to be read against its own floor.
Then accuracy. Averaged over the four models, a shared NVFP4 checkpoint loses 13.05 points on
GPQA-Diamond, 5.88 on MMLU-Pro and 10.92 on Math-L5. NVFP4 reads with an FP16 recurrence cut those
losses to 2.36, 0.15 and 0.10 points. The average hides one model: Nemotron-3-Nano-4B with separate
NVFP4 reads and an FP16 recurrence still loses 6.7 points on GPQA-Diamond (against 31.3 with a shared
state). Shared FP16 and FP8 checkpoints stay within benchmark noise of FP32, which means FP8 state
storage was already close to fine; it is 4-bit where the separation decides whether the model works.
The loss lands where the derivation says it should, in long generations. Split by the length of the FP32
reference answer, the shared NVFP4 checkpoint on Nemotron-3-Super matches FP32 in the two shorter
quartiles and falls 20.9 and 28.2 points behind in the two longer ones, ending at 23% against 51% in the
quartile whose median answer is 61K tokens. Separate precisions keep those gaps at 2.0 and 3.4 points.
There is a speed side too. A smaller read copy moves fewer bytes on every non-flush step: NVFP4 reads
make the non-flush Mamba-2 kernel 3.83x faster than an FP32 state at batch 256, and 3.33x over a whole
window at batch 512 once the high-precision flushes are paid for. Those numbers come from separate
CuTe DSL kernels in the repository, not from the serving integration.
## What actually shipped
The post says ReplaySSM "now runs in production in GLM-5.3-Flash and Kimi-K3, and ships in TensorRT-LLM,
vLLM, FlashInfer, and SGLang." I checked each pull request through the GitHub API. The picture is real,
and narrower than that sentence.
| system | what is merged | what is not |
|---|---|---|
| vLLM | Mamba-2 standard decoding, Triton kernel ([#48018](https://github.com/vllm-project/vllm/pull/48018), July 24); a FlashInfer ReplaySSM backend for Mamba-2 ([#52506](https://github.com/vllm-project/vllm/pull/52506)) and its MTP path ([#52928](https://github.com/vllm-project/vllm/pull/52928), September 30) | the authors' Gated DeltaNet standard and speculative PRs and their Triton Mamba-2 speculative PR were closed unmerged |
| FlashInfer | Mamba-2 checkpointing kernels ([#3975](https://github.com/flashinfer-ai/flashinfer/pull/3975)), GDN speculative kernels ([#4081](https://github.com/flashinfer-ai/flashinfer/pull/4081)), prefix-cache materialisation, autotuning, and SM100 CuTe DSL GDN MTP kernels | the GDN "v cache" variant ([#4140](https://github.com/flashinfer-ai/flashinfer/pull/4140)) is open |
| TensorRT-LLM | Mamba-2 MTP checkpointing replay ([#14203](https://github.com/NVIDIA/TensorRT-LLM/pull/14203), June 19) and GDN MTP replay ([#16464](https://github.com/NVIDIA/TensorRT-LLM/pull/16464)) | |
| SGLang | GDN and KDA buffered output-only decode ([#28451](https://github.com/sgl-project/sglang/pull/28451), June 26) and GDN ring spec-verify ([#28695](https://github.com/sgl-project/sglang/pull/28695)), both written by an SGLang contributor rather than the authors | the post lists no SGLang PR |
So for vLLM users the accurate statement is: Mamba-2 models such as Nemotron get it, including MTP
through the FlashInfer backend; GDN models such as Qwen3.5 do not, in upstream vLLM, as of today. The
repository's own README is more careful than the post. It says upstream vLLM has merged the Mamba-2
standard decoding kernel and that "the other three decoding paths, the emulated quantization and the
low-precision CuTe kernels are only available here." That last clause is the important one for the
third claim: the low-precision state is a research result with emulated quantisation and standalone
kernels. I found no two-precision ReplaySSM state cache in any of the four serving engines; a "quantized
ReplaySSM state cache" PR exists only on a fork. (A single low-precision state is a different thing and
does exist: vLLM's own `selective_state_update` kernel in `mamba_ssm.py` has an
`enable_stochastic_rounding` option for an FP16 state, and the ReplaySSM kernel keeps it for its one flush
write.)
The production claims need the same care. GLM-5.3-Flash is a straightforward yes: the paper's
appendix and Z.ai's own infrastructure report both say ReplaySSM was added to its KDA decode kernel. Kimi-K3 is
different. Its technical report (Section 5.4.2) describes caching only the projected inputs of draft
tokens, rebuilding the accepted states on-chip, and writing back "the states of the verified and bonus
tokens", "a design independently proposed in the concurrent work ReplaySSM." That is input caching for
rollback, but it writes a state every verification step, which puts it closer to the replay-then-verify
family above than to ReplaySSM's write-once-per-window design. The paper's own appendix describes Kimi-K3
accurately, as citing ReplaySSM as a concurrent approach. "Runs in production in Kimi-K3" compresses
that into something stronger.
And integration has a cost the paper does not dwell on: prefix caching. A cached prefix needs an exact
state at a block boundary, and a ReplaySSM layer only has an exact state at a flush. The vLLM fork
forces a flush on the step that completes a cache block (the `align_mode` branch next to
`is_flush_cpu` in `mamba_attn.py`, lines 673-682), and FlashInfer merged a separate "materialization for
prefix caching" kernel. In SGLang, an open issue from September 3 reports that `--enable-linear-replayssm`
requires the `no_buffer` radix-cache strategy, which on a RAG-style benchmark with shared 2,048-token
prefixes dropped the prefix hit rate from 88.2% to 65.9% and raised mean time-to-first-token from 226 ms
to 1,063 ms. That is one user's measurement on Qwen3.8-27B, which I could not rerun, but the mechanism
is real. One reply under the post says current implementations "seem to break/corrupt in many settings";
the long list of open ReplaySSM bug-fix PRs in SGLang's tracker (ring cursors not reset after HiCache
load-back, unflushed checkpoints, pending updates lost on CPU retraction) suggests the integration is
still settling, though none of them says the method is wrong.
## What I make of it
The clever part is not the caching. Mamba in the Llama and STree already cached inputs. It is the
observation that a decoder never needs the state of a step that nothing will resume from, so most steps
can produce an output and no state at all. Everything else follows: roughly half the state traffic, rollback by
pointer, verification as a small masked matrix product instead of a serial loop, and a read copy that
can be as coarse as you like because no future step depends on it. The post's own framing is accurate: "Caching SSM inputs instead of the state is what ReplaySSM does.
What it changes is that computing the output and updating the state can be separated during decoding."
The derivation above is all it takes to see why.
What I would use today: if you serve a Mamba-2 hybrid like Nemotron-3 at batches above roughly 32 in vLLM
or TensorRT-LLM, turn it on, and turn on MTP speculation with it, because that combination is the one where
the paper's numbers say speculation finally pays at serving batch. For GDN and KDA models the merged paths
are in SGLang, FlashInfer and TensorRT-LLM; upstream vLLM does not have them yet. Either way, check your
prefix-cache hit rate before and after. If you run one request at a time, leave it off; the paper's own
table says it is a few percent slower there.
What I would wait on: the 4-bit state. The derivation is sound and the accuracy table is convincing on
three of four models, but it is measured with emulated quantisation, the 4B Nemotron still loses 6.7
GPQA points, and no engine ships it. When one does, the thing to measure is long-generation accuracy by
length quartile, the way the paper did, because short benchmarks will tell you nothing.
[Mamba and state-space models](/architectures/mamba-ssm) has the layer itself, and
[liquid time constants and gated delta rules](/articles/ltc-gated-delta) the recurrence family ReplaySSM
unrolls. [KDA's half-life](/articles/kda-half-life) is the per-channel version of the "long memory" head
in the toy above, and [how LLM inference works](/articles/how-llm-inference-works) covers why decode is a
bandwidth problem in the first place. For training a model natively in the same 4-bit format, see
[Nemotron in NVFP4](/articles/nemotron-nvfp4).
## How I checked
I read the paper (arXiv [2610.13778](https://arxiv.org/abs/2610.13778), v1, all 35 pages including the
appendices), the [NVIDIA blog post](https://developer.nvidia.com/blog/replayssm-rethinking-state-storage-for-faster-hybrid-model-decoding/),
the [June Dao AI Lab post](https://dao-lab.ai/blog/2026/replayssm/) and the X thread with its replies. I
found no earlier ReplaySSM paper on arXiv; the "earlier work" the post refers to is the June blog post and
its code. I shallow-cloned [Johnny-Liou/ReplaySSM](https://github.com/Johnny-Liou/ReplaySSM) at commit
`5d3ea59` (September 30), a vLLM fork, and read the Mamba-2 standard and speculative Triton kernels, the
metadata builder that sets flush flags, and the emulation kernels; line numbers above are at that commit.
I did not run any of its code.
Recounted from primary files: Nemotron-3-Super's layer counts and shapes from its `config.json` on
Hugging Face; the 4 MiB state, 8,192-token equivalence and 81,920-token crossover; the 0.581 traffic ratio
and $L^{*} \approx 13$ from the paper's formulas; the 18,944-byte record from the kernel's buffer layout;
the flush rule from the fork's code. Every speedup and accuracy figure above is read from the paper's
Tables 2, 6, 7, 8, 9 and 12 and its text; 36x is my division of 0.136 by 0.0038. Merge status for each
pull request came from the GitHub REST API on October 13; Kimi-K3's design from its report's Section 5.4.2.
The error toy and the widget numbers are mine: a NumPy version first, then the same model in TypeScript
(`components/articles/replayssm/sim.ts`), which is what the widget runs. Its FP8 and FP4 are simplified (a
per-row or per-16 scale kept unrounded), it compares to an exact rather than an FP32 run, and it measures
relative output error rather than KL, so it shows the mechanism, not the paper's magnitudes. I could not
check the production claims beyond what Z.ai's and Moonshot's own documents say, and I could not rerun the
SGLang prefix-cache measurement.
---
# Representation Guidance: a pixel diffusion Transformer that guides itself with its own bottleneck
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/representation-guidance
> date: 2026-10-13
> tags: diffusion, diffusion-transformers, image-generation, flow-matching, representation-learning
The post that brought this paper to me read: "Alibaba dropped Representation Guidance (RG) - end-to-end pixel-space framework that scales plain Transformers via compact internal rep to guide diffusion (way less compute) - keeps standard blocks, no heavy decoders - tops FLUX.1-dev". The only reply under it said there was no proof of concept.
The paper is [arXiv 2610.12978](https://arxiv.org/abs/2610.12978), "Learning Representations to Guide Pixel Diffusion Transformers", by Zihao Pan, Mengting Chen and seven others at Sun Yat-sen University, Alibaba, HKU and SJTU. Its [repository](https://github.com/Pan-Zihao/Representation-Guidance) has no code yet, so I could not run anything. I derived what the method trains the network to do and checked each clause of the post against the paper's own tables.
| Claim in the post | What the paper shows |
|---|---|
| plain Transformer, standard blocks, no heavy decoder | Holds. JiT blocks, split 8/4 into encoder and decoder, a linear bottleneck between. |
| "end-to-end" | Half true. Diffusion runs on raw pixels, but training aligns a layer to a frozen DINOv2, and RG fails without it. |
| "way less compute" | Fewer parameters for a given FID, and 4x fewer **epochs** to match JiT-B. Per training step, RG costs more. |
| "tops FLUX.1-dev" | On GenEval and DPG-Bench, prompt-following benchmarks, at 512x512. The same model without RG already scored higher than FLUX.1-dev there. |
## Pixel diffusion without a VAE
Pixel diffusion skips the VAE and denoises RGB patches directly. [PixelUMM and PixelDense](/articles/pixelumm-pixeldense) covers why that used to be hard, and the [Diffusion Transformer doc](/architectures/diffusion-transformer) the latent baseline. The short version: a 16x16 RGB patch is 768 numbers, and a Transformer can only carry that if it predicts the clean image rather than the noise. That is [JiT](https://arxiv.org/abs/2511.13720)'s result, and RG builds on JiT.
RG writes flow matching with $t = 1$ as clean data, the opposite of the convention in the PixelUMM article:
$$
\mathbf{z}_t = t\,\mathbf{x} + (1-t)\,\boldsymbol{\epsilon}, \qquad \mathbf{v} = \mathbf{x} - \boldsymbol{\epsilon}.
$$
The network outputs a clean-image estimate $\mathbf{x}_\theta(\mathbf{z}_t, t, \mathbf{c})$ for a class label or prompt $\mathbf{c}$. It is converted to a velocity and trained against $\mathbf{v}$:
$$
\mathbf{v}_\theta = \frac{\mathbf{x}_\theta - \mathbf{z}_t}{1-t}, \qquad
\mathcal{L}_{\text{FM}} = \mathbb{E}\,\big\|\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) - \mathbf{v}\big\|_2^2 .
$$
The denominator is clipped at 0.05 and $t$ is drawn from a logit-normal with mean -0.8 and standard deviation 0.8 (reported, Table 9). At sampling, classifier-free guidance (CFG) mixes a conditional and an unconditional prediction:
$$
\mathbf{v}^{w}_\theta = \mathbf{v}_\theta(\mathbf{z}_t, t, \varnothing) + w\,\big(\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{c}) - \mathbf{v}_\theta(\mathbf{z}_t, t, \varnothing)\big).
$$
Keep that shape in mind. RG's whole trick is the same extrapolation, moved from sampling into the training target, with a learned representation in place of $\mathbf{c}$.
## Why REPA fades in pixel models
[REPA](https://arxiv.org/abs/2410.06940) adds a loss that pulls one intermediate layer of the denoiser toward a frozen DINOv2's features of the clean image. In latent diffusion it is a large win; the PixelUMM article walks through the loss.
The paper's first claim is that in a pixel Transformer this win does not last. They apply REPA to JiT-B/16 at layer 4, 6 or 8 and train for 200 epochs (reported, Section 2):
- At the aligned layer, linear probing jumps. REPA lifts JiT-B's layer-4 accuracy to 60.7%. By layer 12 it is back to 30.8%, and local-structure score LDS falls from 0.662 to 0.025 (reported, Appendix A.5).
- REPA-4 converges faster early, but the baseline overtakes it after about 160 epochs and ends better: FID 4.56 against 4.80. Aligning deeper is worse: 5.38 at layer 6 and 8.33 at layer 8 (reported).
Their reading: per-pixel loss is mostly satisfied by local operations, so the later layers overwrite the imposed semantics. A probe finds a good representation mid-network that the output does not depend on.
The second claim is about the obvious fix: give the network a specialised pixel decoder, so the Transformer above it can stay semantic. Table 1 compares four decoders on a common 8-block encoder, same budget, same REPA at layer 3 (reported):
| Decoder | Params (M) | GFLOPs | FID |
|---|---|---|---|
| Vanilla Transformer blocks | 131.32 | 28.42 | 4.165 |
| PixelDiT (pixel attention and FFN) | 203.37 | 47.38 | 4.256 |
| DiP (convolutional U-Net) | 93.60 | 33.72 | 6.089 |
| DeCo (pixel-wise MLP) | 99.52 | 25.80 | 9.167 |
PixelDiT costs 1.55x the parameters and 1.67x the FLOPs of the vanilla decoder for a slightly worse FID (reasoned, from the table). The probes say why: decoders that model cross-patch structure let the encoder stay semantic but spatially flat; decoders that treat patches independently force it to keep spatial detail and lose semantics. The decoder decides what the encoder keeps; it does not make the representation better. Both experiments are base-size, 200 epochs, one run each: enough to motivate a design, not a scaling law.
## What RG changes
RG has two parts, one for each claim.
**A bottleneck that forms the representation.** The model is split, following DDT, into an encoder $E_\theta$ and a decoder $D_\theta$, both made of unmodified JiT blocks. The encoder's output tokens $\mathbf{h} \in \mathbb{R}^{N\times D}$ go through two linear maps:
$$
\mathbf{u} = \mathbf{h}\,\mathbf{W}_{\text{down}} \in \mathbb{R}^{N\times d}, \qquad
\mathbf{r} = \mathbf{u}\,\mathbf{W}_{\text{up}} \in \mathbb{R}^{N\times D}, \qquad d \ll D.
$$
$d$ is 128 for every size; $D$ is 768, 1,024 and 1,152 for B, L and XL (reported, Table 9). The decoder has its own patch embedding, reads the noisy image again, and sees the encoder only through $\mathbf{r}$, which enters every decoder block as a per-token adaLN condition. The class label also reaches the decoder only through $\mathbf{r}$. The paper calls the space of $\mathbf{u}$ an "implicit latent space": compact and spatial like a VAE latent, but recomputed from $\mathbf{z}_t$ at every step and never decoded on its own.
**A guided target that forces the decoder to use it.** With probability 0.1, $\mathbf{r}$ is set to zero, so the same decoder also learns a representation-free, and therefore unconditional, prediction. The difference between the two predictions is the representation's contribution:
$$
\Delta\mathbf{v}_{\text{rep}} = \mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{r}) - \mathbf{v}_\theta(\mathbf{z}_t, t, \varnothing).
$$
RG adds a multiple of it, with the gradient stopped, to the flow-matching target:
$$
\mathbf{v}_{\text{RG}} = \mathbf{v} + (s-1)\,\operatorname{sg}\big(\Delta\mathbf{v}_{\text{rep}}\big), \qquad
\mathcal{L}_{\text{RG}} = \mathbb{E}\,\big\|\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{r}) - \mathbf{v}_{\text{RG}}\big\|_2^2 .
$$
$s \geq 1$ is the RG scale, 1.4 by default; $s = 1$ is plain flow matching. The representation-free branch keeps the plain target $\mathbf{v}$. So does any sample with $t \leq 0.3$, because representations of nearly pure noise are unreliable.
Every reported result computes the offset with the EMA network, as two gradient-free forward passes, and switches RG on after 20 epochs of plain flow matching; starting at epoch 0, 10 or 30 moved FID by under 0.01 (reported, Appendix A.5).
### What the shifted target converges to
Appendix A.2 solves for the fixed point, and I checked the algebra. Write $\bar{\mathbf{v}}_{\mathbf{r}} = \mathbb{E}[\mathbf{v}\mid\mathbf{z}_t, \mathbf{r}]$ and $\bar{\mathbf{v}}_\varnothing = \mathbb{E}[\mathbf{v}\mid\mathbf{z}_t]$. The dropped branch converges to $\bar{\mathbf{v}}_\varnothing$. If the teacher equals the student at convergence, the minimiser of $\mathcal{L}_{\text{RG}}$ satisfies $f = \bar{\mathbf{v}}_{\mathbf{r}} + (s-1)(f - \bar{\mathbf{v}}_\varnothing)$, which rearranges to
$$
\mathbf{v}_\theta(\mathbf{z}_t, t, \mathbf{r}) = \bar{\mathbf{v}}_\varnothing + \frac{1}{2-s}\big(\bar{\mathbf{v}}_{\mathbf{r}} - \bar{\mathbf{v}}_\varnothing\big), \qquad 1 \le s < 2 .
$$
That is a CFG velocity with the representation as the condition and an effective scale $w_{\text{rep}} = 1/(2-s)$. At the default $s = 1.4$ that is 1/0.6, about 1.67 (reasoned). As $s$ approaches 2 the scale blows up, which is why the slider below stops at 1.9.
So the answer to "is RG a training change or a sampling-time guidance?" is: **training only**. The guidance is baked into the weights. Sampling is a normal 50-step Heun solve with normal class-label CFG on top, and RG adds no inference passes. What changes is how much CFG you still need: the best CFG scale for RG-B falls from 3.95 without guidance to 2.60 with it (reported, Table 10). Why not extrapolate at sampling time instead? Appendix A.2: it "would double the decoder cost and leave the representation unchanged". The second half matters more. The gradient of $\mathcal{L}_{\text{RG}}$ flows through $\mathbf{r}$ into the bottleneck and the encoder. To hit a target that sits further from the unconditional answer, the conditioned branch has to deviate more, so $\mathbf{r}$ has to carry more of what denoising needs. CFG guides with a fixed label. RG guides with a condition it is simultaneously learning.
The toy above is three Gaussian modes in 2D, with $\mathbf{r}$ idealised as "which mode". Both branches are then exact, so the grey field is the fixed point above, not a trained network. The black arrow is one pair's plain target, the orange tip the added offset, and 64 noise draws are integrated with 50 Euler steps. Three things show up (measured, in the toy):
- At $s = 1$ the samples land with close to the data's own spread (Euler's 50 steps lose a little). Raising $s$ tightens them around the chosen mode, the sharpening the paper describes as $p(\mathbf{z}_t)\,p(\mathbf{r}\mid\mathbf{z}_t)^{w_{\text{rep}}}$.
- Push $s$ high and the centre drifts away from the other modes and the spread grows again: the samples overshoot the data, the same failure as too much CFG.
- Untick the $t > 0.3$ interval and the drift starts at much lower $s$, because the offset is largest when the mode is still ambiguous. The paper's version of this: guidance at all noise levels raises RG-B's FID from 3.28 to 5.28 (reported, Figure 9b).
What a toy cannot show is the learned half. Here $\mathbf{r}$ is fixed and perfect; in RG it is a 128-wide vector per token, trained by the same loss.
### How it differs from Internal Guidance
Internal Guidance (IG) has a training variant of the same form: it adds $\omega\,\operatorname{sg}(\mathbf{v}_{\text{final}} - \mathbf{v}_{\text{inter}})$, the gap between the output and an intermediate head. On JiT-B in pixel space it made FID worse, 4.56 to 5.22 (reported, Table 10). The paper's explanation: the shallow head's loss is only 4% above the final one's, so the gap is mostly fine pixel detail. RG's offset removes only $\mathbf{r}$, so it isolates what the representation contributes.
## Checking "keeps standard blocks, no heavy decoders"
This one holds. Every block is a JiT block. The B/16 model is 8 encoder blocks and 4 decoder blocks, the same 12 as JiT-B. Against JiT-B/16 at 131.12M parameters and 25.22 GFLOPs, RG-B/16 is 131.52M and 28.47 GFLOPs (reported, Table 8): 0.4M more parameters and 12.9% more FLOPs per forward pass (reasoned). Table 8 splits it: decoupling into encoder and decoder takes JiT-B to 131.32M and 28.42 GFLOPs, and the bottleneck adds the last 0.2M, which is 768 x 128 twice, 196,608 weights (reasoned). So the extra FLOPs are the decoder's second patch embedding and its conditioning, not the bottleneck. The paper rounds this to "13% FLOPs".
Two qualifiers. The text-to-image model swaps JiT blocks for MiniT2I's MM-JiT blocks, which are also standard but not the same architecture. And the configuration carries 32 in-context class tokens, inserted at block 4, 6 or 8 depending on size (reported, Table 9).
## Checking "end-to-end"
Diffusion runs on raw pixels, and nothing external is needed at sampling time. Training is another matter. RG's full loss is $\mathcal{L}_{\text{RG}} + 0.5\,\mathcal{L}_{\text{REPA}}$, aligned to a frozen DINOv2-B/14 with registers at encoder block 3, 5 or 7 for B, L and XL (reported, Table 9). The component ablation shows this is not optional (reported, Table 8, B/16 at 200 epochs):
| REPA | Bottleneck | Guidance | FID |
|---|---|---|---|
| | | yes | 8.08 |
| | yes | yes | 8.99 |
| yes | | yes | 7.85 |
| yes | | | 4.17 |
| yes | yes | | 3.79 |
| yes | yes | yes | 3.28 |
Guidance without both REPA and the bottleneck makes things much worse. With both, it takes 3.79 to 3.28. RG does not replace pretrained alignment; it is what makes alignment survive to the output. The paper's own phrase for DINOv2 here is "a semantic scaffold". Aligning the encoder's last layer directly, so that $\mathbf{h}$ itself matches DINOv2, gives FID 8.97 (reported, Table 6). The representation the decoder consumes has to be learned for generation, but it starts from someone else's.
## Checking "way less compute"
On ImageNet 256x256, RG-XL/16 reaches FID 1.55 with 725M parameters at 600 epochs, the best pixel-space number in Table 2. PixelDiT-XL is next at 1.61 with 797M. RG-L/16 reaches 1.79 at 459M, beside JiT-G/16 at 1.82 with 2B and PixelREPA-H/16 at 1.81 with 953M (reported). That is 4.4x fewer parameters than JiT-G and 2.1x fewer than PixelREPA-H (reasoned). A 0.03 FID gap from one run each is a tie; "matches with a quarter of the parameters" is the honest reading, and still a strong one. At 512x512, RG-XL/16 reaches 1.74 against PixelDiT-XL's 1.81 and JiT-H/32's 1.94 (reported, Table 3). Latent models are still ahead at 256: RAE-XL is at 1.13.
Per **inference** step, then, RG is cheap: the same blocks, about 13% more FLOPs than JiT at the same size, and fewer parameters for a given FID. RG-L/16 is 99.2 GFLOPs and RG-XL/16 155.8 (reported, Table 9). The sampling budget is ordinary: 50 Heun steps, which is about 100 network evaluations, roughly doubled by CFG inside its (0.1, 1] interval (reasoned; the paper does not count NFE). FIDs are only reported with CFG, with the scale swept over 2.0 to 4.0. No CFG-free number is given.
Per **training** step, RG costs more. The headline speed-up is this:
Epochs are not FLOPs. An RG step adds two gradient-free EMA forward passes. Taking a backward pass as about twice a forward, a JiT-B step is about 3 x 25.22 GFLOPs per image and an RG-B step 3 x 28.47 plus 2 x 28.47 (reasoned). If the EMA passes run on every sample, 600 JiT-B epochs cost 2.1x what 150 RG-B epochs do. They are only needed when $t > 0.3$ and $\mathbf{r}$ is kept: 52.4% of logit-normal draws, times 0.9, so 47% (reasoned). Skipping the rest lifts the ratio to 2.7x. Neither counts RG's DINOv2 forward for REPA, which JiT-B does not pay. So the compute-matched speed-up is roughly 2x to 2.7x (reasoned): real, and less than 4x. The paper gives no training FLOPs, wall-clock or GPU-hours to check this against. The figure's "-28%" at 600 epochs checks out: (3.66 - 2.64) / 3.66 is 27.9% (reasoned).
## Checking "tops FLUX.1-dev"
This is the text-to-image result at 512x512 (reported, Table 4):
| Model | Params | GenEval | DPG-Bench |
|---|---|---|---|
| SD3 | 8B | 0.68 | 84.1 |
| FLUX.1-dev | 12B | 0.67 | 83.8 |
| DeCo-XXL/16 | 1.1+1.7B | 0.86 | 81.4 |
| MiniT2I-L/16 | 912+341M | 0.88 | 85.9 |
| RG-T2I-L/16 | 912+341M | 0.89 | 87.1 |
**These are prompt-following benchmarks.** GenEval checks object counts, colours and positions with a detector over 553 prompts; DPG-Bench asks question-answer checks over 1,065 dense prompts. Neither measures image quality, and the paper reports no text-to-image FID or preference study. "Tops FLUX.1-dev" means "follows these prompts more literally", not "makes better images".
**RG's own contribution is small.** RG-T2I-L/16 is MiniT2I-L/16 with RG added, same size, same data, same recipe. MiniT2I-L/16 alone already scores 0.88 and 85.9, above FLUX.1-dev on both. RG adds 0.01 GenEval and 1.2 DPG (reported). At 553 prompts and 4 images each, a binomial standard error on a GenEval-like score near 0.89 is about 0.007 (reasoned; GenEval averages per-category scores, so this is approximate). A 0.01 gain from one run is about 1.5 standard errors. The DPG gain is probably more solid, but the paper gives no seeds. (MiniT2I-L is also the model [BudgetPix](/articles/budgetpix) adapts to run at any token budget; the two papers attack the same baseline from opposite ends, its training signal and its compute.)
**The parameter column mixes conventions.** The caption says counts include the text encoder. RG's 912+341M does: 341M is FLAN-T5-Large. FLUX.1-dev's 12B is its transformer only; it also runs a T5-XXL text encoder that is not in the 12B. The ratio is still about 10x, but the column is not like for like.
**The data differ.** RG-T2I trains on CC12M with LLaVA-NeXT captions, then fine-tunes on 120K images from BLIP3o-60K, DALL-E 3 and ShareGPT-4o-Image (reported). That fine-tuning mixture is curated, largely synthetic data made for prompt alignment, and I would expect it to lift exactly these two benchmarks (reasoned; the paper does not test it). FLUX.1-dev's training data is not public. A small model tuned for prompt alignment beating a general 12B model on prompt alignment is useful to know; it is not evidence that RG out-generates FLUX.
## Representation quality against generative usefulness
The paper measures representations with three probes on B/16 models at 200 epochs, and no CKNNA: **linear probing** for global semantics, **RMSC** (spread of normalised patch tokens) for how distinct positions are, and **LDS** (neighbour minus distant-pair similarity) for local structure (reported, Appendix A.4).
The numbers support the thesis better than the pictures do (reported, Appendix A.5):
- JiT-B's probing peaks at 29.5% at layer 8. RG-B's encoder holds 52.1% to 55.2% from layer 3 to layer 8, then drops to 22.7% to 29.2% in the decoder. 55.2 over 29.5 is 1.87x, the paper's "nearly twice" (reasoned).
- At the last encoder layer, RG-B has RMSC 0.858 and LDS 0.390 against JiT-B's 0.476 and 0.111.
- JiT-B with REPA probes higher than RG at its peak, 60.7% at layer 4, but has the worse FID: 4.80 against 3.28.
That last line is the paper's best evidence: peak alignment predicted generation poorly, survival to the layer that uses it predicted it well. With three or four models and one seed each, it is a consistent pattern, not a measured correlation.
## What is released
As of 2026-10-13 (measured):
- The [repository](https://github.com/Pan-Zihao/Representation-Guidance) at commit `65b573f` (30 September 2026) holds `README.md`, `assets/teaser.jpg` and `assets/pipeline.png`. No code, no config, no licence file. Its arXiv badge still reads "Coming Soon". Its TODO lists ImageNet and text-to-image training code and weights, all unchecked, "in the coming weeks".
- Hugging Face has no models: searching `representation-guidance` and listing author `Pan-Zihao` both return an empty list.
None of it can be reproduced yet. The README's results table matches Tables 2, 3, 4 and 9 to the digit, and adds two numbers the paper's text does not state: RG-B at 100 epochs, 4.45, and RG-L at 200 epochs, 2.00.
## Things that do not line up
- **The EMA assumption.** The fixed-point derivation assumes the EMA teacher matches the online network. The appendix then finds the online teacher better, 3.18 against 3.28 (reported, single runs), and explains it: at decay 0.9999 the EMA averages about 10,000 steps, roughly eight epochs, so its offset describes an older $\mathbf{r}$. That is the assumption failing in practice, and the paper still uses EMA for every headline number.
- **The 512x512 table** gives no parameter counts or epochs, and the text cites "Table 7" for bottleneck width (it is Table 5) and "Algorithm 10" (it is Algorithm 1). Cosmetic, but it is a v1.
- **Bottleneck width** was chosen at 600 epochs, where $d = 128$ beat 512 (2.64 against 2.89), while at 200 epochs 512 was best (3.18). It is one sweep on B/16; whether 128 is right for XL is not shown.
## How it relates to PixelDense
[PixelDense](/articles/pixelumm-pixeldense) starts from the same observation, that per-pixel loss alone does not build a representation, and goes the other way: **more** frozen teachers, four of them, in two orthogonal streams. RG keeps one teacher as a scaffold and makes the network **depend** on what it learns. The two compose; neither paper tries it.
Related reading on this site: [CSFM](/articles/csfm-flow-matching) learns the other end of flow matching, the source. [EVA](/articles/eva-empirical-vae) keeps a latent and learns its prior. [MrFlow](/articles/mrflow-diffusion-acceleration) cuts pixel-space sampling compute, and [LoT Diffusion](/articles/lot-diffusion) changes how many tokens a DiT spends where.
## What I would want next
- **The code.** The method is a few lines on top of JiT. Until it ships, every number above is reported.
- **GPU-hours** for RG against JiT at matched FID, since the 4x is in epochs.
- **Text-to-image quality** beyond GenEval and DPG, and RG-T2I against MiniT2I with seeds.
## Sources
- Paper: [arXiv 2610.12978](https://arxiv.org/abs/2610.12978) (v1, 9 October 2026), read from the arXiv HTML · [project page](https://pan-zihao.github.io/Representation-Guidance/) · repository `Pan-Zihao/Representation-Guidance`, commit `65b573f`, README and assets only
- The post: [@wildmindai on X](https://x.com/wildmindai/status/2109988271275258078)
- JiT: [arXiv 2511.13720](https://arxiv.org/abs/2511.13720) · REPA: [arXiv 2410.06940](https://arxiv.org/abs/2410.06940)
---
# Streaming Code World Model: the proxy can arrive live now, the pixels still can't
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/streaming-code-world-model
> date: 2026-10-13
> tags: world-models, video-generation, diffusion, distillation, kv-cache, realtime, lora
Yesterday I took apart [StreamRender-H3](/articles/streamrender-h3), NVIDIA's racing game in which code runs the world and a 33B video model paints it. A day later Tingting Liao [posted](https://x.com/tingtingliao000/status/2109616784928358657) the research-lab version of the same bet: "We made Code World Model causal: video is generated chunk by chunk from a live proxy. A 4-step distilled model renders each chunk in ~5 s on one H200." Code and two checkpoints are open.
Two words in that post made me want to read the code. "Live" and "~5 s" do not obviously belong in the same sentence. A chunk here is 17 frames, which plays for 0.708 seconds at 24 fps, so five seconds to render one is about seven times slower than the video it produces. I wanted to know what "live" means in that case, and what it took to make a model that was trained on whole five-second windows work one chunk at a time.
The short answer is that "live" describes the model's information flow, not its speed. Every chunk is generated from the proxy up to that chunk and nothing after it, and that property is built with real care. The README is honest that the speed is not there yet: its highlights say "5.1 s per chunk on one H200 (not yet real time)". What the README doesn't spell out is what that gap does to a live loop, or what a few of the cleverer choices in the sampler cost, and most of this piece is about those two things.
## What Code World Model was
The base is [Code World Model](https://arxiv.org/abs/2608.25927) (CWM), from Yiwen Chen, Guosheng Lin and Chi Zhang at Westlake and NTU, posted in August with [code](https://github.com/buaacyw/code-world-model) and a LoRA. Not to be confused with Meta's CWM, a 32B code-generation language model with the same name and nothing else in common.
The paper's argument is that video world models learn the wrong thing from pixels. They see outcomes, not the rules that produce them, so a door opened in one clip is shut again thirty seconds later. CWM splits the job. A coding agent is the "world brain": it reasons about what should happen and writes code that keeps the world state (positions, health, inventories, timers) in ordinary data structures, where a door stays open because a boolean says so. A small compiler turns that state into a proxy video, and a video model renders the proxy into photoreal frames.
The video model is [MiniMax H3](/articles/minimax-h3), the 33B omni-modal generator, in its Ref2VA mode: a reference picture, a reference video and a caption go in, a 124-frame clip at 1344x768 comes out. CWM fine-tunes it with a rank-128 LoRA on all 50 blocks, about 596M trainable parameters by the paper's count, on 5.6 hours of GTA V gameplay paired with proxies extracted from the game's own runtime.
The proxy is the interface that everything in the streaming work is built around, so it is worth knowing exactly what it is. In the released inference code (`src/cwm_h3_inference/duv.py`) every proxy frame is two files at 336x192: a float32 depth map in metres and an 8-bit semantic-ID map with 12 classes (void, sky, water, terrain, paved road, vegetation, building, infrastructure, human, animal, vehicle, prop). The loader packs them into a three-channel image it calls DUV. Depth is clipped to 0.3 to 256 m and log-encoded into 16 bits. The class ID is split across the other two channels: `class % 4` picks one of four U values and `class // 4` one of three V values. The result looks enough like an image that H3's own video VAE can encode it, and it does, into a 12x21 latent grid per latent frame (padded to 22), a sixteenth of the 48x84 grid of the target video.
The paper's conclusion has the sentence this whole repository answers: "We also do not implement autoregressive real-time generation." The released sampler couldn't, for three reasons I found in its `INFERENCE.md` and constants. It denoises a 124-frame window all at once, bidirectionally, so frame 0 of a window attends to proxy frame 123, about 5.1 seconds in the future. Long videos are windows overlapping by 34 frames, each advancing by 90. And each window's caption carries an "action timeline", a text description of what the protagonist is about to do. A game can't hand over five seconds of proxy it hasn't computed yet, let alone a script of the player's next moves.
## A chunk is one VAE clip, and that choice does a lot of work
The first design decision is the size of a chunk, and it is the right one. H3's video VAE encodes independent 17-frame clips into 5 latents each. So `rtcwm/model/layout.py:17` makes a chunk exactly one such clip: 5 latents, 17 frames, 0.708 seconds. Because the VAE never mixes clips, encoding chunk k's proxy or decoding chunk k's video never needs a frame from chunk k+1. Any other chunk size would have needed a causal VAE, or a lookahead, or a hack at every boundary.
Each chunk is generated inside an attention span of itself and the five chunks before it (`SPAN_CHUNKS = 6`, line 19), 30 latents or 4.25 seconds. The span is packed as a standalone H3 sequence, in the same layout the released Ref2VA model was trained on: caption tokens, the start image, the six proxy chunks, an audio track (silent, but H3 insists on one), then the video chunks.
Positions are where a naive streaming port goes wrong. H3 uses rotary embeddings on a time axis, and the released model has only ever seen positions inside one 124-frame window. If chunk 80 sat at time position 80 chunks, it would be far outside anything the model knows. So positions are counted from the span's first chunk. Chunk 80 is generated at position 5, like chunk 5 was.
That creates a cache problem. The keys of chunk 79 were computed when it sat at position 5; now it needs to sit at position 4. The sampler doesn't recompute them. It rotates them. `_shift_time` in `rtcwm/model/sampler.py:209` has a one-line docstring that carries the idea: "RoPE rotations compose, so rotating the keys of position p by delta gives exactly the keys of position p + delta (values carry no position)." When the span slides by one chunk, the oldest chunk is dropped and every surviving chunk's keys get rotated back one chunk's worth of time. By default the rotation is not even stored: `ROTATE_IN_ATTENTION = True` keeps one copy of each cached key and applies the pending shift inside the attention call.
Then the mask. This is the whole of the causality, and it is short enough to quote (`rtcwm/model/masks.py:35-39`):
```python
if qr == STATIC:
return kr == STATIC or (static_sees_first and kc == 0 and kr in (DUV, VIDEO))
if qr == DUV:
return kr == STATIC or (kr == DUV and kc <= qc)
return kr == STATIC or (kr == DUV and kc <= qc) or (kr == VIDEO and 0 <= qc - kc < SPAN_CHUNKS)
```
Read it as three rules. The static rows (caption, start image, audio) see each other. A proxy chunk sees the static rows and proxy chunks up to itself. A video chunk sees the static rows, proxy chunks up to its own index, and the video chunks of its span. The widget below draws that rule for whichever chunk you pick, next to the released model's dense window.
There is one exception in the mask that I would have missed without the docstring. In the first span of a stream, the static rows also see chunk 0 of the proxy and the video, because that is how the released model's conditions behave: with no history, chunk 0 needs condition rows that look at it. Once there is a history, the conditions see only themselves. The comment says that does better, and I have no reason to doubt it, though there is no ablation to check.
The text side was made causal too, which is easy to forget. In CWM, the Qwen3-VL encoder that turns the caption into conditioning also looks at proxy frames, every 12th frame of the window. In the streaming version (`rtcwm/train/preencode.py:251`) Qwen sees the start image and the start frame's proxy repeated, "nothing of later frames", and the caption is cut to its "Appearance and scene" section, without the timeline. So the text describes who is in the scene and what it looks like. Everything that moves comes from the proxy.
A consequence of the 6-chunk span is that the model's visual memory is short. A chunk sees 3.54 seconds of earlier video. Anything that left the frame before that is gone from its view, and only the start image, the caption and the proxy know about it. I don't think that's a flaw in this design. It is CWM's thesis showing up in the architecture: persistence lives in the code's world state, and the renderer only has to be consistent for a few seconds at a time. Whether that holds up when a character walks back to a building it saw a minute ago is exactly what I'd want to see tested, and the repo doesn't test it.
## The data: CARLA, made deliberately worse
To fine-tune for streaming you need long videos with proxies, and GTA V clips are not something you can regenerate at will. The repo renders its own in [CARLA](https://carla.org): a pedestrian walking a route through a town, followed by a third-person camera, 934 frames per render. After quality control, `data_lists/clips.json` keeps 342 clips and drops 1, which at 934 frames and 24 fps is 3.7 hours. Another 201 caption windows across 90 clips are flagged as bad and excluded from training spans.
The interesting decision is the proxy. CARLA can give you perfect depth and segmentation of the real meshes, and the repo throws that away. `rtcwm/data/proxy.py` rebuilds a coarse proxy from the scene's description using the primitives and ray tracer of open-code-world-model, a separate proxy compiler by Guangyi Liu: buildings become their level bounding boxes trimmed clear of the walking path, poles become capsules, vegetation becomes lobed clumps, roads become flat strips from lane waypoints, and the protagonist is a generic human rig with "position, heading and a procedural gait only -- no body shape or clothing". Its docstring gives the reason in one line: "At test time a coding agent only has coarse geometry, which is what CWM's proxy is."
I like this a lot. Training on the exact segmentation would have produced a model that looks great on CARLA and falls apart when an agent's boxy, approximate world shows up. Training on the proxy you will actually get at inference is the unglamorous choice that makes the open-world results in the video above possible at all.
## Teaching it not to trust its own past
A causal video model trained on clean history learns to copy that history faithfully. At inference the history is its own output, slightly wrong, and copying it faithfully means compounding the error. Exposure bias, in other words, and the repo attacks it twice.
The first pass is diffusion forcing. During training every chunk of a span gets its own independent noise level, so the model routinely predicts chunk 4 from a chunk 3 that is half noise. The noise levels come from `df_levels` (`rtcwm/model/objectives.py:66`), drawn uniformly and flow-shifted with shift 3, and its docstring records a failed experiment: "History chunks are never the clean ground truth: a clean ground-truth history teaches the model to trust its history completely, which it must not do with its own (imperfect) history at inference (v3 drew one with probability 0.5 and drifted more)." A useful negative result, written down where the next person will find it.
The diffusion-forcing run starts from the released CWM LoRA and trains for 5 epochs; the README budgets 8 nodes for about 7 hours. The released `streaming_df.safetensors` carries `step: 1118` in its metadata. Sampled with 35 Euler steps, it is the slow model in the results table.
## Four steps, and a history that stays noisy
The second pass is distillation, and it follows CausVid and Self Forcing closely. `rtcwm/train/distill.py` stacks three LoRAs on one frozen H3: a student (initialised from the diffusion-forcing model), a frozen teacher, and a critic. Each iteration the student streams 8 to 14 chunks on its own output, the way it will at inference, with each chunk stopping at a randomly chosen one of its four steps. The last four chunks of the teacher's window keep their gradient graph. The teacher scores where that rollout should go, the critic learns where the student actually goes, and the student is pushed along the difference, which is the DMD gradient (`distill.py:175-178`):
```python
p = x.detach() - x0_real
grad = (x0_fake - x0_real) / p.abs().mean().clamp_min(1e-6)
stack.use("student")
return 0.5 * torch.nn.functional.mse_loss(x.float(), (x.detach() - grad).float()), ...
```
The critic gets 7 updates for every student update, all on windows cut from the same detached rollout, because the rollout is the expensive part. There are two choices of teacher. With `--teacher-kind released` it is the original CWM, run bidirectionally on a 37-latent window, exactly as released. With `--teacher-kind causal` it is the streaming diffusion-forcing model itself, on the student's own six-chunk span. The README's training command uses the causal teacher, initialises the student from step 800 and the teacher and critic from step 1118, and runs `--iters 100`. The released `streaming_dmd_4step.safetensors` says `iter: 100` and `objective: dmd-4step` in its header. A hundred iterations is a short distillation; it is also enough to beat the 35-step model on the metric they report, which tells you how far the 35-step model had drifted.
The student samples with the "x0" rule: each step predicts the clean chunk, then re-noises it to the next level with fresh noise. At shift 5, four steps visit noise levels 1, 0.9375, 0.833 and 0.625 before the final clean prediction, and `distill.py:64` has the same schedule in a comment. The widget lets you move the shift and the step count.
The detail I didn't expect is what goes into the cache. A finished chunk has to enter the KV cache so later chunks can attend to it. The obvious way is one more forward pass over the clean chunk. The sampler's default, `HISTORY_FROM_LAST_STEP = True` (`sampler.py:166`), skips that pass and caches the keys and values the chunk's rows already had in its last denoising step. That step's input was at noise level 0.625. So every chunk a later chunk looks back at is, in effect, a fairly noisy version of itself.
That saves one of five transformer passes per chunk, and the comment claims a second benefit: "the noisy history damps the drift of a long rollout". It fits the diffusion-forcing story, where the model was trained to read noisy history with suspicion. The distillation rollout is set up to match (`--rollout-history last-step` noises the history to the same level), so the student is trained on the history it will actually see. I'd still like the comparison. `--no-history-from-last-step` exists for exactly that, and the README doesn't report it.
The other speed trick, `--kv-reuse`, is also on in the README's 4-step command. Without it, the sampler re-runs the whole history of the span every chunk, which is exact. With it, history keys and values are kept from when they were first computed and only rotated. The module docstring is upfront that this is "an approximation of the exact default": those keys were computed in an earlier span, against a different window of history and conditions. `tests/test_kv_reuse.py` checks that rotating in attention matches rotating a copy, to within 0.02; there is no test of reuse against the exact re-run. The headline 4-step numbers are all with reuse on.
## What 5.1 seconds buys
The README's results are four held-out CARLA clips, 24 chunks each (17 seconds), one seed, scored by pixel MSE against the ground-truth render:
| model | steps | mean MSE | 0-4 s | 13-17 s | s / chunk |
|:---|:---:|---:|---:|---:|---:|
| streaming diffusion forcing | 35 | 2231 | 1335 | 3332 | 47.9 |
| + DMD, released CWM as teacher | 4 | 1741 | 1049 | 2501 | 5.2 |
| + DMD, streaming model as teacher | 4 | 1660 | 1070 | 2555 | 5.1 |
The 4-step student is 9.4 times faster than the 35-step model (47.9 / 5.1), which the README rounds to "9×", and its mean MSE is about 26% lower. Distillation improving quality is the usual Self Forcing result: the student was trained on its own rollouts, the teacher never was.
Look at the drift, though. From the first four seconds to the last four, the 35-step model's error grows 2.5 times (1335 to 3332). The best 4-step model's grows 2.39 times (1070 to 2555). Distillation lowered the whole curve without flattening it much. And between the two teachers the order flips by time: the released-CWM teacher is better in both the first and last windows, the streaming teacher wins on the mean. With one seed and four clips, I wouldn't read anything into an 81-point gap.
I'd also be careful with pixel MSE itself. A mean of 1660 on 0-255 pixels is an RMS error of about 41 levels per channel. Against a CARLA render, a model that blurs every texture scores better than one that invents a sharp, plausible, different texture, so MSE rewards a smoother sampler. It does catch the failure that matters most here, drift away from the scene, and nothing else in the repo measures quality.
Now the clock. One chunk plays for 17/24 = 0.708 seconds. Rendering it takes 5.1, so the model produces video 7.2 times slower than real time, or about 8.3 seconds of video per wall-clock minute. The 60-second open-world streams in the post are 85 chunks, about 7.2 minutes of H200 time each.
And the 5.1 seconds is not the whole bill. In `rtcwm/eval/evaluate.py`, the per-chunk seconds come from the sampler's `on_chunk` callback, which brackets loading the span's inputs and running the transformer. The VAE decode happens after the whole stream is generated (`evaluate.py:95`), once, outside the timing. A live system would have to decode each chunk as it lands, so the real per-chunk cost is 5.1 seconds plus a decode that nobody has measured here.
To see what that means for a game, imagine one that produces proxy frames at 24 fps and hands each completed 17-frame chunk to one GPU. The first chunk can't start until its last proxy frame exists, 16/24 seconds in; its pixels exist 5.1 seconds after that, 5.77 seconds after the player did something. Every later chunk waits behind the one before, so the lag grows by 5.1 − 0.708 = 4.39 seconds per chunk. After a minute of play, the newest chunk on screen is more than six minutes old.
So the honest description is "causal, ready for a live proxy once the renderer is 7.2 times faster". Or a game whose clock runs at a seventh of real time, which for some uses (a turn-based world, a cutscene generator, a dataset factory) is fine. The README's own "(not yet real time)" is the right summary. The X post's "live proxy" is accurate about information flow and easy to misread about latency.
For scale, StreamRender-H3, the same backbone post-trained by NVIDIA, reports about 273 ms per chunk on eight GB200s with two denoising steps at 832x480. The comparison isn't like for like: different resolution, eight GPUs against one, a semantic map instead of depth plus semantics. It does show where the remaining factor of seven is likely to come from: fewer steps, lower resolution, more GPUs, and a sparser attention.
## Memory: why this needs an H200
The README states 123 GB peak in bf16 and 108 GB with `--fp8`, and that "80 GB GPUs are not supported yet". That's a step backwards from CWM, which was tested on one 80 GB H800, but there it swapped 48 of the 50 blocks to host memory (`blocks_to_swap: 48` in its tested settings), which is fine when you're rendering offline and fatal when you're streaming.
The checkpoint headers explain most of the rest. I read both safetensors headers by HTTP range request: each file holds 600 BF16 tensors, rank-128 LoRA pairs on 200 linears (the QKV, output and two MLP projections of 50 blocks), 596,377,800 parameters, 1.19 GB. CWM's LoRA has the same shape, so either file is a drop-in replacement on the same base. The QKV projection's output is 21,504 wide, three times 7,168, so each block's keys and values are 7,168 wide.
That gives the cache size. A chunk is 5 latents of 1,008 packed rows each, 5,040 rows. Keys and values for 50 blocks in bf16 come to 2 × 50 × 5,040 × 7,168 × 2 bytes, about 7.23 GB per chunk. Five chunks of history is 36.1 GB. Add the H3 transformer's weights, about 66 GB in bf16 for 33B parameters, and you are past 100 GB before activations, the condition rows, or the VAE. The 123 GB peak is about what the shapes predict. Getting this onto an 80 GB card means shrinking the cache (shorter span, quantized keys) as much as the weights.
## Running it, and what it cost to train
This is a research release, not a package. You need the CWM release with its patched musubi-tuner, the MiniMax H3 weights, FlashAttention-3, an H200, and Slurm: every entry point is an `sbatch` script that assumes 8-GPU nodes. To regenerate data you also need CARLA 0.9.15, the open-code-world-model compiler and Qwen3-VL-32B for captions. The sampling commands from the README are short:
```bash
hf download Luffuly/streaming-code-world-model --local-dir ckpt
EVAL_ARGS="--steps 4 --rule x0 --shift 5 --kv-reuse" \
sbatch scripts/eval/eval.sbatch ckpt/streaming_dmd_4step.safetensors out/ clip_0048 clip_0244
```
Licensing is the part to check before you build on it. The streaming repository has no licence file and the Hugging Face card declares none, so by default nothing is granted. CWM's own code is Apache-2.0. And the base model's licence is MiniMax's, which this site covered in the [MiniMax H3 teardown](/articles/minimax-h3): it excludes the US, UK, EU and South Korea.
Someone in the replies asked how many GPU hours training took. The README gives enough to estimate it: diffusion forcing on 8 nodes for about 7 hours is about 448 GPU-hours, and distillation on 4 nodes for about 5 hours is about 160 more, so roughly 600 H200-class GPU-hours for the two stages, before data rendering and pre-encoding. One inconsistency: `scripts/train/distill.sbatch` asks for 8 nodes, not the 4 the README mentions, so the distillation figure could be double.
## What I'd take from it
The causal machinery is the reusable part. If you want to turn any windowed, bidirectional video model with rotary time positions into a streaming one, this repo is a compact, well-commented reference for the parts that are easy to get wrong: chunk on the VAE's own clip boundary, keep positions inside the trained range by counting from the span start, move cached keys with a rotation instead of recomputing them, make the text encoder causal too, and never train on clean history. The `rtcwm` package is about 3,200 lines of Python, and its docstrings explain decisions rather than restating code.
The world-model claim is still mostly CWM's, not this repo's. Streaming makes the code-plus-renderer split usable in a loop, in principle. Whether the coding agent can keep a world consistent for minutes, and whether a 3.54-second visual memory is enough once it does, needs a test where the camera comes back to something it saw a minute ago. The open-world clips don't do that, and the CARLA metric can't.
If you are comparing approaches, the useful neighbours on this site are [StreamRender-H3](/articles/streamrender-h3) for the same backbone made fast with a simpler control signal, [FastH3](/articles/fastvideo-fasth3) for a 4-step H3 distillation aimed at speed, [LongLive-Plug](/articles/longlive-plug) for distilled few-step adapters that transfer across H3 fine-tunes, and [ABot-World-0](/articles/abot-world) for a chunked autoregressive world model built for efficiency from the start.
## How I checked
I cloned `TingtingLiao/streaming-code-world-model` at commit `64c489a` and `buaacyw/code-world-model` at `17fa82b` with `git clone --depth 1` and read both; every file:line above refers to those commits. I did not run any of the code. I read the Code World Model paper on arXiv (2608.25927v1) and took its figures from the arXiv HTML. The `cwm-proxy` (open-code-world-model) repository asked for authentication, so I couldn't read it; what I say about it comes from `rtcwm/data/proxy.py`, which imports it.
The checkpoint facts come from the two safetensors headers on Hugging Face, read with HTTP range requests (no weights downloaded): tensor count, dtypes, shapes, LoRA rank and the metadata fields `step: 1118`, `iter: 100` and `objective`. The KV-cache size is my arithmetic from those shapes and `layout.py`'s 1,008 rows per latent frame; it assumes keys and values are cached in bf16, which is what the sampler stores. The H3 weight size is the 33B figure from this site's earlier teardowns.
The MSE, timing and memory numbers are the README's and the model card's, and I couldn't reproduce any of them without an H200. The real-time factor, drift ratios, lag model and GPU-hour estimate are my arithmetic on those numbers. The lag model is deliberately simple: one GPU, chunks back to back, no decode time, no frame dropping.
---
# SubQ Snap: what 'constant compute per token' has to mean, checked against what Subquadratic published
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/subq-snap
> date: 2026-10-13
> tags: sparse-attention, long-context, benchmarks, ecosystem, agents, multimodal
[Alex Whedon's launch post](https://x.com/alex_whedon/status/2109676361371824229) for SubQ Snap has a list of numbers. One line in it is a different kind of claim: "Snap is the only decision model or LLM where compute per token stays constant as context grows."
That line is about architecture. A transformer with dense attention cannot satisfy it, because every new token compares itself against every token before it, so the work per token grows with the context. Something has to give for the cost per token to stay flat, and there are only a few things that can. I wanted to know which one Subquadratic picked, and whether the rest of the list (384K tokens, 61.7 on the Decision Index, 13 of 16 enterprise benchmarks, under a second for 100K tokens) holds up when you go to the files behind it.
The short answer: the mechanism is not published, but Subquadratic's own technical report gives away its shape, and once you know the shape you know which of Snap's numbers can say anything about it. Most of them can't.
## What a decision model pays for
A decision model takes some context and a question with a fixed set of allowed answers, and returns a probability for each answer instead of generating text. The site has covered this category at length, starting with [TypeSafe's Jev](/articles/jev-system-one-models), and [the limits it inherits from that design](/articles/what-decision-models-cannot-do).
Snap speaks Jev's dialect exactly. Its [API docs](https://docs.subq.ai/markdown/endpoints/decisions-api.md) put it at `POST /v1/systemone`, the same path [llama.cpp adopted](/articles/llamacpp-decision-models) for Jev-style models, with the same three question types: a `noul` (yes/no probability), a `choice` over 2 to 255 named options, and a `score` over 2 to 10 ordered levels. A request "must fit the model's 384K-token (393,216) context window". It takes up to 4 images, at about 64 to 256 tokens each. The response's `usage` block always reports `output_tokens: 0`.
That last detail explains "Output is free." A decision head reads a probability distribution off the final hidden states; there are no output tokens to bill. The phrase is true, and it would be true of every model in this category.
It also means the whole cost of a call is prefill: reading the input. Subquadratic's [launch blog](https://subq.ai/introducing-subq-snap) says so plainly, "For a decision model, nearly all inference compute is prefill." Take away generation and what is left is attention over a long input, which is exactly the part that grows quadratically. So if you want a decision model to read a whole repository, the attention mechanism is the only thing worth changing. As product theses go, I think this one is right.
## Three ways to keep per-token cost flat
For the work per token to stop growing with context, each token has to read something of fixed size. There are three families that manage it.
The first is a recurrent state. Mamba, RWKV and linear attention fold the past into a fixed-size state and read only that. Per-token cost is constant by construction. The price is that the state is a compression, and a specific fact from 300K tokens ago may not survive it. The site's [Mamba explainer](/architectures/mamba-ssm) and [roundup of linear attention's memory problem](/articles/linear-attention-state-roundup) cover this trade.
The second is a hybrid: keep a few full-attention layers among recurrent ones. Qwen3.8-27B, the base of two strong open decision models ([pplx-decider v1.1](/articles/pplx-decider-v1-1) and Cloudflare's clef), is built this way: its config has `full_attention_interval: 4`, so 16 of its 64 layers are full attention and 48 are linear attention. That cuts the bill by a constant factor, but the 16 full layers still grow with context, so per-token cost is not flat.
The third is sparse attention with a fixed budget. Every query still does exact softmax attention, but only over a fixed number of positions it selects from the whole context. If the number is fixed, the read is constant per token. [MiniMax Sparse Attention](/articles/minimax-sparse-attention) does this with 2,048 tokens per query; DeepSeek's DSA does it with 2,048 as well.
SSA is the third kind. The [SubQ-1.1-Small technical report](https://subq.ai/docs/subq-1-1-small-model-card.pdf) (22 pages, Saul Ramirez, Alex Whedon, Ashmal Vayani and Phong Vo) says "SSA is sparse, each query attends to a small, selected subset of positions", and it spends a section explaining why recurrent models fall short at exact retrieval. So no state, no hybrid: content-dependent top-k-style selection.
The catch with the third family is the selector. To pick the best positions, something has to look at the candidates, and if it looks at all of them, every query does work proportional to n again. That is the report's critique of DSA, and its Table 2 makes it concrete for a V3.2-style configuration: the Lightning Indexer costs the same as the sparse attention it feeds at about 52K tokens, 16.1 times as much at 1M, and 190.4 times at 12M. The selection is quadratic even when the attention is not. (The report's background section also calls Native Sparse Attention "also known as DeepSeek Sparse Attention". They are different DeepSeek mechanisms: NSA's paper selects compressed blocks, DSA is V3.2's Lightning Indexer. The comparison in Table 2 is with DSA, which is the right one.)
SSA's claim is that its "selection, retrieval, and attention steps are each linear in sequence length, so the mechanism is linear end-to-end." For the total to be linear, choosing positions for one query must cost something that does not grow with n: hashing into buckets, clustering keys into routes, or descending a hierarchy of summaries would each do it (the last at a log factor). Which of these SSA uses, the report declines to say: "The mechanism by which SSA meets these requirements is outside the scope of this report."
## The budget is about 16K positions
The report does not describe SSA, but it does publish how SSA's cost scales, in three places. Read together, they point to one number.
Table 1 gives attention FLOPs per layer for SSA and for dense attention on the same backbone. The SSA column doubles exactly with context: 0.12 PFLOP at 32K, 0.25 at 64K, 0.49 at 128K, up to 3.9 at 1M. That is the linear part. The reduction column, dense attention FLOPs over sparse, reads 2.1x at 32K tokens, 4.0x at 64K, 8.0x at 128K, 16x at 256K, 31.5x at 512K and 64.5x at 1M. Divide each context length by its reduction and you get the same number every time, within rounding: 32,768 / 2.1 is about 15,600, and 1,048,576 / 64.5 is about 16,260. Dense attention reads n positions per query; SSA's cost is what dense attention would cost at about 16K.
Figure 12, the single-layer speed comparison against FlashAttention-2, says "SSA reaches parity near 16K tokens." Below that there is nothing to skip.
And the 12M-token needle test, the report says, ran "attending to only 0.13% of token pairs". 0.13% of 12M is about 16,000.
Three independent numbers, one budget. My reading is that each SSA query reads a fixed set of roughly 16,384 positions per layer, whatever the context length. I could be wrong in the details (the FLOP accounting could count causal halves differently, which would halve the figure), but the scaling is not in doubt: flat per query above about 16K tokens, dense below it.
This is the most useful thing to know about Snap, because it tells you where its architecture can matter. Slide the widget to the Decision Index's longest request:
At 1,933 tokens, Nimble's longest record, SSA and dense attention do identical work. At 34K, the Decision Index's ceiling, the saving is about 2x in the attention layers alone, and attention is only part of a forward pass. At Snap's 393,216-token limit it is about 24x, and that is where a dense decision model would actually hurt.
Two small things in the report's accounting bothered me. Table 3 compares DSA with SSA "under a matched selected-position budget", and its SSA column (headed "DBSA", an acronym the report never defines) is identical, row by row, to the sparse-attention-only column of Table 2: 71.0T at 128K, 568.3T at 1M. So the comparison charges SSA's selector zero FLOPs. That may be close to true; nothing published lets me check it. And the numbers drift between Subquadratic's own documents: the May SSA blog post gives a 62.8x FLOP reduction at 1M where the report says 64.5x, reports both 56.2x and 52.2x as the 1M prefill speedup, and lists 6.88x at 128K as both a wall-clock speedup and a FLOP reduction. None of this changes the shape. It does tell you the efficiency figures were not proofread against each other.
## "The only decision model or LLM"
The post says Snap is the only decision model or LLM whose compute per token stays constant as context grows. The blog calls SubQ "the first fully subquadratic LLM".
The "or LLM" half is not right, and Subquadratic's own report is the evidence. Its section on recurrent models: "The complexity is O(n) by construction," naming Mamba, RetNet, RWKV and gated variants. A pure recurrent language model has constant per-token compute, and several ship as open weights; the site covered one, [RWKV-7 G1k](/articles/rwkv-7-g1k), a 100% RNN family. What the report actually argues is that those models give up exact retrieval. That is a fair argument and a different claim.
"The only decision model" is plausible. Every [Decision Index](https://huggingface.co/spaces/multimodalart/jev-decision-index) entrant whose base model I know is a dense or hybrid transformer underneath, and I found no published decision model on a recurrent or fixed-budget sparse backbone. I would phrase it as "the first one with a fixed attention budget" and leave LLMs out.
## The latency numbers are almost linear
The launch post gives four speeds: "80ms for minimal context, 250ms for 32k, 660ms for 100k, and 2.5 seconds for 384k." If per-token cost is constant, time after the fixed overhead should grow in proportion to context. Subtracting the 80 ms floor, the marginal cost is 5.2 microseconds per token at 32K, 5.8 at 100K and 6.3 at 384K.
That is close to flat. A purely linear line through the 32K point predicts about 2.07 s at 384K, not 2.5 s. A dense model would be in another world: if all of the 32K request's variable time were quadratic, 384K would take about 23 seconds.
With a crude two-term model (a part that grows with n and a part that grows with n²), all three published points fit when about 2% of the 32K request's variable time is quadratic. That residue could be the selector, memory traffic, the vision path or just rounding in a launch post. Either way, the published speeds leave very little room for quadratic work. Someone in the replies plotted their own three measurements and called them "remarkably linear"; Whedon replied that he was glad they "were able to observe the linear behavior".
The launch video's policy demo is in the same range: about 212,849 tokens answered in 1.62 s, where a line through the published points gives roughly 1.3 to 1.4 s. Two replies are worth noting as leads I could not confirm: one integrator reported that the largest input they managed to send was 295,957 tokens, not 384K, and a closed-beta tester reported 90 to 95% accuracy "up to ~195k tokens" with a 0.55 s median. I had no API key and did not call the service.
One more back-of-envelope, with a caveat. The SSA blog's B200 table gives 380.96 ms for one SSA layer at 1M tokens. Scaled linearly, that is about 140 ms per layer at 384K. Snap does all of its layers at 384K in 2.5 s, which is about 18 such layers end to end, with nothing else in the forward pass. So either Snap is shallow or narrow next to the layer configuration that was benchmarked, or it is served across several GPUs in parallel, or its budget is smaller. The model's size, depth and serving setup are unpublished, so I can't say which.
## Where the scores can and cannot speak
The launch makes five evaluation claims. They split cleanly once you know the budget.
### The short-context wins say nothing about SSA
Two of the headline wins are on short inputs. Bespoke Nimble's public benchmark is 3,880 records from 13 datasets, and its own [documentation](https://github.com/bespokelabsai/nimble/blob/main/docs/PUBLIC_BENCHMARKS.md) says "no record exceeds the 2,048-token budget"; the longest is 1,933 tokens. The Decision Index tops out at about 34K tokens by the launch blog's own description. In both, SSA either does nothing or saves a factor of about two. Snap's 82.9% against Jev's 76.0% on Nimble, and 61.7 against 57.9 on the Decision Index, are wins for whatever backbone and training Snap has, not for its attention.
Jev's 76.0 on Nimble checks out: it is the plain average of Jev's 13 subset accuracies in Nimble's results table (80.4, 89.7, 81.0 and so on), exactly 76.0. Snap's 82.9 is Subquadratic's own run; Nimble's repository has no Snap results.
The Decision Index number needs two qualifications.
First, nobody else has run it. The board's `data/index.json`, regenerated on 12 October, has 148 entrants and no Snap. The 61.7 is Subquadratic's own run of the public kit, the same situation as [Agr's 58.15](/articles/agr-decision-model) a week earlier.
Second, the chart compares Snap with Jev and nobody else. The launch cites "Decision Index v0.2.1". On the 0.2.1 board as it stood on 28 September, Jev led at 57.91, so 61.7 would have been first by almost four points. The board has moved since: on the public-panel score it reports today, seven entrants sit above 61.7, from RSI-Jev v6.1 VL 27B at 65.64 and vLLM-SR d3 at 65.45 down to pplx-decider v1.1 at 62.25, and Cloudflare's clef sits at 61.71. Against the field Snap launched into, 61.7 is eighth, level with clef. That is still a good score for a model that is also the only one built for 393,216 tokens. It does not make Snap "the industry leader in short and long context decision-making", as the homepage puts it, on the short half.
The area bars are worth reading closely. The blog's text says Snap "outperforms Jev on four out the five major categories" and lists those four; the chart adds the fifth, knowledge and reasoning, where Snap is behind, 48.5 to 51.2. That loss comes with CRUXEval, a knowledge benchmark, at +16.7 in Snap's favour, so the rest of the area (GPQA Diamond, MMLU-Pro, BBH, HLE and the others) is further behind than the bar suggests.
The Jev bars are the odd part. Three of them match Jev's area scores on the 0.2.1 board: language 62.0, retrieval 55.4, tools 75.1. Knowledge is 51.2 against the board's 51.4. Arts and human taste is 32.8 in the chart and 37.7 on the board. Weight the chart's five Jev bars the way the index does (25.8% each for knowledge and language, 20% retrieval, 18.3% tools, 10% arts, as [measured in the Agr article](/articles/agr-decision-model) from the kit) and you get 57.3, not the 57.9 printed beside them; Snap's five bars give 61.6, consistent with its 61.7. So Snap's area scores and its total agree, and Jev's do not. Against the board's arts figure, Snap's largest area gain, +12.1, would be about +7.2. I don't know where 32.8 came from; it may be an older run or a different arts panel. It is the number that makes the chart's most striking bar.
### The enterprise table is honest arithmetic over a chosen subset
I compared every Jev number in this table with Jev's chance-corrected skill scores in the board's `index.json`. All sixteen match after rounding: CLINC150 89.21, BANKING77 79.48, PhishNChips 25.10, ESCI 43.82, through to ACOS 27.27. Subquadratic copied the board, not a rerun. The averages also reproduce: Snap's sixteen average 67.54 and Jev's 61.58, a gap of 5.96, and Snap is ahead on 13. The launch materials round 67.54 three ways (67.5 on the homepage, 67.6 in the blog's summary and chart, 67.54 in its table), which is untidy but not wrong.
What the table cannot tell you is how the 16 were chosen from the 37 in the index. The blog's criterion is benchmarks "that map most closely to enterprise workloads", which is reasonable, and the three losses are left in. When2Call, deciding whether to call a tool, ask, answer or decline, is the largest loss at −7.6, and it is the most agent-shaped benchmark on the list.
### Long context is where SSA can show, and the evidence is thinner
Decision RULER is Subquadratic's conversion of 11 of RULER's 13 tasks into four-option decisions; they dropped the two question-answering tasks because models could answer them from memory. Snap scores 99.8 at 32K and between 99.3 and 99.8 out to 512K. The blog's summary leads with "96.3 on Decision RULER at 1MM tokens". The API rejects anything over 393,216 tokens, so the 512K and 1M bars, including the headline, measure a configuration a customer cannot send. They are evidence that SSA keeps retrieval at long range, which is what the report claims for the 1.1 Small model too. For Snap as sold, the number to quote is 99.8 at 256K. Jev appears once, at 32K, with 94.1, because that is as far as it goes.
Decision LCR is the most interesting result in the launch, because it is the only one that tests reasoning, not lookup, over inputs that do not fit in 32K. The protocol is careful: 118 decisions from 100 questions, three option orderings each, credit only when every part of a question is right, random at 19.2 and models with documents removed at about 24 to 25. The confidence intervals in the chart are the right ones; I recomputed Wilson 95% intervals for 74 and 48 out of 100 and got 64.6 to 81.6 and 38.5 to 57.7, as printed.
But it is a protocol comparison as much as a model comparison. Jev cannot see a 100K-token document set, so it was given chunks, and the blog does not say how chunk answers were combined. A 26-point gap between reading everything and reading pieces mostly confirms the premise that whole-document reading helps. The obvious missing baseline is a long-window decision model that reads the whole set too. One exists: [pplx-decider v1.1](/articles/pplx-decider-v1-1) accepts up to 262,144 tokens per request, enough for every LCR document set, though it was trained only up to 8,192. The blog's line that "today's decision models stay at 32K tokens" was already untrue of Perplexity's API when Snap launched.
## Price, and what I would use it for
Snap costs \$0.05 per million input tokens. A full 393,216-token request is about two cents, and a 100K-token request half a cent. pplx-decider is \$0.02 per million, so Snap is two and a half times the per-token price. Perplexity's own docs say "about 190,000 tokens took 14 seconds"; Snap's post says 2.5 s for twice that. Whether Snap bills the state once per request or once per question, as Perplexity's server does, the docs do not say directly; they advise putting related questions in one request "so the state is sent once", which suggests once.
What I take from all this:
The architecture claim is mostly real and badly worded. Every published efficiency number is consistent with a fixed budget of about 16K positions per query and a linear end-to-end cost, and the launch latencies are close to linear. "Only LLM" is wrong; "only decision model" is probably right. The selection mechanism, the part that makes or breaks the linear claim, is unpublished, and the report's own comparison prices it at zero.
The short-context scores are good and unrelated to the architecture. Treat 61.7 as a self-run eighth place behind seven other entrants until it appears on the board, with a weak spot in knowledge-heavy questions.
The long-context results are the reason to try it. I know of no other decision model with a published long-context evaluation, and no other that publishes a latency at 384K. If your decisions really depend on the whole repository or the whole contract (picking the files a change touches, checking an answer against a long policy), Snap is the first decision model built for that, and the API accepts the same requests as Jev's. Before trusting it, measure it on your own labelled long inputs, set thresholds from those, and compare against a strong long-window baseline, not against a 32K model reading chunks. That is the comparison Subquadratic did not publish.
## How I checked
I read the launch post, its thread and the first page of replies through the fxtwitter mirror; Subquadratic's homepage, the Snap launch blog, the "How SSA Makes Long Context Practical" post (updated 15 May), the SubQ 1.1 Small announcement, the 22-page SubQ-1.1-Small technical report PDF (text extracted with `pdftotext`, figures with `pdfimages`), and the API docs at docs.subq.ai (Decisions API, Models, quickstart). The budget inference divides context length by Table 1's reductions and compares with Figure 12 and the 0.13% figure. Board comparisons use the Decision Index Space's `data/index.json` (generated 12 October 2026) and `data/index-v0.2.1.json` (28 September), reading Jev's per-benchmark skill and every entrant's public-panel score. The area check reweights the chart's bars with the index's five area weights and compares Jev's bars with the board's category scores. Nimble's figures come from a shallow clone of [bespokelabsai/nimble](https://github.com/bespokelabsai/nimble), `docs/PUBLIC_BENCHMARKS.md`. Qwen3.8-27B's layer layout is from its `config.json` on Hugging Face. Latency arithmetic uses only the four figures in the post. I had no API key, did not call the service, and could not check anything Subquadratic ran itself: Snap's own scores, the Jev chunking procedure, the model's size or backbone, or how SSA selects positions.
---
# usd-brep: NVIDIA open-sources SMLib, a million-line NURBS kernel with a USD schema on top
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/usd-brep-nurbs-kernel
> date: 2026-10-13
> tags: 3d, geometry, nvidia, mcp, agents, licensing, developer-tools
Every mechanical CAD program you have used sits on a geometry kernel, and almost all of those kernels are closed. Parasolid is Siemens'. ACIS is Dassault's. Autodesk has its own. The one widely used open option, Open CASCADE, is LGPL and has a reputation (fair or not, and one of the replies under the launch post put it in three words: "occ is very bad") for being hard to make behave. So when NVIDIA published [usd-brep](https://github.com/NVIDIA-Omniverse/usd-brep) on 9 October under Apache-2.0, and the people who make [ClassCAD](https://classcad.ch) said on X that it is SMLib, a kernel they have built products on for more than 20 years, I wanted to know what was actually in the repository. Not the announcement. The code.
I expected a thin wrapper over a binary, or a cleaned-up subset. It is neither. The kernel directory alone is 967,172 lines of C++, about 534,000 of them code once comments and blank lines are stripped. There is a C API, roughly 300 pybind11 bindings, an OpenUSD schema for storing exact solids, a Hydra plugin so `usdview` can draw them, a validator, a reader for Open CASCADE's `.brep` files, and an unusually thorough set of docs written for AI coding agents. And within a day a downstream project, [smlibpy](https://github.com/awv-informatik/smlibpy), had shipped patched copies of two kernel files, one of which fixes a bug NVIDIA lists in its own `KNOWN_ISSUES.md` without knowing the cause.
This article is the long version: what a boundary-representation kernel is and why the hard parts are hard, what this one contains, how it puts solids into USD, what smlibpy adds on top, and what is missing next to the commercial kernels. I read both repositories at their release commits; I did not build or run either.
## What a kernel has to represent
A mesh is a list of triangles. It tells you where the surface roughly is and nothing else. Ask a mesh for the radius of a hole and you get an estimate from vertices that happen to lie near it. Ask which faces touch a given edge and you have to infer it from shared vertex indices. Ask it to fillet that edge and it has nothing to work with.
A boundary representation (B-rep) keeps two things apart. **Geometry** is the exact mathematical shape: an infinite plane, a cylinder of radius 18, a NURBS patch with its control net. **Topology** is the bookkeeping that says which bounded piece of which surface is part of the solid and who its neighbours are. A face is a surface trimmed by loops of edges. An edge is a curve trimmed by two vertices. A shell is a connected set of faces, and a closed shell encloses a region of space. The geometry answers "where", the topology answers "what is connected to what", and a kernel's job is to keep the two agreeing while operations tear them apart and stitch them back together.
The AOUSD problem statement that usd-brep vendors puts the split in one table, and the line I would underline is the one about where each kind of geometry comes from: meshes are "derived from exact geometry (or authored directly)", while B-reps are the "authoritative source". Everything downstream (rendering, simulation meshes, toolpaths) is a view regenerated from the B-rep at whatever resolution the consumer needs.
## NURBS, the shape language
Planes, cylinders, cones, spheres and tori cover most machined parts, and SMLib keeps them as analytic surfaces. Everything else, from a car body panel to the blend a fillet leaves behind, is a NURBS: a non-uniform rational B-spline.
A NURBS curve of degree $p$ is defined by control points $P_i$, a weight $w_i$ on each, and a non-decreasing knot vector $U$:
$$
C(u) = \frac{\sum_i N_{i,p}(u)\, w_i\, P_i}{\sum_i N_{i,p}(u)\, w_i}
$$
The $N_{i,p}$ are B-spline basis functions, built by the Cox–de Boor recursion from the knots. Each one is nonzero over only $p+1$ knot spans, which is why moving one control point bends only a local piece of the curve. The weights are what make it *rational*, and rationality is what lets a NURBS represent a circle exactly instead of approximately. The knots set where the pieces join and how smooth the joins are.
The widget below runs the same algorithms a kernel does: Cox–de Boor for the basis (NLib's `N_Basis*` routines in `source/SMLib/src/NL_Basis*.cpp` implement the same recursion) and Boehm's algorithm for knot insertion.
Three things are worth trying. Drag one control point of the cubic and watch only the coloured bump under it in the basis plot react; that is local support, and it is why a designer can tweak one region of a panel without disturbing the rest. Then load the circle. It is the textbook nine-point quadratic: the corners of a square, with weights of 1 on the points that sit on the circle and $\sqrt{1/2}$ on the corners. Every sample lands on the unit circle to within rounding error. Load the same control points with all weights set to 1 and the curve bulges outward; I computed the worst case at the diagonals as a radius of 1.0607, about 6% too big. A polynomial spline cannot make a circle. A rational one can, and a CAD kernel that has to report a hole as exactly 18 mm needs that.
Finally, insert a knot. A control point appears, the polygon changes, and the curve does not move: the readout reports the largest displacement, which is rounding error. Knot insertion is how kernels split curves at intersection points, make two surfaces' parameterisations compatible before blending them, and subdivide for intersection searches. It is the quiet workhorse under a lot of the operations below.
Surfaces are the same idea in two parameters, $S(u,v)$, with a grid of control points and a knot vector in each direction.
## Trimming, and why the topology needs "uses"
A NURBS surface is always a rectangle in its own $(u,v)$ parameter space. Real faces are not rectangles: the top of a plate has a hole in it, a fillet ends where it runs into a wall. So a face is a surface plus **trimming loops** that say which part of the parameter rectangle is kept. The outer loop runs one way, holes run the other, and the material is on the left as you walk along.
The trimming curves can live in two places, and the choice matters. SMLib stores the authoritative curve of every edge in 3D, and treats the 2D trim curve in each face's parameter space as optional derived data it can rebuild. The repository's B-rep format reference is blunt about it: a UV trim curve is "optional, derived data, not a defining characteristic of the BRep". The UsdSolid proposal says the same thing from the other side: "Trim curves are optional in this Brep model because the edge curve defines the model truth."
Then there is the question of how faces know their neighbours. The usual answer is that each edge points at the two faces it separates. SMLib uses something more general, Kevin Weiler's 1986 **radial edge** data structure, and the USD schema inherits it. Each topological object is stored once, and every time it bounds something higher up, that incidence is a separate *use* object: a faceuse for each side of a face, a loopuse, an edgeuse for each time an edge bounds a face, a vertexuse.
Why bother? Because a solid is not the only thing a kernel has to represent. When two boxes share a face, that face bounds two regions at once, and each side of it is a separate faceuse. When an edge is shared by three or four faces (which is what you see locally along any intersection curve before it has been resolved), the edgeuses around it form a radial list ordered by the right-hand rule. A twin-edge structure that allows exactly two faces per edge cannot represent either case, and booleans spend their whole middle phase in exactly that state.
The radial-edge model also pays off in interchange. The proposal maps its entities onto every major kernel; part of that table:
| AOUSD / usd-brep | Parasolid | ACIS | Open CASCADE |
|:--|:--|:--|:--|
| BRep | Body | Body | TopoDS_Compound |
| Region | Lump | Lump | TopoDS_Solid |
| Shell | Shell | Shell | TopoDS_Shell |
| Face | Face | Face | TopoDS_Face |
| Loop | Loop | Loop | TopoDS_Wire |
| Edgeuse | Fin | Coedge | TopExp_Explorer |
| Faceuse | Side | Side | Orientation |
The explorer below is a hand-built model of a block with a hole through it, laid out by the schema's counting rules. It is not the output of a kernel run, and a real kernel might split each circle into two arcs, which would change the numbers. What it does show is the structure.
Click the bore. Its one loop is four edgeuses: the bottom circle, the seam, the top circle and the seam again, in the opposite direction. A cylinder's parameter space is periodic in $u$, so the face needs an edge where $u = 0$ meets $u = 2\pi$, and the face uses that edge twice. The proposal's rules make this mandatory ("A face has a single outer loop (seam edges are required)"), and seams turn out to be one of the places this kernel struggles most, as we will see. The counts on the right come straight from the model: 10 vertices, 15 edges, 7 faces and 9 loops give $10 - 15 + 7 - (9 - 7) = 0$, which the Euler–Poincaré formula turns into genus 1. One hole, as expected. Every edge is used exactly twice, in opposite directions, which is what "closed and consistently oriented" means for a manifold solid.
## Tolerance is where the trouble lives
If geometry were exact, an edge would lie exactly on both of its faces. It never does. The intersection of two curved surfaces has no closed form, so the kernel fits a B-spline to it within some tolerance, and that curve lies on each surface only to within that tolerance. Imported files carry the tolerances of whatever system wrote them. Every comparison in the kernel ("are these two points the same vertex?", "does this curve lie on that face?") is really "are they within tolerance?".
SMLib's defaults are in `source/SMLib/inc/SmMath.h:52-54`:
```cpp
#define SM_ZONE_TOL_3D 1.0e-5 /* Local Neighborhood Size, used by SmTol::GetZoneTol3d() methods */
#define SM_XSECT_TOL_3D (2.0*SM_ZONE_TOL_3D) /* Default Gap and SmTol::GetXSectTol3d Size */
#define SM_APPROX_TOL_3D (SM_ZONE_TOL_3D/2.0) /* Default SmTol::GetApproxTol3d Size */
```
Three numbers, three jobs: a zone tolerance of 1e-5 for "these are the same point", an intersection tolerance of 2e-5 for gaps and loop closure, and an approximation tolerance of 5e-6 for fitting curves. They are in model units, so they mean something different in millimetres than in metres. Individual vertices and edges can carry their own looser tolerances on top of these, which is how a kernel absorbs a sloppy import without rejecting it. The USD side keeps one number per solid, `brep:intersectTol3d`; the cube in the repository's test files stores `0.00002`, which is the 2e-5 default.
This is the part of kernel engineering that never shows up in a feature list. smlibpy's docs give a concrete sense of the scale: faces the kernel fits (B-spline chamfers, shell offsets, intersections of curved faces) "close up only to about 1e-6 to 2e-5 mm", and the volume of such a body depends on that by 1e-7 to 4e-6 of its value. Small, and enough to make two operations that should agree disagree.
## What is in the repository
I counted the release commit, `c9c5979` (version 1.0.0, 9 October 2026). The "Lines" column includes comments and blank lines; "Code lines" strips them:
| Part | Files | Lines | Code lines |
|:--|--:|--:|--:|
| `source/SMLib` (the kernel) | 462 | 967,172 | 534,172 |
| of which `NL_*` (the NLib NURBS layer) | 40 | 253,923 | |
| other `source/` libraries (C++) | | 64,017 | 49,190 |
| `tests/` (C++) | | 63,102 | 44,065 |
| Python, all directories | 79 | 36,914 | |
All C++ together is 1,096,957 lines. The test fixtures are another 580 files (STEP-derived solids, fillet and drop-curve cases, Open CASCADE `.brep` files, and 126 USD files for the validator, most of them deliberately broken in one way each). The C++ regression harness `prog_test` runs 17 suites from topology and booleans through fillets, sweeps, surface-surface intersection and stitching, and I counted about 530 Python test functions across the repository, 400 of them in `test_omni_solid.py`.
The layering is clean once you see it:
- **`SMLib`** is the kernel proper: the topology classes (`SmBrep`, `SmRegion`, `SmShell`, `SmFace`, `SmLoop`, `SmEdge`, `SmVertex` and their `*use` classes), the geometry classes, and the algorithms. The `NL_*` files are a lower NURBS library underneath it. The biggest files say where the effort went: `SmBrep.cpp` (38,863 lines), `SmSurface.cpp` (35,380), `SmCurve.cpp` (27,725), the polygon library `SmPoly.cpp` (27,402), `NL_SrfAdv.cpp` (23,322), `SmFace.cpp` (21,752).
- **`SM_API`** is a C-style wrapper (`SmApi*` functions) over the kernel, with headers for primitives, booleans, curves, surfaces, fillets, intersectors, trimmed surfaces, queries and healing.
- **`SM_API_USD`**, `BREP_SM_USD` and `BREP_USD_DATA` convert between kernel solids and the USD schema.
- **`SmPyLib`** builds the Python module `_omni_solid` with pybind11 2.11.1; the pip package `usd-brep` wraps it as `usd_brep` with submodules (`primitives`, `booleans`, `sweeps`, `fillets`, `usd`, …).
- **`HYDRA_USD_BREP`** is a Hydra imaging plugin that tessellates BrepArray prims on the fly so `usdview` can show them; **`OCCT_BREP_IMPORT`/`EXPORT`** read and write Open CASCADE's `.brep` format by parsing it directly, without linking Open CASCADE.
Where does SMLib come from? The repository never says. Every source file carries a 2026 NVIDIA SPDX header and the README says "Copyright (c) 2020-2026, NVIDIA". The rest I pieced together. An archived 2019 copy of smlib.com, the site of Solid Modeling Solutions, describes "SMLib™ Our flagship product built upon TSNLib™ provides fully functional non-manifold topological structure and solid modeling functionality including the ability to perform NURBS boolean operations, fillets, etc.", stacked on GSNLib and on "NLib™ A comprehensive BSpline NURBs library". The `NL_` prefix and the `NL_KNOTVECTOR` types in the source match that stack, and AWV Informatik says it has used SMLib for 20 years. I could not find how or when the code moved to NVIDIA, so I won't guess.
The same archived page lists "Data Translators" for IGES, STEP, SAT, VDAFS, OpenNURBS and Parasolid XT. None of those are in this release. I come back to that below.
## A Boolean, traced through the code
Booleans are the operation everyone uses and nobody sees. The Python call is one line:
```python
import _omni_solid as sm
box = sm.create_box((0, 0, 0), 10, 10, 10)
sph = sm.create_sphere((5, 5, 5), 4)
diff = sm.boolean_difference(box.copy(), sph.copy()) # inputs are consumed; copy to keep them
```
`SmApiBrep.cpp:231-238` dispatches on whether both inputs are manifold solids, to `SmMerge::ManifoldBoolean` or `NonManifoldBoolean`. The manifold path then calls `SmTopologyIntersector::IntersectInsertRelate()` (`SmTopologyIntersector.cpp:1092`), and its comments lay out the phases:
1. **Intersect topology** (`:1184`): find every place the two solids' vertices, edges and faces meet, including surface-surface intersection curves. Results go into a scratch structure of *proto-topology* (proto-vertices, proto-edges) rather than into the solids.
2. **Resolve intersections** (`:1208`): merge near-coincident proto-vertices, decide which overlapping proto-edges are one edge. The comment after each of the first two phases reads "If that didn't work, quit before we mess up the Breps."
3. **Imprint intersections** (`:1232`), with the comment "now we'll modify the Breps": split faces and edges along the resolved curves.
4. **Relate** (`:1263`): record which new pieces of each solid correspond, so `SmMerge` can classify each face as inside or outside the other solid and keep or delete it by operation.
That separation (find everything, decide everything, only then mutate) is what you want in a kernel, because a half-imprinted solid is unrecoverable. The decide phase is also where the judgement calls live, and one of them is right there in the source, at `SmTopologyIntersector.cpp:1145-1158`:
```cpp
// Capture Distance: simply using entities' tolerances will miss intersections
// that could possibly go into the same ProtoVertex. We should collect candidates
// into the same PVs, and let Resolve() figure it out, making either one or more
// SmVerttices out of each PV.
// This is a concept in progress. We currently check the tolerance value in
// the TopologyIntersector, which ultimately comes from the tolerance in the Breps. [B669]
// We might want to look at Bounding Box sizes, or min Edge length, or such.
// But currently this works well.
double dCaptureDistance = 0.01;
// Make sure it's not smaller than our tolerance value.
double dTopoIntTol = this->GetThisApproxTol3d();
if ( dCaptureDistance < dTopoIntTol * 2000 )
{ dCaptureDistance = dTopoIntTol * 2000; }
```
With the default approximation tolerance of 5e-6, the floor is 2000 × 5e-6 = 0.01, so the two numbers coincide. It is an absolute distance in model units: a hundredth of a millimetre for a part modelled in millimetres, a centimetre for one modelled in metres. I like that this is visible. Every kernel has constants like this; in a closed one you find them by bisecting failures.
The repository's own guide for booleans adds the practical advice: avoid tangent or coincident surfaces ("Offset one body by a tiny amount if needed"), watch for features thinner than the tolerance, and check `is_manifold_solid()` after each step. Those are the classic failure modes of every kernel, open or not.
## Solids inside USD
The other half of the release is the part only NVIDIA would have built. OpenUSD, the scene format Omniverse is built on, has meshes, subdivision surfaces and untrimmed NURBS patches. It has no solids. When CAD data enters USD today it is tessellated at the door, and the exact geometry stays behind in a STEP file.
usd-brep adds a schema, `omniSolid`, with one concrete prim type, `BrepArray`, and a family of applied API schemas for geometry (`BrepSurfaceNurbAPI`, `BrepSurfacePlaneAPI`, `BrepSurfaceCylinderAPI`, cone, sphere, torus; `BrepCurve3dNurbAPI`, line, circle, ellipse; `BrepCurveUvNurbAPI` for trim curves). The whole radial-edge structure is flattened into parallel arrays of indices: `region:type`, `shell:faceuseCount`, `faceuse:faceIndex`, `face:loopCount`, `loop:edgeuseCount`, `edgeuse:edgeIndex`, `edgeuse:nextRadialEUIndex`, and so on. The schema's documentation explains how it compresses the in-memory structure: "There are no attributes for Vertexuses and Loopuses", and each runtime pair of mated edgeuses becomes one array entry. The test cube shows the result: 2 regions, 2 shells of 6 faceuses each, 6 loops of 4 edgeuses each, and 24 edgeuse records for 12 edges.
Geometry is packed by type. All the NURBS surfaces of every face sit in one set of arrays (`brep:surface:nurb:controlVertices`, `uKnots`, `vKnots`, `weights`, …) in double precision; all the planes in another. Several solids can share one prim, their entries laid end to end.
Why so flat? The proposal, which usd-brep vendors unmodified, reports the designs the working group tried first. One prim per surface and curve "was an untenable design": a car surface model came to about 500 B-reps with 100 geometry prims each, 50,000 prims for one model. Moving geometry into one prim per B-rep "improved performance and shrunk file size by 1/3", and the array design packs further. Treating B-reps as an opaque blob (an embedded STEP file, say) was rejected because it gives up per-face materials, USD composition and overrides.
Some limits are stated plainly. The schema is for *using* CAD data in USD (viewing, measuring, clash detection, re-tessellation), not editing it on the stage; the proposal says live editing would need schemas for "the 11 topology and use objects". The AOUSD proposals (OpenUSD-proposals pull requests 108 and 109, opened 20 May 2026) are still marked Draft, which is why NVIDIA ships the schema under its own `omniSolid` name and keeps the proposal's `prelimUsdSolid` copy beside it for reference. And the round trip is lossy in documented ways: per-entity tolerances "collapse to one per-BRep value", edge curves are exported as NURBS, and analytic faces lose their UV trims. NVIDIA's own asset-generation guide tells agents not to round-trip through `BrepArray` at all when the goal is a mesh.
The supporting tools are solid. The validator (`tools/brep_validator`, also available as `pip install "usd-brep[validation]"`) checks the topology arrays rule by rule, with a broken fixture file per rule. The Hydra plugin tessellates once per prim, in parallel across prims, and never writes meshes back to the stage. `usd.heal_file` heals every BrepArray in a composed stage and writes it back with layers, references and instancing intact.
## The Python layer, and docs written for agents
The stable Python module is broad. The operation index lists 16 primitive constructors, 13 boolean variants (including CSG trees and 2D booleans), 13 sweeps, 9 fillet and chamfer calls, 22 curve calls, 11 surface constructors, offsets, stitching, 8 intersectors, cut, project and trim, tessellation, polygon-mesh booleans, healing and the USD import and export calls. Topology is walkable from Python: `face.edges()`, `edge.faces()`, `vertex.point()`, `face.normal_at_uv(u, v)`.
Two conventions deserve a warning before you write code against it. Booleans *consume* their inputs ("Ownership of the named inputs transfers to the operation; use only the returned Brep afterward"), so you `.copy()` anything you want to keep. And the kernel is not thread-safe: the concurrency guide says the process-global context "holds unlocked mutable shared state", operations run single-threaded in the shipped build, and the unit of concurrency is the process. One worker process per job.
The pip wheel is `usd-brep` 1.0.0, for CPython 3.12 only, about 90 MB on Linux x86-64 because it bundles its own OpenUSD 25.11 runtime. Building from source uses NVIDIA's `repo`/premake/Packman tooling; on macOS the README notes that the premake package "is not on the public Packman server", and someone in the launch thread asked about exactly that.
Then there are the agent docs. `AGENTS.md` is 19 KB, and `.agents/` holds 20 per-operation guides, an error-code decoder and 19 skills ("smlib-kernel-booleans", "smlib-kernel-healing", "usd-brep-schema", "text-to-3d-asset-generation" and so on). They are good. The B-rep format reference alone is 1,211 lines and explains invariants (radial list order, which edgeuse carries the trim curve, how seams at poles go wrong) that I have never seen written down for any kernel. NVIDIA clearly expects a lot of the code that calls this library to be written by models.
## smlibpy: the first downstream, and what it found
[smlibpy](https://smlibpy.classcad.ch) is AWV Informatik's Python library on top of usd-brep. AWV makes ClassCAD, and its post announcing the open-sourcing came with a short reel of the products it says run on SMLib; the post lists headless CAD, a GUI modeller, AI/MCP tools and a PDM system.
smlibpy is MIT-licensed and borrows its API from CadQuery and build123d. Its README example is a plate with rounded corners, four holes and a pocket:
```python
from smlibpy import *
plate = Box(80, 60, 10)
plate = plate.fillet(plate.edges().parallel_to(Z), 8)
plate -= GridLocations(60, 40, 2, 2) * Cylinder(4, 10)
plate -= Rectangle(30, 20).on(plate.faces().top()).extrude(-4)
plate.export("plate.step")
```
Shapes combine with `+`, `-` and `&`. Faces and edges are selected by geometry (`faces().top()`, `edges().parallel_to(Z).concave()`), and every operation returns a new part instead of consuming its inputs. The README says the plate builds in about 40 ms; I did not run it. It adds a lot that the kernel's own bindings lack: sketches with a pen-style `Profile`, sheet metal with exact bends and flat patterns, draft analysis, text from font outlines, its own STEP reader and writer, DXF, glTF, and a software renderer that draws technical drawings with hidden lines.
The part I found most interesting is not the API. It is `third_party/patches/`. smlibpy compiles the kernel from the usd-brep submodule as published, except for two files it replaces with fixed copies, and its README documents every change. One fixes a write into freed memory "at the end of every Boolean", which they found with AddressSanitizer on `Box(40, 30, 20) - Cylinder(5, 40)`. Another stops a vertex-gluing step from deleting curves another solid still uses; they say fillet removal on an imported valve body crashed the process in 3 of 3 runs before the fix.
The third is the one I checked line by line, because it explains an item in NVIDIA's own `KNOWN_ISSUES.md`: "`remove_fillet` removes nothing and reports success", workaround "none". Here is `SmVertex::GetVertexuseOfFace` upstream, at `source/SMLib/src/SmVertex.cpp:256-267`:
```cpp
// Remove Composite
// // when VU connects to TgtFace (or one of its member faces)
// if( (!bIsComposite && pThisFace == pFace)
// || ( bIsComposite && ((SmCFace*)pFace)->ContainsFace(pThisFace)))
{
// remember any found Vertexuses
pRet = pVU;
// Use the first Vertexuse found on an upward Faceuse.
if ( pThisFace->GetUpwardFaceuse() == pFU )
{ break; }
}
```
The function is supposed to find the vertex's use on a given face. Someone removing support for composite faces commented out the whole condition, including the plain `pThisFace == pFace` test, and left the braces. The function now returns the use on whichever face comes first, whatever face you asked for. smlibpy restores the face test here and in `GetLoopOfFace`. Their measurement: kernel fillet removal on a box with four rounded edges now gives back 6 faces and a volume of 24000.0, where before it removed nothing and left 10 faces.
The comment `// Remove Composite` appears 458 times across 50 files of the kernel. I did not audit the others. But it suggests that at least part of preparing SMLib for release was stripping a feature by commenting code out, and that the test suite did not catch every place that went wrong. Finding that within a day of release is what an open kernel is for.
smlibpy also replaces some of the kernel's numerics instead of patching them. Its docs say the kernel's mass-property integration "controls its steps by the area alone", so "a volume can be 300 times worse than the accuracy asked for", and that `SmCurve::Length` counts a span starting with a zero tangent as zero (the outline of a cylinder came out with 30 mm top and bottom lines instead of 60). smlibpy integrates volumes and inertia itself, over the trim curves by Green's theorem with adaptive Gauss–Kronrod quadrature, in `src/cpp/measure.cpp`. NVIDIA's known-issues file independently reports area errors "up to about 1.4% at the default `relative_accuracy` of 1e-3" on faces bounded by circles, so the two accounts agree that the kernel's integrals need care. I did not reproduce either number.
Seams come up again in smlibpy's table of kernel behaviour it works around: NX and other systems write cylindrical and spherical faces without the seam edge SMLib requires, so smlibpy inserts one on STEP import, and turns the surface's frame so the seam lies where the angle parameter starts. NVIDIA's known-issues list has ten separate bullets about healing faces on closed surfaces, seams and poles. If you take one thing from the topology explorer, take the bore's doubled seam edge: it is ordinary in theory and a steady source of bugs in practice.
## Where agents come in
AWV's pitch for smlibpy is agents: "It comes with an MCP server, so your agent can build parts too." The server (`smlibpy-mcp`) gives a model a `run_script` tool that executes smlibpy Python with variables kept between calls, a `snapshot` tool that renders the model (drawings, sections, x-ray, highlighted selections, before-and-after diffs), `inspect`, checkpoints, and per-function guides. The skill has about 140 reference pages, and the repository's `ai/training/` directory holds the scripts agents ran while writing them.
I think the case for an open kernel here is real but narrower than the replies suggested. Language models are bad at geometry and decent at writing code that calls a geometry library. What makes the loop work is that a B-rep can answer questions precisely: the volume is 157,979.5 cubic millimetres (in their demo), the selection picked 8 circular edges, the solid is manifold. A mesh pipeline cannot give the model that kind of feedback. Agents that write OpenSCAD or Blender scripts already exist (the [WorldClaw](/articles/worldclaw) paper builds whole terrains that way), but they produce meshes, not parts you can send to a machine shop as STEP. An Apache-licensed kernel removes the per-seat licence question for anyone who wants to host such a server, ship it inside a product, or run thousands of copies as an environment to train CAD agents against, which one reply suggested.
Two caveats from the code. The kernel crashes; smlibpy runs risky operations in a forked copy with a timeout (`smlibpy.defeature`'s guard) and rebuilds sessions from their scripts after a crash, and any hosted agent tool will need the same. And one process per concurrent job is the rule, so a server for many agents is a pool of worker processes, not threads.
## Update, 13 October 2026: smlibpy in the browser
The afternoon after this went up, [@0xca0a](https://x.com/0xca0a/status/2110022758729937313) announced smlibpy's open-source release with three parts: the Python API, "a wasm compile for javascript/ts (1.8mb dl/brotli total)", and the MCP server. The Python API and the MCP server are what the sections above read at `f339a0b`. I re-read the repository at `0aaa37f` ("Plugin: smlibpy 0.2.3"), 67 commits later. The new part is `js/`, published to npm as `@classcad/smlibwasm`.
**How it is built** (read in the code). `js/CMakeLists.txt` links the same `smlib` and `sm_api` static libraries the Python wheel uses, patched files included, plus smlibpy's own C++ from `src/cpp`, into one Emscripten module: embind, `-fwasm-exceptions` (the kernel throws in places it never catches, and this turns that into a JavaScript error instead of a dead module), `MODULARIZE` as an ES module, memory from 64 MB growable to 4 GB, a 4 MB stack, Emscripten SDK 6.0.12, a plain CMake Release build. Kernel objects cross into JavaScript as bare addresses, and a TypeScript layer of about 29,000 lines owns and frees them. The API is smlibpy's in camelCase, operators become `fuse`, `cut` and `intersect`, keyword arguments become a trailing options object, and `js/PORTING.md` says the port keeps "the same algorithms, checks, tolerances, messages and fallbacks". The package README's plate prints a volume of 45440.00. I get the same by hand (reasoned): 48,000 for the block, minus 549.4 for four 8 mm corner rounds, minus 2,010.6 for four 4 mm holes.
**The 1.8 MB checks out** (measured). I downloaded the 0.2.3 tarball from the npm registry and compressed each file with brotli at quality 11:
| What a browser loads | Raw | Brotli |
|:--|--:|--:|
| `smlibwasm.wasm`, the kernel | 6,222,331 B | 1,541,480 B |
| `smlibwasm.mjs`, its loader | 86,184 B | 21,233 B |
| the other 32 `dist/*.js` files | 969,432 B | 243,264 B |
| `_fonts.js`, the font for `Text` | 130,018 B | 46,308 B |
| **Total** | **7.41 MB** | **1.85 MB** |
That is 1.77 MiB, so "1.8" holds in either unit. The docs put the library at 0.21 MB; I get 0.24, which does not move the total. The WebAssembly has no name section, and code is 5.9 MB of its 6.2. The whole is what the docs' size table says, not a budget: CI builds and tests the module but checks no size. The package is MIT for smlibpy's code, with SMLib under Apache-2.0 and the font under OFL, declared as `MIT AND Apache-2.0 AND OFL-1.1`, published with an npm provenance attestation.
**A third kernel patch** (read in the code; the effect is AWV's). Porting surfaced a bug the native build had only hidden. `SmOffsetExecutive::RemoveExcessFaces` deleted faces and kept them in the executive's lookup tables, so the rest of the offset read freed edgeuses and faces. Natively that was silent corruption; in WebAssembly it trapped. `third_party/patches/` now replaces three kernel files instead of two, and AWV reports AddressSanitizer clean on a half cylinder thickened by -12.
**The fillet-removal fix travels with it** (read in the code). The WebAssembly build compiles the patched `SmVertex.cpp`, and `js/test/defeature.test.ts` repeats the Python check: kernel fillet removal on the rounded box must give 6 faces and 24000.0. I did not run it.
**The MCP server barely moved, and the "training data" is documentation** (read in the code). Since `f339a0b` the server learned to show a read STEP file (its curves, coordinate systems and points) and an imported assembly, and its search knows `read_step` and `draft`. The JavaScript package has no MCP server. What the thread calls "real world training data" is `ai/training/`: 34 sessions from 12 and 13 October with 536 probe scripts, each written by an agent following `ai/HOW-TO-TRAIN.md` ("everything is live... If the docstring is wrong, say so"), whose output is the 143 reference pages the server hands out. No model is trained. Nothing in that folder changed today.
Two smaller things for the kernel story: `Part.draft` is back as an operation, built on `SmBrep::LocalOperation`, and STEP import now gives seams to more of the faces NX writes without them (tori, revolutions and extrusions, not only cylinders and spheres), the seam trouble described above. The `usd-brep` submodule still points at `c9c5979`, so none of this changes the kernel the earlier sections counted.
## What is missing next to Parasolid, ACIS and Open CASCADE
Plainly, from what I could check in the code and docs:
- **No STEP or IGES translator.** usd-brep reads and writes USD and Open CASCADE `.brep`. SMS sold translators for IGES, STEP, SAT, VDAFS, OpenNURBS and Parasolid XT, and none were released. smlibpy wrote its own STEP reader and writer in Python and C++ and checks its exports by reading them back with Open CASCADE. Open CASCADE ships STEP and IGES translators itself. For most real CAD work, interchange is the first thing you need.
- **Single-threaded, not thread-safe.** Some parallel code paths exist but are compiled out; scale with processes.
- **Known bugs in the core operations.** Healing that reports success when it failed, ten problems with seams and poles, inside-out triangles next to fillets, inaccurate areas on circle-bounded faces, a crash on malformed BrepArray topology. To NVIDIA's credit, all of this is written down.
- **No outside contributions.** The README says "This project is currently not accepting contributions"; issues are welcome. smlibpy's patches therefore live in smlibpy.
- **A kernel, not a CAD system.** There is no feature history, sketch constraint solver or assembly model. smlibpy sends people who want parametric rebuilds to the ClassCAD MCP. Kernels generally leave history and constraints to the application above them, so this is a scope note more than a gap.
What I could not check: robustness and speed against Parasolid, ACIS or Open CASCADE on the same models. Nothing in either repository compares them, and I did not run anything. One reply under the launch post said that on a personal set of difficult chamfer tests, "Occt passes many more cases than Smlib". That is one person's anecdote, and the honest status is that nobody has published a comparison yet. Neither could I verify how NVIDIA acquired SMLib or how much of the commercial product's history survives in this snapshot (the repository is a single release commit).
The licence comparison is easy, though. Open CASCADE is LGPL 2.1 with an exception; usd-brep is Apache-2.0, apart from MIT-licensed bootstrap scripts and the usual third-party notices (OpenUSD, oneTBB, pybind11 and so on). For a company that wants to modify the kernel and ship it inside a closed product or a hosted service, that difference matters more than any benchmark.
## What I would do with it
If you want to model parts from Python today, start with smlibpy, not the raw bindings: `pip install smlibpy` gives wheels for Python 3.10 to 3.14 on Linux, macOS and Windows at 5 to 7 MB, with the two kernel fixes compiled in and the integrals done properly. If you are in the USD world and need exact solids in a stage, usd-brep's schema, validator and Hydra plugin are the reason to look, and they are the part of this release with no open equivalent. If you are building an agent that designs parts, copy smlibpy's structure: scripts in a separate process, a renderer the model can look through, and a numeric check after every step.
And if you work on geometry kernels, read `.agents/skills/smlib-brep-model/references/brep-format.md`. It is the best description of a production B-rep's invariants I have seen in public, and now it comes with the million lines it describes.
## How I checked
I shallow-cloned `NVIDIA-Omniverse/usd-brep` at `c9c5979` and `awv-informatik/smlibpy` at `f339a0b` on 13 October 2026 and read them; I did not build or execute either. Line counts are `git ls-files` totals for `.cpp`/`.h` (and `.py`) files; "code lines" strip `/* */` and `//` comments and blank lines with a short script. Test counts are `def test_` occurrences and the `gProgSuites` table in `tests/prog_test/src/prog_test.cpp`. The smlibpy kernel patches were checked by diffing their copies against the upstream files; the effects of those patches (6 faces, volume 24000.0, the AddressSanitizer reports, crash counts) are AWV's measurements, not mine. The 6% bulge of the unweighted circle is my own calculation, sampled at 2,001 points, and the widget recomputes it live. The SMLib product description comes from the Internet Archive's December 2019 copy of smlib.com. PyPI metadata (wheel sizes, Python versions, upload dates) is from the PyPI JSON API. For the update I re-read smlibpy at `0aaa37f`, diffed it against `f339a0b`, and measured the npm tarball of `@classcad/smlibwasm` 0.2.3 with Node's brotli at quality 11; I did not build or run the WebAssembly either. Quotes from the UsdSolid proposal are from the copy vendored in `proposals/`, which NVIDIA records as an unmodified snapshot of OpenUSD-proposals pull request 109 retrieved on 16 September 2026.
---
# Xiryn: every agent gets its own Linux desktop, and a vault it fills without reading
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/xiryn
> date: 2026-10-13
> tags: agents, computer-use, security, privacy, multi-agent
The post that sent me here makes a large claim in one line. Xiryn is "the open-source personal agent that does everything Muse, Grok Bot and Dots do — on your own computer, without ever getting in your way." Sourav Bera posted it on 12 October with a minute of screen recording, a [GitHub repository](https://github.com/SouravBeraAkaSpeed/xiryn) and a [site](https://xiryn.com/).
Claims like that usually come attached to a thin wrapper: a chat box, a system prompt, and a call to somebody else's computer-use API. So the first thing I did was count. The repository at `ae9ad97` has about 18,000 lines of TypeScript, TSX and shell outside `package-lock.json`, of which 1,767 are stock shadcn UI primitives. There is an agent runtime, a tool belt of 22 tools, a Docker image the project builds for its agents, a Telegram bridge, and an encrypted vault with its own approval flow. It is MIT. It is not a wrapper.
What it is, mechanically, is simple to state. Each agent you create gets a Linux desktop of its own in a Docker container on your machine. The model looks at that desktop through screenshots, moves the mouse and types through `xdotool`, runs commands in a shell, and remembers things in a SQLite table. When a website wants your password or your card, the agent asks Xiryn to type it, and Xiryn types it without the model ever seeing the value. That last trick is the most interesting thing in the repository, and it is also where the README's wording runs ahead of the code.
## Two days of history
A repository's history tells you how much of it has been lived in, so I fetched all of it. There are 110 commits, every one by Sourav Bera, and 101 of them land on 11 October. The first 39, from `chore: initialize Next.js 16 project` to `chore(release): 0.1.0`, are stamped within the same minute, 18:32. Somebody finished the project first and then committed it as a series of logical steps. It is tidy, and it means the log cannot show how the code was developed.
The later commits are real iteration, and some of it is large. Version 0.1.0 ran each agent's machine on Daytona, a hosted sandbox service; 0.2.0, the same evening, replaced that with local Docker and says so in the changelog. The vault, payment approvals and the phone take-over page arrived in 0.3.0 and 0.4.0, also on the 11th. The project was called Oryn until commit `771cfd9` on the 12th renamed it.
There are no tests. CI runs `typecheck`, `lint`, `format:check` and `build`. For an app that types card numbers into web pages, that is the gap I would close first.
## One process, one file
Xiryn is a Next.js 16 server bound to `127.0.0.1:6796` and a SQLite database at `data/xiryn.db`. There is no queue and no worker. When you send a message, `POST /api/threads/:id/turn` calls `startTurn` in `lib/agent/runtime.ts`, which saves your message, assembles a system prompt, builds the tool belt, and calls the Vercel AI SDK's `streamText`.
The interesting choice is where the turn lives. Its output is pushed into an in-memory `Run` object that buffers every chunk and fans them out to however many browser tabs are listening. Close the tab and the turn keeps going; reopen it and `GET` on the same route replays the buffer and follows it live. The README calls these durable turns. They are durable against the tab, and the code comment says what they are not durable against: "Runs are kept in memory: they survive the tab, not a server restart."
The tool loop is `streamText` with four ways to stop:
```ts
// lib/agent/runtime.ts:537-542
stopWhen: [
isStepCount(maxSteps),
hasToolCall("request_takeover"),
hasToolCall("request_payment_approval"),
hasToolCall("request_secret"),
],
```
`maxSteps` defaults to 40 (`XIRYN_MAX_STEPS`). The three `request_*` tools end the turn the moment the model calls them, which is how Xiryn implements "ask the human": the agent does not wait inside a tool for an answer; it stops, and your answer arrives later as an ordinary user message that starts a new turn. On the 40th step, `prepareStep` switches tools off with `toolChoice: "none"` and appends an instruction to report what it found and what is left, so a long task never ends mid-click with nothing said.
Models come through the AI SDK's provider packages: OpenAI, Anthropic, Google, xAI, Groq, Mistral, DeepSeek, OpenRouter, and OpenAI-compatible endpoints for Ollama, LM Studio and anything else. Those are the README's "11 providers". Each agent has its own model, chosen as `provider/model`. Nothing here is a computer-use API from any vendor; every tool is a plain function tool with a Zod schema, so any model that can call tools can drive Xiryn, and any model that can also read images can drive the desktop.
## The computer is a container
Most of the engineering is here. `lib/workstation.ts` creates one container per agent, named `xiryn-ws-`, from an image the project builds on your machine out of `workstation/`. The image is Debian trixie with Xvfb, the xfwm4 window manager and an XFCE panel, Chromium, x11vnc and noVNC, `xdotool`, `scrot`, Node 24, Python, and `cloudflared`. The container is created like this:
```text
# lib/workstation.ts:131-164, abridged
docker create --name xiryn-ws- --hostname
--env XIRYN_VNC_PASSWORD=<8 random chars> --env XIRYN_IDLE_MINUTES=15
--volume xiryn-home-:/home/xiryn
--publish 127.0.0.1::6080/tcp
--shm-size 1g --cpus 2 --memory 4096m
```
The home folder is a named volume, so files and browser sign-ins survive restarts and image updates. The noVNC port is published on loopback with a random host port, and your browser connects to it directly with the container's password, which only the Xiryn server hands out. A watchdog inside the container (`xiryn-desktop`) stops it after 15 idle minutes, counting a running command or anyone watching the screen as activity, and it does this even if Xiryn itself is closed. That last detail is the kind of thing you only write after a laptop fan has told you about it.
The model sees the desktop the hard way. `screen_capture` runs `scrot` and returns a JPEG of the 1280×800 display, at quality 70 with the pointer drawn in, as base64. Every action tool, `pointer_click`, `pointer_move`, `pointer_drag`, `pointer_scroll`, `key_type` and `key_press`, runs one `xdotool` command and then waits and takes another screenshot, so the model sees the result without spending a step to look:
```ts
// lib/agent/tools.ts:139-147
async function thenLook(agentId: string, done: T, settleMs: number) {
await new Promise((resolve) => setTimeout(resolve, settleMs))
try {
return { ...done, ...(await capture(agentId)) }
} catch {
// The action happened; only the look afterwards failed
return { ...done, image: null, mediaType: "image/jpeg", width: null, height: null }
}
}
```
The waits are fixed: 900 ms after a click, 400 ms after a move, 500 ms after typing, and 1,500 ms after Enter, because Enter usually submits a form. There is no accessibility tree, no DOM and no element ids. The model reads pixels and answers with coordinates. Pixels-and-coordinates is the most portable way to do computer use and the slowest, and the [arc-driver piece](/articles/arc-cua) showed how much of a desktop agent's wall time hides in exactly these fixed sleeps. A screenshot after every action also means every step carries an image, which brings us to the part of the runtime I liked best.
## Keeping forty screenshots out of the prompt
A tool loop resends the whole conversation on every step. If each step adds a screenshot, step 40 resends 39 of them, and the total input over the turn grows with the square of the step count. Xiryn's answer runs in `prepareStep`, before every model call:
```ts
// lib/agent/runtime.ts:273-275
const IMAGES_KEPT = 2
const OUTPUTS_KEPT = 6
const STUB_CHARS = 600
```
Walking the history from the newest end, the two most recent screenshots stay as images; every older one is replaced by its text caption and the words `[earlier screenshot omitted]`. The six most recent tool outputs stay whole; older text outputs are cut to their first 600 characters with a note of how much was trimmed. Saved history gets a similar pass on the way to disk, keeping only the screenshots the agent chose to look at or show plus the latest action's, so the database does not grow by megabytes a task.
The architecture doc makes a specific claim about the effect: "A long desktop session therefore costs roughly the same per step at step 40 as at step 4." It is close, and the widget shows where it bends. The rules are Xiryn's; the token costs are my assumptions and you can move them. With a 4,000-token fixed prefix (my estimate for a system prompt of about 5,700 characters plus 22 tool schemas), 1,365 tokens per screenshot (Anthropic's published width × height / 750 rule at 1280×800), and 60 tokens of call and caption per step, step 4 resends 6,910 tokens and step 40 resends 9,070. Without the pass, step 40 would resend 59,575. Over the whole 40-step turn that is 311,905 input tokens against 1,271,500, about a quarter.
So "roughly the same" is fair, with one honest caveat: the per-step cost still rises, by about 30% from step 4 to step 40 under these numbers, because every call and caption stays in the history forever. Compaction turns a steep line into a shallow one. It also changes what the model can know. After two more actions, the screenshot it just reasoned about is gone, and if it needs to compare a price on page one with a price on page three, it has to have written the first one down in its own reply. The instructions push it to do exactly that ("write what you found into your message — every option with its details"), which is a prompt doing the job of memory within a turn.
The 132k input tokens in the flight-search screenshot above is the same arithmetic summed over a real run. I can't tell the step count from the picture, but at roughly 7,000 to 9,000 tokens a step it is consistent with a run of fifteen to twenty steps.
## Memory is a list in the system prompt
Long-term memory is a table of notes per agent. `memory_write` inserts one; `memory_forget` deletes one by id. On every turn, `buildInstructions` pulls the agent's notes newest-first until a 12,000-character budget runs out and pastes them into the system prompt under `# Memory`, each with its id, and says how many older notes were dropped.
There is no retrieval, no embedding, no summarisation and no decay. A flat list is defensible for a personal agent whose memory is a few dozen facts about one person, and it fails in an obvious way once it is not: the 12,001st character pushes the oldest fact out, whatever it was.
When someone asked on X whether they would have to re-teach yet another agent, the author replied that Xiryn "learns as you work" and saves what is worth keeping "on its own". Mechanically, that is the model deciding to call `memory_write`, prompted by a paragraph in the instructions: "Save a note in the same turn you learn something worth knowing later … Saying you will remember is not the same as saving." It works as well as the model follows that paragraph. The [TencentDB memory piece](/articles/tencentdb-agent-memory) is a good contrast for what a memory system looks like when it is designed around what may enter a cached prompt; Xiryn rebuilds the whole prompt every turn, memory included, which is simple and cache-unfriendly in equal measure.
## A squad is a relay
You can put agents in a squad and address any of them. If the addressee decides another member fits better, it calls `delegate` with that member's id and a brief. The tool flips a `current` variable in the runtime, and on the next step `prepareStep` swaps both the system prompt and the model for the new agent's. Every tool resolves "the agent holding the turn" at call time, so after the hand-off the shell, the desktop and the memory are the new agent's.
It is a relay, one agent at a time, inside one turn. Nothing runs in parallel, and a request cannot go back to anyone who already held it: `held.includes(target.id)` throws "already had this request. Handle it yourself." That one rule prevents the ping-pong you would otherwise get from two polite models each insisting the other is better suited. It is small and it is right.
## The vault types what the agent never reads
Now the part that justifies the comparison with hosted agents that shop for you. A personal agent that books flights needs your login and your card. The question is who gets to hold them.
Xiryn keeps logins, cards, UPI IDs, personal details and one-time codes in a `vault_items` table. Each item is sealed with AES-256-GCM under a 32-byte key, with the item's id and kind bound in as associated data, so a ciphertext cannot be moved onto another row. The key is not in the data folder. It comes from the operating system: `security` and the Keychain on macOS, DPAPI through PowerShell on Windows, `secret-tool` and the Secret Service on Linux, or a `0600` file in `~/.xiryn` when none of those exist. If items exist and the key is missing, Xiryn refuses to make a new one rather than silently orphaning them.
The agent gets four tools. `vault_list` returns labels and masked hints ("Visa ···· 4242 · 12/29"). `request_secret` ends the turn with a form in the chat; what you type goes straight into the vault and the agent gets back an item id. `request_payment_approval` ends the turn with a card showing merchant, amount and site, which you answer with a saved card, by taking over the screen yourself, or with a decline, in the app or with buttons on Telegram. And `vault_fill` does the typing:
```ts
// lib/workstation.ts:469-478
// Types a vault value. It goes in on stdin, so it never shows up in a
// process list, a log or a tool result.
export function typeSecret(agentId: string, value: string) {
return execOrThrow(
agentId,
["xdotool", "type", "--delay", "12", "--clearmodifiers", "--file", "-"],
{ input: value, timeoutMs: 60_000 },
)
}
```
After typing, `xiryn-mask` connects to Chromium's debugging port inside the container and sets `-webkit-text-security: disc` on the focused field in every page and frame, so the card number shows as dots in the next screenshot and to anyone watching the live desktop. The tool returns `{ filled, item, site }` and no image. The value goes from the vault to `docker exec`'s stdin to the X server, and the model's context only ever holds a hint.
Before it types, `vault_fill` checks where the browser is. `currentPage` asks Chromium on `127.0.0.1:9222` for its open tabs and asks `xdotool` for the focused window's title, and picks the tab whose title matches. Then:
```ts
// lib/agent/tools.ts:392-409
if (item.kind === "login" && !hostMatches(where?.host ?? null, item.site)) {
throw new Error(`"${item.label}" is a login for ${item.site}, but the browser is on …`)
}
if (item.kind === "card" || item.kind === "upi") {
const approval = activeApproval(env.threadId, item.id)
if (!approval) throw new Error("… Call request_payment_approval first.")
if (approval.host && !hostMatches(where?.host ?? null, approval.host)) {
throw new Error(`The payment was approved on ${approval.host}, but the browser is now on …`)
}
}
```
`hostMatches` accepts the same host or a subdomain of it, on a dot boundary, after stripping `www.`. As a phishing check it is sound: `swiggy.com.login-help.example` does not pass for `swiggy.com`. I expected the payment side to be built the same way, as a one-shot token for one amount. Reading `activeApproval` and the fill path, it is a looser thing. Step through the scenarios:
Three behaviours fall out of the code, and none of them is visible from the README.
An approval is not spent by a fill. `activeApproval` returns any approved row for this thread and this card whose `expiresAt` is in the future, and nothing marks it used. For 20 minutes (`APPROVAL_MS`) the agent can fill that card on that site as often as it likes. I suspect it is on purpose, because a payment form that fails validation needs a second attempt. It still means you approved a card on a site for 20 minutes rather than one payment.
The amount is never checked. The agent writes the amount into `request_payment_approval` and you see it; nothing compares it with the page at fill time. The approval is honest if the agent was, and the README's phrase "every payment needs your yes" is true of the first payment in each 20-minute window.
An approval recorded with no site is not site-locked. The host check is guarded by `approval.host &&`. If `currentPage()` returned null when the agent asked for approval, because the debugging port timed out at 4 seconds or no web page was focused, the approval is stored with `host: null`, the card on Telegram shows no site, and the later fill skips the host check entirely. I reasoned this from the code; I did not run it. It is a one-line fix: refuse to create an approval without a host, or treat a null host as a mismatch.
There is a softer fourth. `currentPage` matches the focused window's title against the tab list and falls back to `pages[0]`, the first tab, when nothing matches. With several tabs open and a title that changed between the two reads, the host it checks may not be the page that will receive the keystrokes.
And then there is the limit that no check addresses, because it is architectural. The fill lands in a form field of a browser that the same agent controls through a shell in the same container, and Chromium listens on `127.0.0.1:9222` there; `xiryn-mask` uses that very port. An agent that has been talked into it by a malicious page can read the field back with one `Runtime.evaluate`. To the project's credit, `SECURITY.md` says this in so many words:
> A value typed into a page is in that page, and the agent controls the browser it's in. The vault keeps secrets out of the model's context and out of Xiryn's records, and makes their use deliberate and approved; it doesn't make a compromised agent harmless.
It is the right description of what the vault is. The README's "Agents never see the values" is the marketing statement of the same thing, and the difference matters exactly when the agent has been browsing pages you did not choose.
## Open by design
The container is where Xiryn's trade-off is most visible. The agent works as user `xiryn` with `NOPASSWD:ALL` in `/etc/sudoers.d/xiryn`, so it is root inside its box. The container has the default bridge network and full internet access, no `--cap-drop`, no `--read-only`, no `--pids-limit`, no seccomp profile beyond Docker's default, and Chromium runs with `--no-sandbox` because Docker withholds the kernel features its own sandbox wants. `SECURITY.md` describes the workstation user as "unprivileged". With passwordless sudo in the image, that word is wrong; the accurate claim is that the agent is root in a container that has no host mounts and no Docker socket.
I do not think the openness is a mistake. An agent that books flights and installs a package to finish a task has to reach the internet and has to be able to `apt install`. The cost is that the boundary is Docker's default isolation and nothing else, and the blast radius of a prompt injection is "anything a root shell with your home IP address can do, plus whatever is in that browser's cookie jar". The README's best line on the matter is accurate: sites see "an ordinary home connection rather than a data center". It is why the agent gets through on sites that block cloud IPs, and it is also whose IP any abuse comes from.
The [OpenMuse teardown](/articles/openmuse) is the useful mirror. OpenMuse runs its terminal in a container with `--network none`, `--read-only`, `--cap-drop ALL`, `no-new-privileges`, 512 MB, one CPU and 128 processes, re-checks 38 predicates on that container before every attach, and keeps the browser in a separate worker with its own egress filter. It also binds an approval to a hash of the exact action and spends it once. Xiryn chooses one container that can do everything, which is why it can do things OpenMuse cannot, and why its approval needs to be tighter than it is.
The phone take-over adds one more exposed surface. When an agent hands off at a login or a payment and Telegram is paired, `xiryn-tunnel` starts a Cloudflare quick tunnel to the container's noVNC for 30 minutes and sends you the link, with the desktop's password after the `#` so it stays out of the tunnel's logs. That puts a live desktop, already signed in to whatever the agent was doing, on a public `trycloudflare.com` address protected by an 8-character VNC password. VNC's classic authentication cannot use more than 8 characters anyway. The feature is on by default; `XIRYN_PHONE_TAKEOVER=off` removes it, and I would turn it off unless I needed it.
The Xiryn server itself has no accounts. `proxy.ts` refuses any request whose `Host` or, for writes, `Origin` header is not loopback, which defeats DNS rebinding from a page in your own browser. Telegram pairing uses an 8-character code from a 32-symbol alphabet that rotates after five wrong guesses, and the bot answers exactly one account. One-time codes you reply with on Telegram are deleted from the chat and stored once in the vault. These are careful.
## The claims, one by one
"Does everything Muse, Grok Bot and Dots do." I can't check what those hosted products do from here, and the README's comparison table is about where things run, not what they can do. The README's own roadmap is the honest answer: scheduled and recurring tasks, MCP connectors for email, calendar and docs, more remote channels, and voice out are listed as not shipped. Xiryn today does chat, a desktop, a shell, web research through Firecrawl or its own browser, memory, squads, and Telegram. Without scheduling and without mail or calendar access, it is not yet a personal assistant in the sense of one that acts while you sleep.
"On your own computer, without ever getting in your way." The tools execute only in containers, and nothing from your machine is mounted into them. That holds. Each agent's box defaults to 2 CPUs and 4 GB, and two agents working at once is 4 CPUs and 8 GB of your laptop, which the FAQ admits. Running Xiryn itself under `docker compose` mounts `/var/run/docker.sock` into its container, which the compose file correctly calls root-equivalent.
"Your data stays on your disk." Conversations, memory, settings and the vault do. Everything the model reads goes to the model provider you configured, screenshots included, unless you point it at Ollama or LM Studio with a model that can both call tools and read images.
"Agents never see the values." By default, in the model's context, yes, and the stdin path plus the mask is well built. Against an agent that decides to look, no, and `SECURITY.md` agrees.
"Every payment needs your yes." Every first payment in a 20-minute window on one site does. Subsequent fills, different amounts and approvals with no recorded site do not ask again.
## Where it sits among the agents on this site
The agents I have read for this site fall on one axis: whose computer the agent is using.
| | where tools run | browser | approval to pay or send |
|---|---|---|---|
| Xiryn | one Docker desktop per agent, root, full network | its own Chromium in that box | card + site + thread, 20 minutes, reusable |
| [OpenMuse](/articles/openmuse) | sealed container, no network; separate browser worker | its own Playwright profiles | hash of the exact action, 30 minutes, spent once |
| [BrowserSkill](/articles/browserskill) | your machine, through any harness | your real, signed-in Chrome | consent switch kept in browser storage the agent cannot reach |
| OpenClaw, [Hermes Agent](/articles/openhuman) | your machine | your apps and files | per the harness |
Xiryn's position is the middle one the README draws: not your desktop, but a whole desktop. It is the right shape for the one thing it does best, which is a visible, interruptible agent working real websites from your own IP while you watch and take the wheel. The take-over and hand-back flow is the most finished part of the product: `request_takeover` ends the turn, the UI opens the desktop with input enabled, and handing back sends a message flagged `handback` that tells the agent to take a fresh screenshot before it trusts anything it remembers about the screen.
Would I run it? On a spare machine, for research and for clicking through sites I would otherwise click through myself, yes. The runtime is clean, the compaction pass is a good idea worth copying into any screenshot-driven loop, and the vault's design, values out of the model's context and typed from stdin, is better than most agent projects bother with. I would not save a card in it until an approval is spent on use and refused without a site, and I would switch phone links off. Those are small changes against a codebase that is two days old, and the author ships fast.
## How I checked
I shallow-cloned `SouravBeraAkaSpeed/xiryn`, then fetched the full history, and read it at `ae9ad97` (12 October 2026, version 0.4.1): the runtime, the tool belt, the instruction builder, the workstation and Docker modules, the image's Dockerfile and its scripts under `workstation/rootfs`, the vault and its server actions, the Telegram bridge's pairing and code paths, `proxy.ts`, `SECURITY.md`, the README, the architecture doc and the changelog. Line counts are `wc -l` over `.ts`, `.tsx`, `.mjs`, `.sh` and `.html` files; commit counts and timestamps are from `git log`. File and line references in the code blocks are to that commit.
I did not run Xiryn, did not build its image, and did not execute any of its code. The behaviours of the vault in the widget are read from the code paths named in each row; the null-host and title-fallback cases in particular are reasoned, not observed. I also did not test whether a workstation can reach the Xiryn server on the host. On Docker Desktop that route would be `host.docker.internal`, and `proxy.ts` checks a `Host` header that any client can set, so that is the first thing I would probe. Token costs in the compaction widget are my assumptions, stated in the widget; the 132k and 221k figures come from the project's own screenshots. The four screenshots are the README's, resized; the video is the post's, trimmed to 60 seconds and re-encoded without sound. I could not check what Muse, Grok Bot or Dots do.
---
# fsearch: whole-disk search on a Mac, and what the 17,000x is measured against
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/fsearch
> date: 2026-10-08
> tags: rust, search, performance, developer-tools, benchmarks
Noah Dunnagan posted a terminal recording on October 8: a search box, every keystroke answered across the whole disk of his M4 Max in a few milliseconds, "~60mb of ram". Later the same day he open-sourced it as [fsearch](https://github.com/noahdunnagan/fsearch) with the line "Full disk search, in milliseconds. 17,000x faster than Finder." A follow-up post in the same thread sets the tone better than any launch copy could: "this isnt mean to compete with fff or really any other file search. I was just curious what could be done for funzies."
I took him at his word and read it the way you read a friend's weekend project: all of it, with interest, and with the numbers checked. Two things surprised me. The name search, the part that answers in a millisecond, uses no clever index at all. And the 17,000x is not measured against Finder.
The repository is about 5,000 lines of Rust in ten files under `src/`, MIT-licensed, 27 commits written between October 5 and October 8. It is macOS only, and says so: the FSEvents bindings link Apple frameworks, which is why the first person in the replies who tried `cargo build` on Ubuntu got `link kind 'framework' is only supported on Apple targets`. I could not run it on this Linux machine, so everything below comes from reading the code, the two demo videos, the benchmark script and its JSON, plus one small measurement of my own that I describe at the end. There is no test suite in the repo; the earlier README instead reports accuracy measured against a fresh fd listing of the disk, with 99.95% of 2,000 random entries found by name.
## The number on the last frame
The launch video ends on this card. Everything else in this section is about the bar chart in it.
The 17,789x is fsearch against [fd](https://github.com/sharkdp/fd), not Finder. 62 seconds divided by 3.5 milliseconds is about 17,700; the printed figure presumably comes from unrounded times. The content number is the same shape: grep -r, killed after 7.7 minutes, against 8.2 ms.
So "faster than Finder" is a loose word for "faster than walking the disk". That matters because the comparison is between two different kinds of work. fd has no index. Every query reads every directory on the volume, 7.6 million entries, then filters. fsearch read those directories once, about 20 seconds by its own README, and every query after that touches a file it already holds. The ratio between "read the disk" and "read my copy of the disk" has no ceiling; it grows with the disk. Comparing an index lookup to a crawl tells you the index works. It does not tell you the index is fast compared to other indexes.
Two more details make the fd bar longer than it would be on most Macs. An earlier version of the README, at commit `f43261b`, measured "`fd` for one name, no index: 25.9 s" on the same machine, so the 62 s in the video is a slower run of the same thing, and the repo contains no script for it, so I can't tell how fd was invoked. And this Mac is unusually hostile to crawling. A comment in `src/walk.rs:104-107` explains:
```rust
// Children are opened with openat() relative to the parent's fd, so paths
// never get rebuilt and PATH_MAX never bites. Measured on this Mac: open() +
// close() is ~19us per directory (two Endpoint Security clients tax every
// open), getattrlistbulk ~14us; past ~8 threads the kernel side stops scaling.
```
Two Endpoint Security clients (the old README names MDM and a VPN) inspect every `open()`. A crawler pays that per directory, an index lookup never does. fsearch's own first crawl takes about 20 s under the same tax, roughly a third of fd's 62 s, which says something good about how it crawls (next section) and something about how much of fd's bar is the machine.
What would Finder actually be? Finder's search window and `mdfind` both query Spotlight, which is itself an index, kept live by the same kernel change stream, and which by default matches file contents and metadata as well as names. That is the comparison the tweet implies: index against index, names against names. Nobody has run it in public, I can't run it here, and I would expect Spotlight's `mdfind -name` to land in the tens to hundreds of milliseconds rather than a minute. That would still be a large win for fsearch. It would not be four orders of magnitude.
The headline compares a sports car to walking. The car is still nice, and the rest of this piece is about how it is built.
## Crawling without a stat per file
The usual way to list a tree is `readdir` for the names, then `stat` on each one for its size, type and time. On 7.6 million entries that is 7.6 million extra syscalls. macOS has a better call, `getattrlistbulk(2)`, that returns a buffer of entries with the attributes you asked for already attached. fsearch asks for exactly what the index stores (`src/walk.rs:165-170`): name, object type, modification time, BSD flags (for the Finder's hidden bit), the mount status of directories, and the data length of files. One syscall fills a 256 KB buffer with hundreds of entries.
Directories fan out over a rayon pool of 8 threads (`SCAN_THREADS` in `src/engine.rs:25`, chosen because the kernel side stops scaling past that). Each child is opened with `openat()` against its parent's descriptor, so no path string is ever rebuilt during the crawl. Mount points are flagged and not crossed; firmlinks are, which is how `/Users` on the data volume shows up under `/` exactly once (`src/walk.rs:1-6`).
Two small things in here show someone who has been bitten by macOS. `main.rs:81-84` and `engine.rs:110-118` call `setiopolicy_np` with `IOPOL_TYPE_VFS_MATERIALIZE_DATALESS_FILES` off, so that listing an iCloud-evicted folder fails fast instead of silently downloading it. And without Full Disk Access, the walker skips the consent-gated folders (Desktop, Downloads and friends) instead of opening them, because opening one pops a privacy prompt and blocks the call until somebody clicks (`src/walk.rs:16-19`). A crawler that hangs on a dialog nobody can see is a classic background-daemon bug, and he avoided it.
This is the macOS answer to what [Everything](https://www.voidtools.com/) does on Windows. Everything reads the NTFS Master File Table directly, a flat array of every file record on the volume, which is why it can index a Windows disk in seconds. macOS has no public equivalent of reading the APFS catalog wholesale, so fsearch does the next best thing: the fattest directory-listing syscall there is, in parallel.
## One file, laid out folder by folder
The whole name index is one blob, written to `index.bin` and read back with `mmap`. The format is sixteen flat arrays, each a section of the file (`src/index.rs:22-41`). There are no pointers, no tree nodes and no hash table on disk.
The layout trick is in the module comment, `src/index.rs:3-7`:
```rust
//! Layout trick: entries are emitted one directory *block* at a time, blocks
//! in depth-first order. So every directory's children are contiguous (and
//! sorted, for path lookup), and every directory's whole subtree is the single
//! range `dir_start..dir_end`. Scoping a search to a folder is a range bound,
//! not a filter.
```
Pick a folder below and watch the range. The green rows are the folder's own children, one sorted block, which is what lets `lookup()` resolve a path with a binary search per component (`src/index.rs:183-200`). The blue rows are everything beneath it.
Because a folder's block is emitted and then its child folders' blocks are pushed on a stack before any sibling's, everything under a folder ends up between two numbers. `in:~/Developer` becomes "scan entries 16 to 32" in the toy and "scan this slice" on the real disk. There is no per-entry path check. `Searcher::scope_range` in `src/query.rs:670-676` is four lines: look the folder up, read `dir_start` and `dir_end`, done.
The second trick is interning. On Noah's disk, 7.5 million entries share about 2 million distinct names (`src/index.rs:9-11`): every `README.md`, every `index.js` in every `node_modules`, every `Info.plist` in every app bundle. Each distinct name is stored once, with a precomputed 64-bit mask, and each entry holds a 4-byte name id. The query engine scores names, not entries, so `index.js` is scored once even if it appears 40,000 times.
### What an entry costs
The section sizes are in `section_lens` (`src/index.rs:449-451`), so the memory per file can be read straight off the code:
| Per | Bytes | What |
|---|---|---|
| entry | 21 | name id 4, kind and flags 1, parent folder 4, size 4, mtime 4, slot in the name-to-entries list 4 |
| folder | 21 | entry index, block start, block length, subtree end, parent, all 4 bytes, plus a 1-byte location prior |
| distinct name | 16 + its bytes | mask 8, offset 4, entries-list offset 4 |
Size fits in 4 bytes by going lossy above 2 GiB (`src/index.rs:453-460`): exact below, 2 MiB granularity above, which is fine for a `size:>1gb` filter.
To see what that comes to on a real tree I listed `/usr` on this Linux box: 189,110 entries, 20,155 folders, 99,825 distinct names averaging 15 bytes. The formula gives about 7.5 MB, around 40 bytes per entry all-in. Noah's disk interns harder (2 million names for 7.5 million entries, against roughly half on `/usr`), and the old README reports "280 MB names" on disk for it, which works out to about 37 bytes per entry.
The daemon's reported memory is a different number: 30 to 135 MB in the README, 50 MB on the benchmark card. The difference is not a contradiction. After a build, `src/engine.rs:492-493` re-maps the index from the saved file "so the index is clean, evictable page cache rather than anonymous memory." macOS's `footprint` tool, which is what the benchmark script calls (`demo/vs_fff.py:19-22`), charges a process for memory it has dirtied, not for clean pages mapped from a file. So the 280 MB index is in RAM when you search, but it shows up as file cache, and the kernel can drop it under pressure and fault it back in. That is a legitimate design and the right one for a daemon. It also means "50 MB" and "358 MB" in the fff comparison below are not the same kind of number.
## The name search has no index
I expected a trigram index for names. [plocate](https://plocate.sesse.net/) uses one over whole paths, and fsearch already has one for file contents. Instead, the name search is a brute-force pass over every distinct name, made cheap by a filter that costs one AND.
Each name's 64-bit mask records which character classes it contains: one bit per letter (case folded), one per digit, one each for `.`, `-` or `_`, space, non-ASCII and anything else. That is 41 bits (`char_bit`, `src/index.rs:509-520`). A fuzzy query token can only match a name that contains every one of its letters, so the test is whether the token's mask is a subset of the name's: `token.mask & !name.mask == 0`. For most queries that rejects most of the disk without reading a single name byte.
The spare high 23 bits hold a hash of the first letter of the name and of each space-separated word in it (`name_mask`, `src/index.rs:496-506`). That is for typos, and it lets the whole test stay branch-free (`src/query.rs:30-39`):
```rust
/// Can a name with mask `m` (`index::name_mask`) match? Cleanly it has
/// every char class; with a typo, a word in it starts with this token's
/// first letter and at most one `loose` class is missing. Branchless, so
/// the scan over every name stays vectorized.
#[inline(always)]
fn fits(&self, m: u64) -> bool {
let miss = self.mask & !m;
(miss == 0) | ((((miss & !self.loose) | (miss & miss.wrapping_sub(1))) == 0) & (m & self.start != 0))
}
```
Read it as two cases OR'd together. Either nothing is missing. Or exactly one class is missing (`miss & (miss - 1) == 0` is the power-of-two test), it is not the first letter's class (`loose` excludes it), and some word in the name starts with the query's first letter. Only names that pass go to the real scorer, an fzf-v1-style matcher (`src/query.rs:405-471`) that finds the leftmost-ending subsequence, shrinks it from the right, and adds bonuses for word boundaries, camelCase and consecutive runs, plus a big bonus when the query is the whole name or its stem.
I wanted to know how much the mask actually rejects on a real tree, so I ported `char_bit`, `name_mask` and `fits` to C and ran them over the 99,825 distinct names from `/usr`. For `json` 0.78% of names survive the mask; `zlib` 1.21%; `config` 2.94%; `python` 4.78%; `readme` 5.94%; `main` 8.00%. A one-letter query is the worst case, and `x` lets 17.00% through. The mask pass ran at about 1.5 ns per name on one core of a busy shared machine, against about 57 ns per name for a naive subsequence check over every name. At 2 million names that is around 3 ms on one thread for the mask, split across the M4 Max's cores, which is consistent with the 1.3 ms median the README reports.
After the names are scored, the query has to get back to entries, and there are two paths (`src/query.rs:686-701`). If the matching names cover at most 60,000 entries (`SELECTIVE`, line 660), fsearch walks a name-to-entries list, a second CSR-style array in the index, and touches only those entries. Otherwise it does one sequential pass over every entry in scope, looking each one's name id up in the scored table. Folder tokens (`dev main` meaning "a `main` somewhere under something matching `dev`") are resolved with a per-folder memo folded down the tree in parallel.
One more thing makes typing fast. The scored name table is cached, and when the new query only narrows the last one (`mai` becoming `main`), the scorer revisits only the names the previous query matched (`NameKey::narrows`, `src/query.rs:1053-1070`). With one exception that shows the author thought about it: "Gaining a typo widens the match: score afresh." A four-letter token takes no typos, and a five-letter one does, so `mian` to `mianr` cannot reuse the narrower set.
Try it on the toy disk. Each lane runs the same query through a different structure. `pyhton` and `wllpaper` show why the trigram lane is the wrong tool for this job: a typo or a fuzzy gap leaves no contiguous three-letter run to look up, so the index never proposes the right file. The sorted array is fast and almost useless, since binary search can only answer a prefix.
The toy is too small for the costs to mean much: it has 41 entries and 38 distinct names, so interning saves almost nothing, and a one-letter query like `a` lets so many names through the mask that the fsearch lane costs more than the plain scan. The shape is what carries over. The mask pass is a linear scan, but over the smallest thing that can answer the question, with the cheapest possible test first. Trigram indexes win when the query is a literal substring and the corpus is huge; fsearch wants fuzzy, typo-tolerant matching over a few million short strings, and for that a vectorised scan is simpler and fast enough.
### Typos at word starts only
Words of five or more letters forgive one typo: a wrong letter, an extra one, a missing one, or two letters swapped (`one_edit_prefix`, `src/query.rs:533-555`). Digits are never edited, because `hat_18` and `hat_98` are different files. The typo has to sit inside a word that starts with the query's first letter, and the comment at `src/query.rs:505-510` gives the reason:
```rust
/// ... Other word starts (`_`, `-`, camelCase) would cost
/// a scan of every name per query, ~10x the price.
```
That is a deliberate trade, and the first letter is the price: `mian.rs` finds `main.rs`, `pyhton` finds `python3`, but `ython` with a dropped `p` finds nothing. The benchmark script knows this, which matters in a moment.
## Staying current: diff a folder, never trust an event
The index is immutable once written. Changes go in a `Live` layer on top (`src/live.rs:1-7`): a bitset of base entries that are gone, plus an overlay map of entries added since. Every search scans the overlay too, at about 1 ms per 100,000 entries, so once 50,000 changes pile up (or every 12 hours), the overlay is folded into a new base: about 1 s of CPU and a 280 MB write (`src/engine.rs:19-24`).
Changes come from FSEvents, watching `/` at directory granularity with a 0.1 s latency (`src/fsevents.rs:84-86`, `src/engine.rs:308`). The handling is the part I like most. An event never says what changed, only which folder. fsearch lists that one folder again and diffs the listing against what it holds. Because the diff is idempotent, duplicate events, events replayed from history after a restart, and events racing a compaction are all harmless. That is how a new file shows up in about 0.1 s.
Restart is cheap for the same reason. The index header stores the last FSEvents id it applied, and FSEvents can replay history from an id, so a restart reloads the mmap'd file in milliseconds and replays only what changed while it was down. If the history is gone (FSEvents drops it, or the gap is too long), it does not recrawl. Adding, removing or renaming an entry bumps its folder's mtime, so one `lstat` per folder, in parallel, finds the folders to relist (`src/live.rs:320-324`). The old README timed that recovery at 8 to 13 seconds instead of a full crawl.
## Contents: here is the trigram index
File contents do get a trigram index, built the way Russ Cox described for Google Code Search, and the same idea our [tgrep teardown](/articles/tgrep) covers in depth. Every text file under your home folder is broken into case-folded three-byte runs; each run maps to the list of documents that contain it. A query becomes an AND/OR tree of trigrams. `regex-syntax` parses a regex into its HIR and the planner (`src/content.rs:1102-1209`) works out which literal strings every match must contain. The posting lists are intersected, and the surviving candidates are read fresh from disk and matched for real (`src/content.rs:1-8`), so results are never stale even when the index is a couple of seconds behind.
The storage is careful. Segments are immutable mmap'd files; a posting list is delta varints, or a bitset over the segment's documents once more than one document in 8 contains the trigram (`src/content.rs:33-35`, `:559-574`). Small segments merge in tiers of 8. Files are debounced per folder: indexed 2 s after the folder goes quiet, or 5 minutes after the first change if it never does, so a log written every second costs one reindex per 5 minutes (`src/engine.rs:398-401`). `sym:` is a nice extra: identifiers that follow a declaring keyword (`fn`, `def`, `class`, `struct` and about twenty-five more) are hashed into the same key table above the 24-bit trigram space, so "where is `apply_dir` defined" is a single posting list (`src/content.rs:287-325`).
The scope is narrower than "whole disk", and that is a reasonable choice that should be said out loud. Only files under `$HOME` are content-indexed. A file over 1 MiB (`MAX_FILE`, `src/content.rs:27`) is skipped, as is anything whose extension is not on a list of about 130 text types, anything with a NUL byte in its first 8 KB, and anything under `node_modules`, `.git`, `target`, `build`, `dist`, `vendor`, `Library`, the caches and about forty other names (`src/content.rs:37-72`). Outside the indexed area, `grep:` with `in:` picks files from the name index and reads them, which is slower but correct.
## Against fff, read closely
The repo's real comparison is with [fff](https://github.com/dmtrKovalenko/fff), Dmitriy Kovalenko's file finder: it began as a Neovim plugin, is now an MIT library and MCP server that powers file search in opencode and others, with typo-tolerant path and content search, frecency ranking and a background watcher. Someone in the first thread said "just use fff", and the benchmark was the reply. Unlike the fd card, this one has a script, `demo/vs_fff.py`, and its output is committed.
The method is decent. Both engines get the same 300 target names, picked at random from files fd also sees, each unique in the tree, then four typo variants of each, 1,500 queries in all; the engines alternate going first; content search uses 67 patterns. fsearch is even handicapped: it is timed through its Unix socket with JSON on both sides (`vs_fff.py:89-95`), while fff is called in-process through its Python binding. The name-search row, 1.05 ms against 13.8 ms at the median, I believe as stated. On the Linux kernel tree (95,939 files) the same script gives 1.15 ms against 1.04 ms, which the README honestly calls a tie.
Three rows need context.
The typo row tests the kind of typo fsearch handles. The generator's docstring, `vs_fff.py:46`: "One typo on a letter, never the first char (fsearch needs that one)." That is disclosed, and it is fair to test the common case, but a typo in the first letter would score zero for fsearch by design. The 98% against 88% is "typos after the first letter".
"Ready after launch" compares two different things. For fsearch it is the time from a killed daemon to the first answer, with a saved index already on disk: load the mmap, answer. For fff it is the time to scan the folder from nothing, 2.5 s, and 12.8 s until its content cache is warm. fsearch's equivalent would be its first crawl, about 20 s for the whole disk. Both numbers are true; they measure different events.
The memory row compares footprints, and as covered above, fsearch's index lives in clean file-backed pages that `footprint` does not count, while fff holds its index in anonymous memory. "7x less" is real in the sense that matters under memory pressure (fsearch's pages can be dropped and re-read), and not in the sense of bytes resident while you search.
And one honest disclosure from the README itself: fff finds about 9% more files with content matches on Chromium (353,276 against 323,688), because fsearch skips `build/`, `vendor/` and some file types. Faster partly because it reads less.
## Where it sits
| | How it learns the disk | Name matching | Stays current | Contents |
|---|---|---|---|---|
| fsearch | parallel `getattrlistbulk` crawl, once | mask filter, then fuzzy scan of distinct names, one typo | FSEvents, relist and diff the folder | trigram index under `$HOME` |
| Everything | reads the NTFS MFT | substring, wildcards, regex | NTFS USN change journal | not indexed by default; scans on demand |
| plocate | `updatedb`, a periodic full crawl | substring over paths, via trigrams | not live; as fresh as the last `updatedb` | none |
| mlocate | `updatedb` | linear scan of the whole database | not live | none |
| fd | none, walks every time | regex or glob | always current, by construction | none |
| fzf | none, filters whatever you pipe in | fuzzy scoring | n/a | none |
| fff | per-folder scan in a long-running process | typo-tolerant fuzzy, frecency | background watcher | in-memory content index |
| Spotlight (Finder, `mdfind`) | `mds` importers | names, metadata and contents | the same kernel change stream | yes, with importers |
The closest relative is Everything: same idea of a name index that is effectively free to query, built from the filesystem's own bulk metadata and kept live from the kernel's change stream. fsearch adds fuzzy and typo matching that Everything does not do by default, and a content index. It is also a fresh take on something people have built many times, which a few repliers pointed out with less warmth than they could have.
## What I'd take from it
If you build search for a few million short strings, the main lesson is the order of operations. Intern first so you score distinct values. Put the cheapest possible test in front of the expensive one, and make it a bitmask so it vectorises. Reach for an inverted index only when the query is a literal and the corpus is large, which is file contents, not file names. The layout trick, emitting blocks depth-first so a subtree is a range, is worth stealing for anything tree-shaped you need to scope.
On the claim: "17,000x faster than Finder" should read "about 17,700x faster than fd on a Mac whose security agents tax every directory open". fsearch's first-crawl time on that Mac is the honest bound on what an unindexed tool can do there, and against fff the honest summary is "about 13x faster on name search in a big tree, a tie in a small one".
Would I use it? On a Mac, as a launcher backend or an agent tool, yes; its JSON-lines socket and the crate API make it easy to wire in, and the owner/follower design (the first process owns the index files, others follow and take over when it exits) is thoughtful for that. On Linux it needs a port: fanotify or inotify instead of FSEvents, `getdents64` and `statx` instead of `getattrlistbulk`. The index and query code would carry over almost untouched.
## October 10: seven times faster than itself
Two days after the launch Noah posted again: "I made fsearch 7x faster. Find any file, in microseconds." The link goes to the same repository, which by then had 110 commits on top of the `af9476d` I read: 103 of them his, the rest from three outside contributors, +5,775 / −1,350 lines, almost all of it in `query.rs` and `content.rs`.
The first thing I wanted to know was what the 7x is measured against, since the last headline number was measured against fd. This time it is fsearch. The README's table compares the current build with commit `76d612f`, the October 9 merge that came seven commits after the one this article read. Its search code differs from `af9476d` only by a `type:app` fix for symlinked apps and some formatting. Same M4 Max, the same 8.3M files, medians, and two words that the launch card never had: "Same results."
This is a better kind of number than the launch card's, and the repository now ships tools to make one. `examples/rig.rs` samples a fixed query set from a frozen copy of an index (sixty unique names of 6 to 32 letters, a typo of each, common short words, folder-plus-name pairs, twenty filter queries, typing runs, and content searches), times them in process, and hashes every answer. `demo/abba.py` runs two builds in A B B A order so that a busy machine's drift hits both equally. `examples/e2e.rs` does the same from outside the daemon, through its socket or `fsearch stdio`. Most of the 110 commit messages paste their own A/B table and end with "0 queries with different results". The run behind the README table is the one thing that's missing. There is no corpus, no output and no note of which harness produced it, so I can follow the 7.7x through the commit messages but I can't reproduce it.
Read plainly, the rows say: name search went from 1.0 to 0.13 ms, typing a whole filename from 23 to 3.0 ms, content search from 14 to 2.3 ms, and the CLI from launch to answer from 5.5 to 3.3 ms. The "microseconds" are the 130 µs inside the daemon. Typed into a shell, a search still costs about 3 ms, almost all of it launching a process. The side-by-side in the same video uses the Chromium tree and shows a smaller gap per keystroke, 0.23 against 0.48 ms at the median.
### Where the 7x came from
I expected one trick. Instead there are dozens of commits, each with its own measured ratio. For names, three of them carry most of the gain.
The first turns the mask into an index. At `af9476d` each distinct name carried one 64-bit mask, and a query tested every mask in turn. Commit `2572bbd` turns that table on its side. For each of the 41 character classes, the index now stores a bitmap over every distinct name, one for names with that class in their first half and one for their second half. A query reads 64 names per machine word, rarest class first, and stops as soon as a word goes to zero. The two halves add a test the old mask could not express. A fuzzy match keeps the query's letters in order, so some prefix of the query has to fit the first half of the name and the rest has to fit the second. A later bitmap marks the classes a name contains twice, so `readme`, with its two e's, skips every name that has only one. The commit message says the order test "rejects ~93% of the names the old char-class mask let through", and puts name-scoring CPU at 6 ms before and 0.3-0.7 ms after.
I ported the new filter (`TokenBits::words` in `src/query.rs`, typo path included) to C next to my old port of `fits`, and ran both over the same 99,827 distinct names from `/usr`. My old port reproduces the earlier numbers exactly, which is a good sign the new one is faithful.
| Query | Old mask lets through | New bitmaps let through |
|---|---:|---:|
| `json` | 0.78% | 0.54% |
| `zlib` | 1.21% | 0.18% |
| `config` | 2.94% | 1.65% |
| `python` | 4.78% | 4.15% |
| `readme` | 5.94% | 1.98% |
| `main` | 8.00% | 4.67% |
| `x` | 17.00% | 17.00% |
On this tree the new filter removes between none and 85% of what the old one let through, depending on the query, not 93%. His figure comes from his own query set of long, unique filenames, which is where an order test bites hardest. A one-letter query gains nothing by construction. The speed per name changes more than the counts do. The old filter took 1.3 to 1.6 ns per name on one core of this busy machine, and the bit-sliced one takes 0.08 to 0.44 ns, because a single AND now covers 64 names.
The second change stops broad queries from visiting everything. Commits `69509bf` and `e1c758d` give each distinct name four bytes (`NameInfo`) that bound how well any entry with that name could score, without reading the name. Names are visited best bound first, in batches, and the visit stops once nothing left can beat the 50th hit. A query like `a` or `add` used to walk all 8.3 million entries; now it scores a few dozen names. Another commit, `a21af25`, keeps per-first-letter lists, so a 1- to 3-byte token never touches the bitmaps at all.
The third is one I would not have thought of. Commit `1a14bb8` notes that after about 20 ms of idle, macOS clocks the cores down, and the next search runs 3 to 6 times slower while they ramp back up. Every keystroke of someone typing pays that. So after it answers a client that is still connected, the daemon now spins one thread for 250 ms. Keystrokes 120 ms apart went from 1.442 to 0.525 ms each. The commit is just as specific about the bill: "+20.8 s daemon CPU over a 26 s session typed at 120 ms gaps". In other words, one core stays busy for as long as you type. For a launcher I think that is a fair trade, but on battery it is a real cost, and it is easy to miss when the release only says "faster".
The rest is smaller and well documented: NEON compares in the fuzzy scorer, JSON written straight from the hits (20 to 30 µs), `target-cpu=native` (0.92x) and a hoisted vector load (0.85x), both from an outside contributor, and `pgo.sh`, a profile-guided build trained on your own index that makes name searches 6 to 37% faster. The README's build line is plain `cargo build --release`, and the table doesn't say whether its "now" column used the PGO build.
Content search got per-file bloom filters over 5- and 7-grams, so a search can skip files that contain the right trigrams but not the text. That is where "slowest 10% inside files: 38 to 2.3 ms" comes from, and it is why the Chromium index on disk grew from 0.32 to 0.76 GB. Identical files are now indexed once.
### What it changes above
The heading "The name search has no index" is out of date. There is still no trigram index for names; trigrams are still for contents only. But per-class bitmaps over every distinct name are an index in the plain sense: a precomputed structure that answers "which names could match" without visiting each name. It is closer to a bit-sliced signature file than to an inverted index of n-grams. The mask section, the C numbers in it and the race widget all describe `af9476d`.
Memory per entry is unchanged at 21 bytes. Per distinct name it about doubled. The 8-byte mask gave way to 148 bitmap bits (18.5 bytes), plus 4 bytes of `NameInfo`, a 1-byte location prior and a 1-byte extension slot, so with the two 4-byte offsets that comes to 32.5 bytes plus the name's own bytes, up from 16. On `/usr` that moves my estimate from about 7.5 MB to 9.1 MB, or from about 40 to 48 bytes per entry. His own commit puts the step from mask to bitmaps at 17.7 MB to 29.7 MB on his disk.
The 17,000x is gone from the README, and nothing has replaced it. The new claim compares fsearch with itself, on hashed answers, with a harness in the repository, and that is the comparison I asked for.
The fff comparison had a short second life. Commit `ec25ffa` reran `demo/vs_fff.py`, which was unchanged: on Chromium, 0.21 ms against 15.4 ms by name; on the Linux kernel, 0.16 against 1.8 ms. The tie I described above had become an 11x win. Seven commits later, `ffcb37d` ("Remove the fff comparison") deleted the table, the script, both JSON files and the video, with no reason given. One detail from the rerun is worth keeping. fff's own median on the Linux kernel went from 1.04 to 1.78 ms between the two runs, a 70% swing on the side that none of fsearch's commits touched. That is how far a single run of that script can move. The "Against fff" section above cites files that are no longer at the head of the repository; they are still in its history at `af9476d`.
## October 13: 0.2, where speed stopped being the point
Two days later came a version number. "FSearch 0.2", Noah posted on October 12, with five lines under it: 150x faster to grep a file you just edited, a 74% smaller index ("for me 1.5g -> 388mb"), 35% more files covered, 65% less daemon memory on Chromium, and 32% less peak memory while indexing. Someone in the replies asked when it would hit sub-picosecond latency, and his answer explains the whole release: "Once in nanosecond I see no point in speed being faster. This version actually regresses by .4x in some areas when it comes to speed. Primary focus is better system impact and ability."
I liked that before I had read a line of it. After two rounds of headline multipliers, a release that says what it gave up is easier to trust.
The repository has tags now. `v0.2.0` is a single commit, `e96ebeb`, that adds 9,436 lines and removes 1,142 across 24 files, including four new modules (`alloc.rs`, `frecency.rs`, `git.rs` and a 1,338-line `mcp.rs`). The README's "before" column is commit `b88f4e9`, which is the parent of `ffcb37d`, the commit the October 11 update read, so for once the baseline is code this article has already been through. `v0.2.1` followed the same afternoon with 16 small fixes credited to @domfahey's issues, and a few outside pull requests, one of which (Leonhard Breuer's, #25) frees an FSEvents context the old code leaked on every stream restart. The numbers below all come from 0.2.0's own boards, so that leak fix is not in them.
One more thing from the replies. Asked to rerun the fff benchmark, Noah said he'd talked to Dmitriy, fff's author, and that comparing the two "is not the best thing... My project is just meant to replace finder nothing more!" That explains the commit that deleted the comparison, which I couldn't explain on the 11th.
### 150x: the edit you just saved
The demo appends `fsdemosamq9s` to `init/main.c` in the Linux tree and greps for it every few milliseconds. 0.1 finds it after 2.06 s, 0.2 after 13.8 ms, which is 149x.
The 2.06 s is no accident. The content worker debounces each folder until it has been quiet for two seconds (`CONTENT_QUIET`, `src/engine.rs:26`), and that constant is still 2 s in 0.2. I expected the fix to shorten that wait. Instead the index still waits, and grep stops waiting for it.
When a batch of FSEvents arrives, the worker now does one extra thing before it files the folders away for later. It asks the name index which text files those folders should hold right now, compares each one's size and mtime with what the content index holds, and puts the ones that differ on a list (`Dirty::mark`, `src/content.rs:2832-2880`). Every grep takes the files on that list that pass the query's filters, newest first, and reads them before it touches a posting list (`search_with`, `src/content.rs:2733`). The candidates that come out of the index then skip any path already read. When the two-second sync lands, the folder's entry is cleared. A folder with more than 4,096 changed files (`DIRTY_MAX`) only gets the first 4,096 on the list; the rest wait for the sync, which is the right bound for a `git checkout` that rewrites half a repo.
This works because of a decision from the very first version. Matches were always read fresh from disk, and the index only picks candidates, so reading a few extra files directly costs no correctness at all. The only thing that was ever stale was the candidate list, and the dirty list patches exactly that gap.
What's left is FSEvents itself. The stream's latency dropped from 0.1 s to 10 ms (`src/engine.rs:550-553`), and the comment says why: "a change shows in ~12 ms (fseventsd's own delay); at 0.1 s, one made within 0.1 s of another (save then rename) took ~100 ms". 10 ms of latency plus fseventsd's own delay is about the 13.8 ms on screen. The same change is why the README's "a rename or delete visible" row went from 45 and 47 ms to 18 and 15.
The README and the closing card print a different number for the same row, 2.1 s to 1.2 ms (1.3 ms in the README), labelled "in-process". I couldn't work out what it measures. The rig in the repo, `examples/fresh.rs`, starts the clock at the write and polls every 5 ms, and the code can't see a write before FSEvents reports it, which takes longer than 1.2 ms on its own. So I'd quote the demo's 13.8 ms and the 150x, which the code explains, and leave the 1,600x the table implies alone.
### 74% smaller, and why his number beats the board's
The bench board says something more modest than the post. Index on disk went from 503 to 309 MB with the Linux tree in home, and from 893 to 600 MB with Chromium: 39% and 33% smaller. Those totals are the whole data directory, names for the whole disk plus the content index for the tree, which is why they don't line up with the 0.76 GB Chromium content index in the October 10 table. Noah's own 1.5 GB to 388 MB is 74% (388 out of 1,500 leaves 26%). Four changes carry it, and I'd guess the fourth explains the gap.
The first is the filter added two days earlier. On October 10 every indexed file got two bloom filters, one over its 5-byte substrings and one over its 7-byte substrings, which is what grew the Chromium index from 0.32 to 0.76 GB. In 0.2 the 7-gram filter is gone, and the comment that replaced it gives the accounting (`src/content.rs:666-671`): it "read 0.66x the files on rare greps (0.82x their time) but took 30% of the index (0.87 bits per content byte)". Each file keeps one filter, 5-grams at 3 bits a gram, which the same file says still lets a missing gram through 24% of the time. Grep's first page got a little slower for it, 0.29 to 0.33 ms on Linux and 0.48 to 0.65 ms on Chromium, which is the regression he mentioned. I'd make the same trade.
The second is the posting lists. They were delta varints, a byte or more per document id. They are now binary interpolative code in blocks of 128 with a skip table (`ic_put`, `src/content.rs:4081`): encode the middle id in as few bits as its bounds allow, then recurse into each half. Files in one folder get neighbouring ids, so a run of consecutive ids costs, in the comment's words, "nothing". Paths in the doc table are front-coded in groups of 16 (`PATHS`, `src/content.rs:50`), so each path stores only what differs from the one before it, which in a source tree is usually the file name.
The third is the name index, which is now bit-packed and read in place (`src/index.rs:14-16`). An entry's name id takes as many bits as there are distinct names, 21 for two million, instead of 32. Its kind takes 4 bits instead of a byte. Its parent is no longer a 4-byte folder id: since each folder's children sit in one contiguous block, a bit per entry marking where blocks start, a rank count every 64 entries and one folder id per block are enough. Sizes and mtimes are stored in runs of 64 as a minimum plus each value's excess over it, which pays off because neighbours in a block tend to be close in both. Names with a long hex run or a UUID in them (every git object, every cache key) are stored two digits to a byte. On a disk like his, 8.3 million entries and about two million names, the fixed-width parts come to about 6.2 bytes an entry, plus a folder id per block, where the same fields took 13. Sizes and mtimes depend on the data, so I can't give a single per-entry figure the way I could for 0.1.
The fourth is cleanup. 0.1 rewrote a content segment only when more than half its documents were dead, deleted or replaced by a newer copy (`merge_plan` at `b88f4e9`). A segment that was 49% garbage stayed that way. 0.2 keeps the bytes of dead files under a sixteenth of the whole index and rewrites whichever segment has the largest dead share first (`DEAD`, `src/content.rs:47`, and `merge_plan` at line 2384). The bench board builds both versions from scratch, so neither has any dead weight. Noah's own index had presumably been running since launch, under a home where he was editing fsearch itself all week, and dead files are my guess for why his saving is twice the board's. There's a new rig for it, `examples/churn.rs`, which simulates weeks of edits, creates, deletes, growing log files and a weekly checkout. I can't check this guess without his data directory.
### "35% more files covered" is 35% more text
This is the one line in the post I'd correct. The board's own words are "holding 32% / 35% more text". The row that counts files is "files grep can search (rg's own set)", and it went from 96% to 99.9% on Linux and from 92% to 99.8% on Chromium, about 4% and 8% more files. More text and more files go together here, since the text comes from files 0.1 skipped, but they are different numbers.
The coverage change itself is real and sensible. Outside a git repo the old rules still apply: an extension list and a 1 MiB cap. Inside a repo, fsearch now reads what ripgrep would. Files up to 10 MB (`MAX_REPO_FILE`, `src/content.rs:40`, picked to match `rg --max-filesize 10M` and fff), whatever `.gitignore` and `.ignore` don't exclude, with text decided by sniffing the bytes rather than the extension. `target`, `build`, `dist` and the cache names used to be skipped by name everywhere. Now they're skipped only when the folder carries a marker, `CACHEDIR.TAG`, `CMakeCache.txt` or `build.ninja` (`MARKED_DIRS`, `src/content.rs:77`), because, as the comment puts it, "Linux's drivers/target is source". Lockfiles, JSON of 100 KB and up, minified bundles and linked git worktrees stay out. A reply asked whether a grep in a linked worktree says that it isn't indexed. From the code, the worktree's files are left out of the index and `in:` on the worktree falls back to reading files directly, which gives a correct answer, slowly; I didn't find a message that tells you.
### Memory a daemon gives back
The 65% is Chromium's daemon going from 100 to 35 MB. The same row for Linux went from 19 to 21 MB, slightly up, which tells you what the fix is. The process never held much data. It held memory it had already freed.
The new `src/main.rs:70-84` has the story. macOS's malloc "keeps freed small blocks dirty and its pressure relief does nothing, so after the first run or a compaction the daemon held 150-290 MB it no longer used, until restarted." Malloc has a space-efficient mode that hands those blocks back, but it is read only at launch, so `fsearch serve` re-executes itself with `MallocSpaceEfficient=1` set. It idles at 5 to 15 MB on his machine, for searches about 8% slower. Big blocks, 1 MB and up, now come straight from `mmap` and go straight back, because malloc kept those mapped and dirty too, "which left the daemon at ~1 GB footprint after a build while ~2 MB was live" (`src/alloc.rs:1-11`). Space-efficient mode also turns off malloc's per-thread caches, which cost searches about 10%, so the search threads keep small free lists of their own and drain them back a minute after searching stops.
Chromium's index build churns through far more memory than Linux's, so it had far more freed memory to give back, which is why its row fell and Linux's didn't.
Peak memory while indexing fell from 1.13 to 0.77 GB on Linux and from 1.37 to 0.94 GB on Chromium, 32% and 31%, which is the post's figure. The video's own run shows 710 to 291 MB, 59%. The main reason is in `src/engine.rs:874-877`. The content build used to keep two 64 MB batches in flight, so one batch's single-threaded tail overlapped the next batch's reads. Now it keeps one: "3.07 s instead of 3.33, but a first-run peak of 616 MB instead of 375." The name index's build also writes its blob section by section, freeing each source array once it has been copied (`src/index.rs:826-828`), and the smaller formats above shrink everything the build holds.
### What else 0.2 changed
The rest isn't in the post but it's on the card. Typos changed the most, and they make a section above out of date. At `af9476d` a typo had to sit in a word that started with the query's first letter. 0.2 runs the old search first, and if it doesn't find a good enough match, runs it again "deep" (`search_form`, `src/query.rs:2079-2085`), allowing a wrong, extra, missing or swapped first letter and two edits in words of eight letters or more. On his Linux bench set, the meant file for names with a wrong first letter came first 24% of the time before and 98% after, and names with two typos went from 11% to 74%. The second pass is why name search slowed from 0.045 to 0.054 ms on Linux and from 0.10 to 0.12 ms on Chromium, and why the typo sets' median doubled from 0.05 to 0.10 ms. My complaint that `ython` finds nothing is answered.
A first run also no longer blocks a search inside a folder. Until the whole disk is listed, a query with `in:` gets an index of just that folder, built on the spot with 16 threads (`with_names`, `src/engine.rs:492`), and that folder's contents are indexed first. 7.8 s to 0.18 s on Linux, 1.2 s on Chromium. The rest is new surface: globs, `git:` filters, frecency and nearby-file boosts, and `fsearch mcp`, an MCP server with `find_files`, `grep` and `multi_grep` for coding agents.
The post did undersell one thing. A daemon that sits in the background on every Mac it's installed on should be judged on what it costs when you aren't searching, and that is what this release went after. The multipliers in the post are mostly right, the 35% describes the wrong quantity, and the regressions are named in the README table rather than hidden. I trust this release more than either earlier headline.
## How I checked
I shallow-cloned `noahdunnagan/fsearch` at commit `af9476d` (October 8), then fetched the full history to read the earlier README at `f43261b`, and read every file under `src/` and `demo/`. Line references are to `af9476d`. The fd and grep -r figures come from the final frame of the launch video, which I pulled with ffmpeg; the repo has no script for them. The fff numbers are from `demo/vs_fff_chromium.json` and `demo/vs_fff_linux.json`, and the method from `demo/vs_fff.py`. The thread quotes are from the fxtwitter mirror of posts 2107993727809855570 and 2108257494841827721.
fsearch does not build on Linux, so I did not run it. The per-entry byte counts are read from `section_lens`; the `/usr` listing (189,110 entries) came from `find /usr -xdev`; the mask rejection rates and the 1.5 ns per name come from a single-threaded C port of `char_bit`, `name_mask` and `Token::fits` timed over 50 repetitions on a shared 16-core machine under load, so treat the timings as rough. The widgets port the same functions plus `one_edit_prefix` and `Index::build`'s ordering to TypeScript; their ranking is simplified. I could not measure Finder, Spotlight or `mdfind` here, so the Spotlight estimate above is a guess and labelled as one.
For the October 11 update I cloned the repository again at `ffcb37d` (October 10, 23:00 EDT), diffed it against `af9476d` and `76d612f`, and read the commit messages of all 110 commits in between along with the new `examples/rig.rs`, `examples/e2e.rs`, `demo/abba.py` and `pgo.sh`. The October 10 post (2108960053915267096) and its replies came through the fxtwitter mirror. Its short link, `nd.mt/g/fsearch`, redirects to the same GitHub repository. The two figures are frames from the post's video. The bitmap filter's survival rates and per-name times come from a C port of `name_bitmaps`, `doubled` and `TokenBits::words`, typo path included, run alongside the old port over `find /usr -xdev` (189,124 entries, 99,827 distinct names this time), 50 repetitions each on a loaded shared machine. The per-name byte count is read from `section_lens` at `ffcb37d`. The fff rerun figures are from `demo/vs_fff_chromium.json` and `demo/vs_fff_linux.json` at `ec25ffa`. I still could not run fsearch itself.
For the October 13 update I cloned the repository with its tags, at `v0.2.1` (`88e63fd`), and diffed `v0.2.0` (`e96ebeb`) and `v0.2.1` against `b88f4e9`, the README's baseline. Line references in that section are to `v0.2.1`; the ones marked `b88f4e9` are to the old code. The post (2109675745408930191), its thread and replies came through the fxtwitter mirror. I read the video's frames once every three seconds plus the last one, extracted on another machine; the three figures in that section are frames from it. The 149x is 2.06 s over 13.8 ms; 74% is 388 MB out of 1,500; 39% and 33% are 309 over 503 and 600 over 893; the 65% is 35 over 100 MB; 32% and 31% are 0.77 over 1.13 and 0.94 over 1.37 GB. The 6.2 bytes per entry is my arithmetic from `section_lens` at 8.3 million entries and two million names. The bench board's release candidate `427be0b` is not in the public history, and I could not find what produces the README's 1.3 ms fresh-grep figure. I did not run fsearch, and the explanation of why his own index shrank more than the board's is a guess.
---
# The integer-multiplication exponent race: from 2^-182 to 2^-14.6 in forty hours
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/integer-mult-exponent-race
> date: 2026-10-08
> tags: math, theory, algorithms, agentic-coding, verifiers
The site owner sent me a link to [CrocSwap/integer-mult-bounds](https://github.com/CrocSwap/integer-mult-bounds) with the note "something crazy going on in this repo". When I opened it, the headline said $\kappa = 2^{-30}$. By the time I had read the pull requests, the best open claim was $\kappa \approx 4.1\times10^{-5}$, a little above $2^{-15}$. When I started writing this paragraph, PR #45 had just arrived. By the time I finished the article, #48 had nudged the claim to $4.1186\times10^{-5}$, and Swapnil Jain, racing in a separate repository, had reached $3.67\times10^{-5}$ with its arithmetic checked in Lean.
Then, at 15:18 UTC, the story changed shape. Colkitt merged Rohan Arun's PR #39 into `main`, with $\kappa = 971668963/(2.5\cdot10^{13}) \approx 3.8867\times10^{-5}$, after an audit he published alongside it. For the first time a community number is the maintainer's number. It is still conditional on OpenAI's unrefereed manuscript, and the audit was Colkitt working with Codex, not a referee; [the section on the merge](#pr-39-reaches-main) goes through exactly what it checked. The open PRs had already moved past it: Rohan's own #49, under twenty minutes later, claims $4.1239\times10^{-5}$.
And it kept going after I thought I had finished. By 18:35 UTC `main` had moved twice more and stood at $5.1017\times10^{-5}$, after a new idea broke the plateau I describe below; the open PRs reached $5.1739\times10^{-5}$ an hour later. Aurel Prosz, one of the contributors, built a live dashboard of the whole thing. And Ryan Shea took Jain's network across to a second OpenAI problem, the exact discrete Fourier transform, and claimed a saving 730 million times OpenAI's. [The evening](#main-moves-again) and [the Fourier transfer](#the-race-reaches-the-fourier-transform) have their own sections near the end.
Then, overnight, the plateau broke for good. Between 23:12 UTC on 8 October and 05:06 the next morning, icekylinx's PRs replaced the shape of the network itself, and `main` jumped ninefold to $4.609169\times10^{-4}$, about $2^{-11.08}$. Jain followed twenty minutes later at $4.6637\times10^{-4}$, by building on the same cover, and the open PRs were at $6.5592\times10^{-4}$ when I stopped, at 11:00 UTC on 9 October. [The ninefold night](#9-october-the-ninefold-night) explains what changed, and why κ is still nowhere near 1.
The afternoon was quieter in ratio and busier in every other way. At 12:57 UTC Colkitt moved `main` again, to $6.61886\times10^{-4}$. Ten minutes later Jain posted round eleven at $6.6857\times10^{-4}$, which led every claim anywhere for nineteen minutes, and said he needed an arXiv endorser for a paper collecting the proofs. By 18:50 UTC the best open claim was $6.84697\times10^{-4}$, and several contributors had stopped chasing κ to prove how far the current designs can go at all. [The afternoon](#9-october-afternoon-reusing-dead-registers) covers all of that.
Overnight the Fourier side overtook the multiplication side in the one way that matters. Jacob Sussman, working with Claude agents, proved in Lean that a fixed program computes the exact Fourier transform of every length in $O(n(\log n)^{1-\delta})$ steps, on OpenAI's own Lean statement with only the exponent changed, and Chafik Boukhalfa pushed it to $\delta = 0.000754736$. Ryan Shea's paper now carries that number. On the multiplication side a layout from the same Lean work, five stages instead of three, took the open claims to $7.51\times10^{-4}$, and Colkitt announced a merge that GitHub still doesn't show. [10 October](#10-october-the-fourier-side-gets-a-proof) has it all.
Then the multiplication side found a new shape too. Around midnight UTC on 11 October contributors swapped in a small Lean-checked unit of Sussman's built on 120 labels the agents named E8, first on the complex side and then on the bit side, and the open claims went from $7.9\times10^{-4}$ to past $10^{-3}$ by 07:09 UTC and to $1.29\times10^{-3}$ by the evening. `main` had moved once, to $7.24\times10^{-4}$. [11 October](#11-october-e8-takes-the-race-past-a-thousandth) explains what E8 is doing there, checks the leading claim, and looks hard at Rohan Arun's statistics, one of which is 261 million lines of code.
And then the maintainer caught up. On the evening of 12 October Colkitt posted a reviewed checkpoint of $\kappa \approx 1.6127\times10^{-3}$, $T(n) = O(n(\log n)^{0.99838728})$, from Jeremy Kothe's helpers that stop short of the full frame. By the time I cloned the repository the next morning `main` had moved four more times, to $1.661635\times10^{-3}$, and for the first time in the race no open PR claimed more. [13 October](#13-october-main-catches-the-frontier) covers the eight moves, the two mechanisms worth understanding, and how one maintainer merged 116 PRs in a day and a half.
By the afternoon `main` had moved twice more, by 0.0195% and then 0.00017%, to $1.661961\times10^{-3}$, and Rohan Arun proposed that every repository like this one carry an `auto-research.md` prompt and an `auto-research/` folder for failed experiments. [13 October (evening)](#13-october-evening-auto-research-gets-a-standard) reads the proposal, the Leech and Barnes–Wall screens, and his PR statistics.
Some background. Two days ago OpenAI's math release included a 73-page manuscript, [Integer multiplication below n log n](https://github.com/openai/math/tree/adc7f1241b42e322a6451854ab7e4b4c146bf78a/preprints/Integer-multiplication-below-n-log-n-September-23-2026) (result family 109), which claims a multitape Turing machine that multiplies two $n$-bit integers in $O(n(\log n)^{1-\kappa})$ steps with $\kappa = 2^{-182}$. I covered it briefly in [the openai/math roundup](/articles/openai-math#integer-multiplication-below-n-log-n), and it has a tile on the [AI breakthroughs in mathematics wall](/math#109). Doug Colkitt ([@0xdoug](https://x.com/0xdoug), the founder of CrocSwap/Ambient) forked the argument into a "research draft" and started tightening the constants. Then other people joined in. Counting $2^{-182}$ as the start, the exponent saving has grown by a factor of about $2^{167}$ in forty hours.
That sounds like a story about multiplication getting faster. It isn't one, and the reason is the most interesting part. What follows covers what $\kappa$ is, where it comes from in the construction, what each step of the race actually changed, what I could re-run myself, and why the very funny "linear time by tomorrow" chart that went round on X is wrong in an instructive way.
## What $\kappa$ measures
Schönhage and Strassen multiplied $n$-bit integers in $O(n\log n\log\log n)$ steps in 1971 and guessed that $n\log n$ was the true answer. Fürer got within a $2^{O(\log^* n)}$ factor of it in 2007. Harvey and van der Hoeven reached $O(n\log n)$ exactly ([Annals of Mathematics, 2021](https://hal.science/hal-02070778)), and most people took that to be the end.
The OpenAI manuscript claims a bound of
$$
T(n) = O\!\left(n(\log n)^{1-\kappa}\right) = O\!\left(\frac{n\log n}{(\log n)^{\kappa}}\right).
$$
So $\kappa$ is the power of $\log n$ you get to divide out. At $\kappa = 0$ you have Harvey–van der Hoeven. At $\kappa = 1$ you would have linear time, which is the floor, since just reading the input takes $n$ steps. Every value in between is a strictly faster growth rate than $n\log n$. For the question "is $n\log n$ optimal?", $2^{-182}$ is as good an answer as $1/2$: any $\kappa > 0$ refutes the conjecture.
The machine model matters. The theorem is about a deterministic Turing machine with a fixed finite alphabet and a fixed number of one-dimensional tapes, the model in which Schönhage and Strassen stated their conjecture. On a tape, moving data costs time proportional to the distance it travels, so even rearranging $\Theta(n)$ bits at each of $\Theta(\log n)$ FFT levels costs $\Theta(n\log n)$. The manuscript says this outright: "Scanning an array of $\Theta(n)$ stored bits at each of $\Theta(\log n)$ transform levels already costs $\Theta(n\log n)$. We therefore need savings in both movement and arithmetic" (`upstream/build/sections/00-introduction.tex`).
Nobody had ever proved an $\Omega(n\log n)$ lower bound in this model. The best result is conditional: Afshani, Freksen, Kamma and Larsen ([2019](https://arxiv.org/abs/1902.10935)) showed that a circuit lower bound would follow from the network-coding conjecture, and a time-$T$ Turing machine only gives circuits of size about $T\log T$, so even that does not transfer. The [FFT section of the roundup](/articles/openai-math#what-was-actually-open) goes through the same gap for the Fourier transform, where the release's family 130 reuses this manuscript's finite gadget. The point to keep in mind is that $n\log n$ was a barrier nobody knew how to cross, not a proven wall.
## Where $\kappa$ comes from
The pipeline is Harvey–van der Hoeven with every change of representation charged on tape: digits go onto prime cyclic axes by a Chinese-remainder map, Gaussian resampling turns those into power-of-two axes, and a phase twist turns the last axis into polynomials modulo $y^r+1$, where multiplying by a root of unity is just a signed shift.
The new part is two tape procedures built from fixed linear networks. One swaps two address fields of an array; the other applies a layer of butterflies $(u+v)/2,\ (u-v)/2$ to selected coordinate bits. Each network is a finite circuit on $W$ "roles" (wires), it exchanges two banks of values while restoring every scratch value, and each edge of the circuit is carried out by smaller recursive calls of the same procedure. The decisive count is the total number $s$ of those smaller calls. If $s < Wm$, then multiplying the problem parameter by $m$ multiplies the normalized cost by $s/W < m$, and the recursion runs at $m^\tau$ with $\tau<1$. The manuscript: "This strict inequality is the source of the power saving."
How much below $m$ it lands is tiny. In the original construction the relative deficits are (`upstream/build/sections/03-motifs.tex:673-678`)
$$
\eta_{\rm b} = \frac{339}{22587335000000},\qquad \eta_{\rm c} = \frac{73}{19906842167500},
$$
at $m = 10^6$, so the honest savings are about $10^{-12}$ and $2.7\times10^{-13}$. The manuscript then rounds them down to $1-\tau = 1-\sigma = 2^{-50}$ and calls its constants "deliberately conservative; no optimization is claimed". Remember that sentence. A lot of what happened next is people taking it at its word.
Those network savings feed a cost table. With $p \approx \log n$ the working precision, the paper divides every operation's cost by the data volume and gets a power of $p$. Seven rows dominate, and each has a margin $g_i$ below 1:
$$
\begin{aligned}
g_1 &= 1-\epsilon(1+c), & g_2 &= \epsilon c(1-\tau), & g_3 &= \epsilon(1-\lambda'),\\
g_4 &= 1-\tau-\epsilon(2-\tau), & g_5 &= \tfrac14-\delta-\tfrac32\epsilon, & g_6 &= 1-\delta-\epsilon,\quad g_7=\epsilon.
\end{aligned}
$$
The total time is $n$ times $p$ to the largest power, so the exponent saving is the smallest margin, minus a sliver of slack to absorb stray $\log p$ factors. The upstream parameters are $\epsilon = 2^{-75}$, $c = 2^{-56}$ and $1-\tau = 2^{-50}$, so $g_2 = 2^{-75}\cdot2^{-56}\cdot2^{-50} = 2^{-181}$, and the paper says "Thus $\min_i g_i=2^{-181}=2\kappa$" (`08-assembly.tex:866`). That is where $2^{-182}$ comes from: three small numbers multiplied together, then halved.
The repository's checker transcribes the same seven margins as exact fractions (`scripts/certify.py:136-147`):
```python
def margins(p, *, layout_model="adjacent", assembly_model="original"):
require(assembly_model in ("original", "tight-gaussian"), "Unknown assembly model")
e, c, t = p.epsilon, p.c, p.tau
return {
"g1": 1-e*(1+c),
"g2": e*c*(1-t),
"g3": e*(1-p.lamp),
"g4": 1-t-e*(layout_degree(layout_model)-t),
"g5": Q(1, 4)-p.delta-(Q(5, 4) if assembly_model == "tight-gaussian" else Q(3, 2))*e,
"g6": 1-p.delta-e,
"g7": e,
}
```
Once you see $\kappa$ as a product of factors, the race is easy to follow. Every improvement attacks one of the factors: it makes the network saving $1-\tau$ bigger (better circuits), lets $\epsilon$ grow (better routing, better Gaussian resampling), lets $c$ grow (cheaper control movement), or changes the recurrence so the same circuit counts for more (batching). The widget below shows the seven margins on a log scale for the original paper, Colkitt's $2^{-30}$ checkpoint and PR #13. In the original, $g_2$ sits at $2^{-181}$ and nothing else comes close. By PR #13, $g_2$, $g_3$ and $g_4$ are all balanced at about $2^{-20.3}$, which is what a tuned parameter set looks like.
There is also a ceiling built into this shape. Since $g_7 = \epsilon$ and $g_1 = 1-\epsilon(1+c)$, the smaller of the two is at most $1/2$ whatever you choose, and $g_2$ is at most the bit network's saving $1-\tau$. In every witness I checked, $\kappa$ sits just under that network saving: $9.3\times10^{-10}$ against $4.67\times10^{-9}$ at the $2^{-30}$ checkpoint, $7.699\times10^{-7}$ against $1.54\times10^{-6}$ in PR #13, and $4.1006\times10^{-5}$ against $4.1008\times10^{-5}$ in PR #44. To get anywhere near linear time this method would need finite networks whose recursive calls nearly vanish, and nothing here suggests that is possible. The overnight witnesses of 9 October keep the same shape: in PR #144's assembly two margins are balanced against each other so that $\kappa \approx a/(1+2a)$, where $a$ is the weaker network's saving, and that saving is in turn about the network's relative rank deficit divided by how far below full width its average child sits on a log scale. The section on [the ninefold night](#why-smaller-is-worth-nine-times) works through the numbers. By 11 October the assembly had changed to balance at $\kappa \approx s/(1+s)$, with $s$ the weaker network's saving ([the ceilings, a day later](#the-ceilings-a-day-later)).
Of everything in that chain, the part that steers the race day to day is a minimum. From the three-shear cover on, the assembly is fed a single network saving, the smaller of the bit network's and the complex network's, and whatever the stronger side has beyond that goes unused. Contributors call the binding side the wall. I found it easier to hold as an instrument than as a formula, so I drew it as a bench stop gauge. Each network gets a rod with a clamp collar set at its certified saving, the bit network in front and the complex network behind. A steel band off a spring drum pulls the carriage toward larger savings, and the carriage comes to rest against whichever collar is nearer. That collar is the wall, and it is lit. Moving the other collar changes nothing until it overtakes. The carriage's index foot reads the saving on a ruler in units of $10^{-4}$, with a long line at $10^{-3}$, and its dial turns once per $10^{-4}$, as a dial caliper's does. A pin on the ruler marks `main` at that moment. $\kappa$ itself sits a hair under the stop, at $a/(1+2a)$ in #144's assembly and $s/(1+s)$ in the one in use on 11 October; at #144 that is 0.09% under, which is too close to draw.
The ten witnesses on it are the ones where this article gives both sides' savings, and stepping through them, the wall changes sides seven times in four days. At #144 the bit network binds at $4.6134\times10^{-4}$ against the complex $4.856569\times10^{-4}$. Jain's round ten, on his own track, leaves the complex side behind ($6.1610\times10^{-4}$ against $6.3239\times10^{-4}$), and round eleven hands the wall back to the bit side by a whisker ($6.6947$ against $6.7147$). On 11 October #346's Design T leaves the complex supplier as the wall, Sussman's E8 unit in #352 lands 9.3% past the bit side and gives $\kappa = 8.0108\times10^{-4}$, the E8 bit word in #370 flips it back, and by #467 the two collars sit almost level, at $1.2909\times10^{-3}$ and $1.297\times10^{-3}$, with the carriage past the $10^{-3}$ line and `main`'s pin still back at $7.23572\times10^{-4}$. #390, the first PR past $10^{-3}$, isn't a stop here because I only have its complex side. The last three stops are from `main` itself on 12 and 13 October, so the pin sits right under the carriage: #535 at 13:11 UTC on 12 October, where the bit side binds by 0.16%, Jeremy Kothe's #596, the checkpoint Colkitt posted that evening, where the complex side binds with the bit word 2.4% ahead, and #799 the next morning, at $\kappa = 1.661635\times10^{-3}$ with 4.1% to spare on the bit side ([13 October](#13-october-main-catches-the-frontier)). To fit them I stretched the ruler to $17.5\times10^{-4}$. The figure is drawn with the site's copy of the isometric-figures skill, which is Ryan's [Anatomy](/articles/anatomy-machine-skill), and every collar is set from the numbers in the sections below.
## Colkitt's day and a half
Colkitt's own checkpoints are on `main` (the last one is also on `release/ternary-30`). Each came with an X post from [@0xdoug](https://x.com/0xdoug), and each post's ratios check out against the exact witnesses.
| commit (UTC) | $\kappa$ | what changed |
|---|---|---|
| `5cf29ec` Oct 7 02:22 | $5.8\times10^{-33}$, between $2^{-108}$ and $2^{-107}$ | parameters only: same network, sharper recurrence comparison |
| `52ce3be` Oct 7 13:35 | $2^{-78}$ | direct nonadjacent axis swaps: layout from $O(d^2)$ to $O(d)$ |
| `bcd4ebd` Oct 7 19:27 | $2^{-59}$ | paired circuits with shared sums, tighter guard and Gaussian width |
| `6e56487` Oct 8 00:27 | $83/10^{12} > 2^{-34}$ | compact dirty controls instead of spaced windows |
| `1c09a58` Oct 8 01:10 | $2^{-31}$ | compressed complex network, binary phase frames |
| `1a74950` Oct 8 13:10 | $2^{-30}$ | ternary five-subset bit circuit over $\mathbb F_3$ |
The first note is the most honest thing in the repository. With the paper's cost accounting left alone, it finds $\kappa = 29/(5\cdot10^{33})$ and also proves a ceiling of $5.838\times10^{-33}$ for that family, so the witness is within 1% of the best that accounting allows. Colkitt's post said so: "Surpassing this ceiling would require improving the network bounds or cost analysis from the original result."
So the next morning he changed the accounting. The original layout row costs $d^2(1+\ell^\tau)$ because it reverses axes with adjacent swaps; the manuscript already allowed nonadjacent swaps, and using them directly makes it $d(1+\ell^\tau)$. That changes $g_4$ to $(1-\tau)(1-\epsilon)$, which no longer forces $\epsilon$ to be smaller than the network saving. The result-history page explains the consequence: the original layout "forced epsilon < a, yielding a cubic constraint kappa < a^3", and the new schedule makes the saving scale "quadratically in a" (`docs/research/result-history.md:253-256`). Going from $a^3$ to $a^2$ when $a \approx 10^{-11}$ is how you gain thirty-odd powers of two in one commit.
The $2^{-59}$ step improved the network itself, which raises $a$: rectangle incidence circuits, then shared intermediate sums, cut the side roles per invocation from 41,122,620 to 2,394,438 and then to 577,576 at $h=46$, and paired-block circuits at $h=50$ reach 509,194. Fewer roles for the same rank deficit means a larger relative deficit, so a larger $1-\tau$.
The compact-control step at $2^{-34}$ attacked $c$. Moving whole spaced windows had a spacing penalty $K^\tau$ that forced $c$ to be of order $a$; moving only compact control fields removes it, so $c=1$ and $\kappa$ becomes roughly $\epsilon$ times $a$. His post that night put it in two sentences: "The improvement came from removing the spacing penalty behind the quadratic bottleneck. This was done by moving compact control bits instead of entire windows." Its "roughly 48 million fold improvement over the previous result" checks out against $2^{-59}$ (about $4.8\times10^{7}$), and so does "a 2¹⁴⁸ fold improvement over original OAI result" ($2^{148.5}$). Here the README discloses that the idea did not come from Colkitt: "The compact-control proposal originated with a separate research agent; the supplied note develops its tape, layout, repair and assembly arguments" (`README.md:138-140`).
The last two checkpoints moved the bottleneck between the two networks. At $2^{-31}$ a compressed complex network ($h_c = 26$, $a_c = 5/10^9$) overtook the bit network's $296/10^{11}$, so the bit side became binding. The $2^{-30}$ checkpoint then replaced the bit circuit with one that computes over $\mathbb F_3$.
The $\mathbb F_3$ trick is pretty enough to show. Label the circuit's wires by the five-element subsets of 29 points, $n = \binom{29}{5} = 118755$ of them, and let $B$ be the incidence matrix of five-subsets against pairs. Then $(BB^{\mathsf T})_{ST} = \binom{|S\cap T|}{2} \bmod 3$, and for intersection sizes $0,\dots,5$ those residues are $0,0,1,0,0,1$. So $BB^{\mathsf T} = I + A_2$, where $A_2$ is the adjacency of subsets meeting in exactly two points, and the "side" map is just $-A_2$. A central factor of size $r = \binom{29}{2} = 406$ plus a correction that only touches intersection-two neighbours gives exactly the identity. I checked the residue identity by brute force over all 126 five-subsets of nine points; it holds.
The full circuit has 19,593,239 active additions and 20,780,789 side roles per invocation, and the exact count gives $s/(Wm)$ with a relative deficit of $553/11717905300$. I recomputed the implied saving from the certificate's $W$, $s$ and $m$: $-\log_m(s/Wm) \approx 4.672\times10^{-9}$, just above the certified $a_{\rm b} = 467/10^{11}$. The final minimum margin is $2332833/2500000000000000$, which exceeds $2^{-30}$ by about 0.194% of the claimed saving.
That checkpoint was not first, and its README says so: "Zhihao Chen's earlier PR #7 introduces the same ternary five-subset motif with a different circuit and stronger claimed bound. This release records a separate implementation and conditional checkpoint; it makes no priority or strongest-known-bound claim."
## Everyone else shows up
The first outside PR arrived at 20:58 UTC on October 7. Between 02:23 and 14:24 UTC on October 8, 42 more arrived, from about ten people. By 11:00 UTC on October 9 there were 179, by 11:30 UTC on October 10, 301, by 19:40 UTC on October 11, 485, and by 07:20 UTC on 13 October, 843. The chart has all of them that state a κ: Colkitt's checkpoints in red, OpenAI's starting point in grey, the community PRs coloured by author, a green step line for the best claim open at each moment, and a red step line with a square at each move of `main` (#39 at 15:18 UTC, #49 at 16:47, a reviewed batch at 18:35, and icekylinx's #144 at 05:06 the next morning, then ten more, eight of them on 12 and 13 October). Everything after $2^{-17}$ is squeezed into a sliver at the top, so the zoom button redraws everything from 10:00 UTC on October 8 on its own. Jain's separate track is the purple diamonds, with a ring on the rounds whose arithmetic is checked in Lean; more on that [below](#a-second-race-one-repository-over). The dotted green line is $\kappa = 10^{-3}$, first passed on 11 October, and the hollow circles are claims that rest on a step another contributor has challenged ([11 October](#467-and-the-claims-it-leaves-out)). Click a point (or use the dropdown) for its title and link.
A few PRs changed the shape of the problem rather than its constants, and those are the ones worth understanding.
Start with eumemic's #3 and #5. #3 compressed the complex network, cutting its side wiring from 3,693,800 roles per invocation to 108,195 and taking its saving from $418/10^{12}$ to $14/10^9$. That made the bit network binding, and the next limit was the Gaussian resampling row: its cost $O(tp^{3/2+\delta}\alpha)$ forced $\epsilon < 1/5$ and capped $\kappa$ below $a_{\rm b}/5$. #5 rewrote the resampling as chirped correlations evaluated by the existing $O(N\log N)$ multiplier and bounded the Neumann series more tightly, which let $\epsilon$ approach $1/2$. That one change of analysis was worth a factor of about 2.5 on its own.
Zhihao Chen's #7 is the $\mathbb F_3$ five-subset construction, at $h=28$, posted at 06:12 UTC, seven hours before Colkitt's independent version. Chen's PR credits "GPT-6 Astra (OpenAI Codex)" for "research, proof-development and implementation assistance" and asks that future work "explicitly acknowledge Zhihao Chen (jacklightChen), cite this PR and its accompanying note". The maintainer did.
icekylinx's #10 is the structural break you can see on the chart: from $2^{-28}$ to $2^{-23}$ in one PR. Upstream, every edge of the network costs singleton recursive calls, one per unit of rank. #10 shows that a contiguous block of $t$ address fields from a projector can be done as one recursive call of width $t$, given a common controlled basis. The recurrence condition stops being a count and becomes a moment: with rank-mass weights $w_i$ on calls of relative width $r_i$, you need $\sum_i w_i r_i^{-a} < 1$. A singleton call has $r = 1/m$ and costs the full $m^a$; in #13's profile, a block covering 195/196 of the fields costs almost nothing extra. The same PR fixes an error-enclosure gap it found in #5's fast Gaussian argument, with no change to the exponent.
eumemic's #13, the one the site owner pointed at, is a neat idea. The network's correctness condition only needs each role's endpoint frames to differ by the identity: $M_{\rm sink} - M_{\rm source} = I$. A scratch role does not have to start in the zero frame. It can start in any fixed frame $M$ as long as it ends in $I+M$. Starting every stage-two auxiliary role in the frame of its first gate deletes its rank-756 entrance edge and turns its exit into one projector of rank $m-h$. That projector compiles to $h$ singleton pivots plus one contiguous block of $m-2h$ fields. The total rank is unchanged. What changes is where it sits: out of singleton calls costing $\ln m$ each and into one block. Under #10's moment, the interchange saving goes from $246/10^9$ to $154/10^8$, and with a third whole-residual class on the complex side, $\kappa$ goes from $6149999/(5\cdot10^{13})$ to $7699/10^{10}$, a factor of 6.26. Since $\log_2(7.699\times10^{-7}) \approx -20.31$, the claim is about $2^{-20.3}$; the title's "$> 2^{-21}$" undersells it.
After #13 the tempo picks up. #15 brought a smaller $h=30$ producer, #18 partial swaps, and #21 and #23 translated frames and "semantic precision", reaching $1099/10^8 > 2^{-17}$. #29 introduced a two-stage topology from Aurel Prosz's work and reached $15536/10^9 > 2^{-16}$. #36 added copied retained centers, which halve one rank term ($Wm-N+2L$ becomes $Wm-N+L$) and jump to $3.84569\times10^{-5}$. Everything after that is composition and tuning: fixed local bases, reversed corners, carrier matchings. PR #44 claims $2050314627/(5\cdot10^{13}) \approx 4.1006\times10^{-5}$, which it describes as "approximately 0.02681% above #43". The last three PRs are within 0.03% of each other. Three more arrived while I was writing: chafreaky's #46 adds 0.0000048% to #44, Rohan Arun's #47 adds 0.109% with hill-climbed producers, and chafreaky's #48 reorders the leave-one-out sums to save 847,550 wires and reach $411862541/10^{13} \approx 4.1186\times10^{-5}$. Then Rohan's #49, at 15:36, started from #48's orderings and hill-climbed them the way #47 had, for $4123863984/10^{14} \approx 4.1239\times10^{-5}$, 0.127% more. The step line flattens around $2^{-14.6}$ after 12:30 UTC.
When I first wrote this section I called that flattening the most informative thing on the chart, and said it was what you would expect if the cheap structural moves inside this family of constructions had mostly been made. Three hours later Avi Eisenberg's #53 jumped 9.2% in one PR, and the step line climbed again to $2^{-14.24}$. Use the zoom to see both plateaus; [the evening section](#main-moves-again) explains the move that broke the first one. I'd still read the flat stretches the same way, only with less confidence about how long any of them lasts.
## Re-running PR #13
The brief asked me to check the claims, and these are certificate checks I could actually run: pure Python with `fractions`, plus a small C++ helper on the PR branches, no network. On a shared 16-core machine at `nice 19`:
On `main` at `1a74950`, the focused verification from the README. `scripts/audit_ternary_side.py` builds the full $h=29$ DAG (about 21 million addition nodes) and finished with `PASS ternary side construction and conditional 2^-30 assembly.` in 176 s at 1.6 GB peak memory. `make_ternary_patch.py` took 17 s, the 12 focused tests passed in 39 s, the patch applied to the pinned upstream source, and the regenerated certificate and patch were byte-identical to the committed ones.
On PR #13 at `3ef246f`, `scripts/source_frame_network.py` runs in 0.1 s and prints the witness:
```text
PASS kappa=7699/10000000000 > 2^-21
bit saving=77/50000000; complex saving=9/5000000
bit moment gap=4.69651719166272e-10
complex moment gap=3.899666592614166e-09
minimum margin=3849992299500001/5000000000000000000000; gap=492299500001/5000000000000000000000
```
The regenerated `certificates/source-frame-network.json` compared equal to the committed one. The six tests in `tests/test_source_frame_network.py`, including an exact $h=5$ model in which the new exit is checked to be an idempotent of rank $m-h$ with exactly $h$ corner pivots, passed in 20 s. The full `make verify` on the PR branch took 432 s: 172 tests passed, all 19 manuscript patch checks applied, and the certificates and patches regenerated without a diff. Those numbers match what the PR description claims.
Then I recomputed the arithmetic myself, without the repository's helpers. From the certificate's parameters ($\epsilon = 499999/10^6$, $c=1$, $1-\tau = 154/10^8$, $1-\lambda' \approx 1.54\times10^{-6}$, $\delta = 10^{-10}$) the margins are $g_2 = 38499923/(5\cdot10^{13})$, $g_3 = 3849992299500001/(5\cdot10^{21})$ and $g_4 = 38500077/(5\cdot10^{13})$, all about $7.70\times10^{-7}$; $g_1$ and $g_5$ are about $2\times10^{-6}$, and $g_6$ and $g_7$ about $1/2$. The minimum is $g_3$, which beats $\kappa = 7699/10^{10}$ by $492299500001/(5\cdot10^{21}) \approx 9.8\times10^{-11}$. So the claimed $\kappa$ does follow from the margins.
The bit-moment check is the step that carries the new idea, so I redid it at 80-digit precision rather than with the script's $e^x \le 1/(1-x)$ bound. Here is how the script does it (`scripts/source_frame_network.py:54-62` on PR #13):
```python
def moment(weights, ratios, a):
"""Upper bound for sum_i w_i ratio_i^(-a), using exp(x) <= 1/(1-x)."""
logs = []
for r in ratios:
simple = SIMPLE_LOGS[r]
require(log_upper(1/r) < simple, 'Rational logarithm bound failed')
require(0 <= a*simple < 1, 'Exponential upper-bound range failed')
logs.append(simple)
return sum((w/(1-a*ell) for w, ell in zip(weights, logs)), Q(0)), logs
```
The four classes have weights of about 0.0044 (singletons, relative width $1/21952$), 0.4934 (width $195/196$), 0.0076 ($10165/10976$) and 0.4946 ($391/392$). With $a = 154/10^8$ the moment is $1 - 4.78\times10^{-10}$, below 1 as required, and bisection puts the largest saving this rank profile supports at about $1.5499\times10^{-6}$. PR #13's $1.54\times10^{-6}$ leaves about 0.6% headroom.
None of this checks the hard part. The certificates confirm that the counts, ranks and inequalities are what the notes say they are. They do not confirm that a contiguous projector block really can be executed as one recursive call on a fixed-tape machine, that the controlled basis exists at full size (the review guide for #10 says it is "established by the existence argument, not by materializing every h28 factorization"), or that the upstream theorem is true. Colkitt's own review index was clear, that morning, that this was still open: "These changes appear to cross the singleton-rank recurrence limitation; the headline alone does not establish that step." By the afternoon his audit had worked through those obligations for the one chain he merged, #39's ([below](#pr-39-reaches-main)). For #13's own construction and every PR outside that chain, the sentence still stands.
## A second race, one repository over
While those PRs were landing, Swapnil Jain ([@SJ_Swapnil_Jain](https://x.com/SJ_Swapnil_Jain)) was running a race of his own. He never opened a PR on Colkitt's repository; none of the 13 accounts that did is his. He kept [Swapnil-jain/integer-mult-kappa](https://github.com/Swapnil-jain/integer-mult-kappa), posted six numbered updates on X between 07:20 and 14:03 UTC, and the sixth ended with a sentence nobody on the CrocSwap side could write about their frontier: "Both interchange certificates and the full assembly are checked in Lean's kernel." I cloned it and read all of it, and I wanted to know two things. Is this really a separate result, and what exactly did Lean check?
I lined the six posts up against what was open on CrocSwap at the same minute. The witnesses are the exact rationals from the commit each post announced; they are the purple diamonds on the chart above.
| post (UTC) | Jain's $\kappa$ | best open CrocSwap claim then | what was new |
|---|---|---|---|
| 07:20 | $249/(5\cdot10^{10}) \approx 4.98\times10^{-9}$ | #7, $3.73\times10^{-9}$ | the "ε → 1 stack" |
| 08:15 | $7499/10^{12} \approx 7.5\times10^{-9}$ | #9, $3.8\times10^{-9}$ | Gaussian correction inverted block by block |
| 10:05 | $\approx 4.188\times10^{-6}$ | #18, $1.885\times10^{-6}$ | batching on a two-stage bit network |
| 11:10 | $\approx 1.1972\times10^{-5}$ | #28, $1.226\times10^{-5}$ | PR #24's bit network, complex source frames |
| 12:39 | $\approx 1.5479\times10^{-5}$ | #37, $3.851\times10^{-5}$ | his own bit network on one flag basis; first Lean file |
| 14:03 | $3666565558019/10^{17} \approx 3.6666\times10^{-5}$ | #43, $4.0995\times10^{-5}$ | copied centres; both savings and the assembly in Lean |
So he held the lead twice: for about an hour after his first post, until icekylinx's #10 at 08:26, and for nineteen minutes after his third, until Zhihao Chen's #21 at 10:24. His fourth post went out one second after #28 was opened. From round five on, the CrocSwap side was ahead, by 2.5 times at 12:39 and by about 12% at 14:03. His own ratios check out: round six is 2.37 times round five, and $2^{-182}$ to $3.67\times10^{-5}$ is a factor of $2^{167.3}$. The one soft spot is the first post's comparison with Colkitt's $8.3\times10^{-11}$ ("a roughly 60 fold improvement"): by then Colkitt's `main` was already at $2^{-31}$, so against the newest checkpoint it was about ten.
"Independent" is the wrong word for it, and Jain doesn't use it. His NOTICE pins Colkitt's `6e56487` and cites fourteen open CrocSwap PRs (#3 through #36) plus Aurel Prosz's fork. Round four reads PR #24's published network counts from a JSON file pinned by SHA-256. Round six's copied centres are, in the first sentence of his note, "PR #36's copied retained-centre schedule (icekylinx)" applied to his network. The traffic went the other way as well. Chen's #29 and Rohan Arun's #31 credit "Swapnil Jain for the linked two-stage batching development", icekylinx's #36 links Jain's round-three commit with the line "no source code imported from this repository", and #44 and #47 list him with everyone else. These are two forks of the same argument, reading each other's work all morning.
What is his is the part of the stack that turns a network saving into $\kappa$. Through PR #13, $\kappa$ sat at about half the bit network's saving: $7.699\times10^{-7}$ against $1.54\times10^{-6}$. His README explains why. A few costs tie the saving either to the number of axes $d$ or to the axis width $\ell$, with $d\ell \approx \log n$, so the saving gets split between them, and the Gaussian resampling's precision budget, about $8d^2$, has to fit inside $O(\log n)$, which keeps $\epsilon$ below $1/2$. His first round made five changes at once. The coefficients become $(\log n)^{1+x}$ bits long instead of about $\log n$, so precision no longer competes with address length. The Gaussian correction is inverted block by block (every block is "the same Toeplitz matrix up to a geometric rescaling, so one stored inverse serves the whole line", as the second post puts it). Each axis becomes one chunk, only the low bits of an axis are moved for resampling, and the axis reversal reuses Colkitt's selected-bit additions. With all of that, $\epsilon$ can go almost to 1, and that is the "ε → 1" in every post.
I recomputed the round-six witness from the numbers in his certificate, without his code. The bit saving is $a_{\rm b} = 36667/10^9$, the complex saving $a_{\rm c} = 36926111/(5\cdot10^{11})$, so $\tau = 1-a_{\rm b}$ and $\lambda' = \tau + 2\cdot10^{-16}$. He sets $\epsilon = 2499908335861049/(2.5\cdot10^{15}) \approx 0.9999633$. Of his five cost margins, $1-\epsilon$, $1-\epsilon$ minus a $10^{-22}$-sized term, and $\max(\epsilon,1-\epsilon)(1-\tau)$ all come to $3.6665655580\times10^{-5}$ and a bit; $\epsilon(1-\lambda') \approx 3.666565558020684\times10^{-5}$ is the smallest; and $\epsilon$ itself is about 1. The claimed $\kappa = 3666565558019/10^{17}$ sits $1.7\times10^{-17}$ under the binding margin. The tidy way to say it: $\epsilon$ is chosen so that $1-\epsilon = \epsilon\,a_{\rm b}$, which makes $\kappa = a_{\rm b}/(1+a_{\rm b})$ to eleven digits. Every bit of the network's saving reaches $\kappa$.
The CrocSwap side got to the same place by another road. RaD/hipotures' #20 and Chen's #23, at 10:20 and 10:43 UTC, used "completed-child semantic precision" and bulk resampling, and neither cites Jain. Round four is a neat accidental experiment. Jain and Dominik Scholz's #27 took the same PR #24 bit network, with $a_{\rm b} \approx 1.19724\times10^{-5}$, put it through their two different assemblies, and got $1.19722292\times10^{-5}$ and $1.19720853\times10^{-5}$. Both stacks now pass essentially the whole network saving through, so the 12% gap between 3.67 and 4.12 is entirely the bit network: $36667/10^9$ here against #48's $82375901/(2\cdot10^{12})$.
The ε → 1 stack also has a price that the headline hides. With $\epsilon$ near 1, the coefficient-depth guard needs $\epsilon C_1 < 1+x$. Round six uses the crude guard $C_1 = 14694$ (from $2 + 576\ln 119453132304 \approx 14693.6$), so $x = 14695$ and each coefficient is $(\log n)^{14696}$ bits long. In round five the same rule gave $x = 842135$. That is legal, because $x$ is fixed and the volume stays $\Theta(n)$, and his certificate says as much ("The exponent is huge but fixed, so the O() statement holds", `scripts/certificate_round3.py:15`). It is one more reason the runtime section below holds for this track too. The stack notes also list "deliberate departures from upstream hypotheses": the prime number theorem replaces the manuscript's explicit interval lemma, the resampling hypothesis is weakened, and so on. So this track rests on more than the CrocSwap PRs do: the manuscript, the fourteen PRs it cites, and its own replaced hypotheses.
### What the Lean files prove
There are two, `lean/Round5.lean` and `lean/Round6.lean`, and neither is written by hand. `lean/gen.py` emits them from JSON files that `dump.py` and `dump6.py` write out of the Python certificates. They are core Lean 4: no `import`, no Mathlib, no lakefile and no `lean-toolchain`, so no version is pinned; `make lean` runs `lean lean/Round6.lean` with whatever is on the PATH. Rationals are pairs of naturals with a hand-written gcd normalisation, $\ln$ is a 30-term atanh series with a geometric tail bound, and each file ends in five theorems. Round six's, from `lean/Round6.lean:74-81` with the histogram lists cut:
```lean
theorem bit_rank_sum : rankSum bitHist = 78410006675 := by decide
theorem bit_moment : momentOK bitHist (36667, 1000000000) 529 148225616 = true := by decide
theorem cx_rank_sum : rankSum cxHist = 119453132304 := by decide
theorem cx_moment : momentOK cxHist (36926111, 500000000000) 576 207387136 = true := by decide
theorem kappa_assembly : kappaOK (36667, 1000000000) (36926111, 500000000000) (1, 1000)
(2499908335861049, 2500000000000000) (14695, 1) (14694, 1)
(3666565558019, 100000000000000000) 576 119453132304 = true := by decide +kernel
```
There is no `sorry`, no `axiom` and no `native_decide` in either file. Both `decide` and `decide +kernel` end in a proof term that the kernel checks; `+kernel` only skips the elaborator's own evaluation first. `native_decide` would have compiled the decision procedure and trusted the compiled code, and it isn't used. The kernel does use GMP for natural-number literals, which is part of Lean's normal trusted base. So yes, it really is the kernel.
The more important question is what the theorems say. They say that three particular Boolean functions return `true` on particular lists of numbers. They do not say those lists are the child-width histograms of the networks; `gen.py` pastes them in from Python. They do not say `lnUp` bounds the logarithm or that the moment sum controls the recurrence. The generator's docstring is explicit: "The analytic facts behind (2) and (3) -- the atanh tail bound and (m/w)^a <= 1/(1 - a ln(m/w)) -- are premises, not checked here: Lean core has no real logarithm." And `kappaOK` encodes his own five-margin cost table, transcribed from `certificate_round3.py`; whether those five margins are the right costs is a question for the written notes. The upstream theorem isn't in it at all. His README is honest about this; its evidence table lists the full upstream multiplication theorem as "Assumed" and independent review or formalisation as "Not supplied". So "checked in Lean's kernel" means that the arithmetic the Python fraction certificate already did has been redone by a much smaller and better-trusted checker. The check is real, and narrow.
I couldn't run it: there is no Lean toolchain on this machine, and I wasn't going to install one for this. So I wrote the five definitions out again in Python on integer pairs, including Lean's `qsub`, which is natural-number subtraction and quietly floors at zero, and evaluated the theorems on the committed inputs. All five come out true, and no subtraction ever hits the floor. The bit moment sum is $1 - 2.79\times10^{-10}$ of its bound and the complex one $1 - 7.9\times10^{-14}$. At 80-digit precision, with the real exponential, the largest bit saving this histogram supports is about $3.66702\times10^{-5}$, so the claimed $3.6667\times10^{-5}$ leaves 0.009% headroom; round five's file checks out the same way.
Lean wasn't new to the race, either. princezuda's #26 on CrocSwap, at 10:54 UTC, brought Lean 4 v4.21.0 with Mathlib ("no sorry and no native_decide") and 310 theorems over the certificate arithmetic of PRs #3 to #12, and it found one misstated ceiling. alejandrozu's #45 adds a Lean-checked Gaussian parity audit. But no CrocSwap claim after #12 has its arithmetic in Lean, and Jain's last two rounds do.
His Python checkers all ran. `make verify` took 194 s at 750 MB peak and passed everything, including 23 unit tests and a deliberately broken inverse that has to fail and does. `independent/complex-twostage/run.py 24` took 39 s: 140,064 edges, 0 bad labels, and $a_{\rm c} \ge 73861113/10^{12}$, slightly more than the $36926111/(5\cdot10^{11})$ the assembly uses. `scripts/certificate_round6.py` gave $a_{\rm b} = 36667/10^9$ at $h = 23$. The histograms these scripts regenerate are identical to the ones pasted into `Round6.lean`, so the Lean file and the Python checks are about the same objects.
Where that leaves the race, at 15:46 UTC: the highest claim is Rohan Arun's #49 on CrocSwap, $4.1239\times10^{-5}$, 12.5% above Jain's $3.6666\times10^{-5}$. The number on CrocSwap's `main`, #39's $3.8867\times10^{-5}$, is 6% above Jain's. So there are now three kinds of checking on the board: Jain's arithmetic in Lean, `main`'s new steps read through by its maintainer, and the open PRs' own Python certificates. And `main`'s README now credits Jain, with Aurel Prosz, for "attributed two-stage development and the paid copied-stream endpoint construction", so the two races have formally met. I don't think the difference in arithmetic checking matters much, because the arithmetic was never the likely point of failure. Lean makes "the numbers add up" airtight; it says nothing about whether a contiguous projector block really is one recursive call on a tape, or whether the manuscript underneath is right. Jain said it himself, replying to someone who asked for the unconditional value: "Nothing past OpenAI's result is unconditional yet."
## PR #39 reaches main
Rohan Arun opened #39 at 12:58 UTC, "Fixed middle basis and copied reversed corners", at $3.8867\times10^{-5}$. It was never the best open claim for long; his own #40 beat it by 0.82% seventeen minutes later. But at 13:23 Colkitt had pinned it, head `70ae241`, in a merge commit (`fd8c563`) on an `integration/community` branch, and he spent the next two hours on it. At 15:12 UTC he committed "Complete conditional audit of the PR39 community witness". At 15:18 the release commit `0605a24`, "Publish audited community bound with contributor attribution", moved `main`, the tag `community-kappa-15-2026-10-08` went on it, and GitHub marked #39 merged. Since #39 was stacked on earlier work, the same push also marks icekylinx's #10, #18, #24, #32 and #36 and Rohan's #37 as merged; their commits are in its history. Twelve minutes later Colkitt posted:
> Validated and merged Rohan's PR. Big gain on a really difficult regime (that frankly I was stuck at). Incredible work.
The maintainer admitting he was stuck, and that someone else's agent-assisted PR got him unstuck, is my favourite sentence of the whole episode. His headline, "κ = 2⁻¹⁵", is the floor the witness clears rather than its value; the exact number is $2^{-14.65}$, which the README puts at "27.36% above 2^-15". His "500 thousand fold improvement over the previous result" checks out against $83/10^{12}$, the last exact witness he had posted (about 468,000-fold). Against the $2^{-30}$ checkpoint that `main` carried until 15:18 it is about 42,000-fold. The "2 ^ 167 fold improvement over the original OpenAI result" is right: $2^{167.35}$.
What did "validated" mean? The audit, `docs/research/community-final-audit.md`, is plain about who did it. Its header reads "Reviewed by the project's Codex assistant for Douglas Colkitt", and it calls itself "a maintainer mathematical assessment, not independent human peer review or formal verification." Within that, it is a real proof review, and it goes after exactly the obligations my PR #13 check said the certificates never touch. It works through the partial-swap compiler's block algebra; the claim that one fixed basis serves every gate (each required minor is a nonzero rational function on one irreducible family, so a single rational basis avoids every exceptional set); the copied-centre schedule restoring both operands for arbitrary dirty values; the complex side's phase compiler; the semantic-precision recurrence; the arbitrary-coordinate router; the Gaussian inverse; and the bulk tape schedule. In one place it swaps in a published theorem, Baker, Harman and Pintz on primes in short intervals, and labels it "an actual new input". Its verdict: "no unresolved additional construction or inequality was identified that blocks this particular witness."
The executable side is separate. The audit added its own checker, `scripts/audit_community_candidate.py`, which imports none of the contributor's code, encloses both recursive moments with 80-term rational logarithm bounds, rebuilds all seven margins, gets the same final gap of about $6.25\times10^{-15}$, and shows that the next bit saving on the $10^{-14}$ grid fails. It reports 218 tests, 20 upstream patch checks and certificate regeneration in Docker (GCC 13.3, Python 3.12), a GitHub Actions run on Python 3.11, 3.13 and 3.14, a one-header GCC build fix, and 20 files missing from a vendored manifest restored from PR #34 at their recorded hashes. I re-ran `make verify` on `0605a24` and it passed, 218 tests and all, in about sixteen minutes ([How I checked](#how-i-checked) has the details).
Two limits are written into the audit itself. "The original OpenAI #109 framework ... remains assumed." And "PR40 and subsequent submissions are outside this pinned review." So `main` is deliberately behind the frontier, by 6.1% when #49 arrived.
## Main moves again
I thought the merge was the end of the story. When I fetched everything again at 19:46 UTC, `main` had moved twice more, 28 more PRs had arrived, a contributor had built a live dashboard for the race, and the race itself had jumped to a second problem.
The first move was routine. At 16:47 UTC GitHub marked Rohan's #49 merged, together with #43, #45, #46, #48 and princezuda's Lean PR #26, and the README went to $4123863984/10^{14}$. It was behind again at once: Rohan Gupta's #50 and RaD's #51 had been open for half an hour with higher claims.
The second move is the one that broke the plateau. Avi Eisenberg's #53 (he is `ikeboy` on GitHub), opened at 16:31, changes nothing except the scalar producer. Every vertex of the network needs its leave-one-out strip sums: given $k$ values, all $k$ sums that leave one of them out. The usual way builds prefix sums $A$ and suffix sums $B$ and takes $s_j = A_{j-1} + B_{j+1}$. Each prefix then has two consumers whose envelopes don't nest, so its second use always costs a fresh wire. #53 shifts the bracket by one, $s_j = A_{j-2} + (v_{j-1} + B_{j+1})$, and now the two consumers of a prefix are nested, so the carrier matching from #44 can keep the existing wire going instead of allocating a new one. It costs more additions (40,329 against #48's 37,098 at $h = 23$), but the carrier links double, from 6,002 to 12,719, and the roles drop from 36,432 to 32,946. It bought 9.2% in one step, after three hours in which every PR had gained a few hundredths of a percent. I like it because it is counter-intuitive: more arithmetic, fewer wires, and fewer wires is what $\kappa$ actually pays for.
After that the ideas stacked. eumemic's #57 compiles every group of operations that share a frame as one invertible binary map and pays to reclaim retired wires, which takes the physical wire count from 160,799,739 to 153,481,944. Eisenberg's #62 replaces the strips with every cyclic interval, each built from an interval one item shorter. It needs $k(k-2)+1$ additions instead of about $4k$, but almost every one of them continues a carrier. On its own it claims $4986133/10^{11}$, and with #57's compiler on top, $5101691/10^{11}$. Alejandro Zarzuelo Urdiales's #61 then refined the parameters on that same graph, by $1.0170078\times10^{-11}$, with the arithmetic checked in eleven Lean proofs.
At 18:35 UTC, Colkitt merged #50 to #62 (all except Rohan Garg's #59, which is kept as a reviewed alternative) in one push. `main` now reads
$$
\kappa = \frac{25508460085039}{5\cdot10^{17}} \approx 5.1017\times10^{-5} \approx 2^{-14.26},
$$
23.7% above the #49 release and 31% above #39's. The network underneath has changed too: the bit side now runs at $m = 575$ on dimensions 23 and 25, with 137,151,806 physical roles.
The review behind it, `docs/research/community-round2-review.md`, again calls itself "a conditional maintainer review with Codex assistance, not independent human peer review or a formal proof of multiplication." It reads more like a replay ledger than the #39 audit did. Each PR gets a disposition ("Full pinned replay passed", "Complete word and profile replay; used by selected witness"), and the places it holds back are stated. #51's separate witness had its "arithmetic … freshly replayed, but its entire standalone source-graph and negative-basis data argument was not", and #62's claims that its layout is optimal "were not independently reproduced". The general residual compiler, the all-size recursion, the tape implementation and the analytic transfer "remain written inherited obligations, not consequences of a successful finite test alone." So this move rests on finite replays plus reading, on top of interfaces that were audited at #39.
The same afternoon, `main` imported Jain's repository whole: 87 files, pinned by SHA-256, under `research/swapnil-parallel/`. It did the thing I couldn't. It compiled both Lean files under Lean 4.31.0, checked that the generator reproduces them byte for byte, and inspected all ten theorems, none of which depends on an axiom. Its reading of what they prove matches mine: they "do not formalize logarithmic analytic bounds, construct a Turing machine, or prove integer multiplication end to end", and the fixed-tape side of the ε → 1 stack has "not received a complete independent audit in this pass." It also notices something that matters for the next section. Jain's complex network saves $36926111/(5\cdot10^{11}) \approx 7.385\times10^{-5}$, more than the 7.17e-5 of the complex network `main` uses, but `main`'s bit side binds at about $5.10\times10^{-5}$, so the better complex network would not move the headline. Jain himself has not posted a seventh round. His last commit is still round six, at 14:01 UTC.
The open PRs didn't wait. Fifteen more arrived between 18:05 and 19:46. Dominik Scholz's #63 and rfu08's #64 published the same construction within fifteen minutes of each other, and #64 says so ("Both decompressed words, complete profiles and finite bridge are identical"). The best open claim when I stopped was Scholz's #76, opened at 19:44, which combines three other open PRs (#70, #71 and #74) to reach $51738676045141/10^{18} \approx 5.1739\times10^{-5}$, 1.4% above `main`.
So at 19:46 UTC the board reads: `main` at $5.1017\times10^{-5}$, the best open claim at $5.1739\times10^{-5}$, and Jain at $3.6666\times10^{-5}$, now 28% behind `main`. There have been 77 PRs from 20 accounts; GitHub marks 25 merged, 47 are open and 5 were closed without merging. The last fifteen open claims sit between 5.10 and 5.17 times $10^{-5}$. It's a plateau again, and I'm no longer going to predict how long it lasts. (About three hours, [it turned out](#9-october-the-ninefold-night).)
### A dashboard, built by one of the racers
At 15:13 UTC Aurel Prosz ([@aurel_pr](https://x.com/aurel_pr), Paureel on GitHub), whose two-stage topology and endpoint correction run through half the PRs above, posted a live tracker: [Beyond n log n](https://beyond-n-log-n.netlify.app/). Colkitt shared it fifty minutes later as an "awesome dashboard from the great @aurel_pr". The site owner's reaction was "damn, things are getting serious", which is fair. The page names no author in its text, but its footer links to Paureel, and the source is open at [Paureel/beyond-n-log-n](https://github.com/Paureel/beyond-n-log-n).
It is a React app with a Netlify function that re-reads GitHub every 15 minutes. It covers every PR in every state, all 26 public forks and the 6 PRs inside them, each checkpoint in `main`'s history, a "field map" of which areas of mathematics each PR mentions, and an idea-lineage graph built from explicit reuse language in the PR descriptions. The $\kappa$ values are parsed from PR titles and bodies, so they are claims, and the methodology box says as much: "No full-theorem verification is inferred from tests or certificates."
I pulled its JSON (`/api/research`, fetched at 19:33 UTC) and compared it with my chart. All 46 PR claims from #1 to #49 agree to within 0.2%, which is the rounding in my labels. There are two real differences. The tracker puts OpenAI's $2^{-182}$ at the `openai/math` commit time, 21:58:50 UTC on October 6, where I had used the 3:19 PM Pacific (22:19 UTC) printed on Julian Schiavo's chart. I checked the commit and moved my point to match. And its "repo checkpoint" line steps at the commit times of everything in `main`'s history, including contributors' commits that came in through merges, so it has `main` at 4.1239e-5 from 16:16 UTC (`ed8201c`), where I use 16:47, when GitHub marks the merge. Both choices are defensible. Its 19:33 snapshot also predates #75 to #77, so it showed #71 as the leader.
## The race reaches the Fourier transform
At 16:52 UTC Ryan Shea ([@ryaneshea](https://x.com/ryaneshea)) posted:
> We're publishing a research draft on OpenAI Problem #130: computing the exact discrete Fourier transform below n log n. Our draft proposes an all-length bound of T(n) = O(n(log n)^(1−δ)), with δ = 7.3×10⁻⁵. That's a 730-million-fold increase in the exponent saving over OpenAI's published δ = 10⁻¹³.
The repository was created at 15:26 UTC with Shea's own earlier construction, a reduced centre and shared sums, worth about $5.5\times10^{-10}$, or roughly 2,600 times the saving OpenAI's construction actually achieves. An hour later, at 16:27, he replaced it with Jain's round-six complex network and got $7.3\times10^{-5}$. Asked whether this was AI-assisted, he replied: "The heavy lifting was AI, and I shepherded it towards the improvements."
Why should a network built for multiplying integers say anything about the FFT? The [roundup's FFT section](/articles/openai-math#how-you-beat-nlog-n-save-on-one-tiny-matrix-then-amplify) explains it. OpenAI's explicit Fourier paper (family 130) takes its finite gadget from the multiplication manuscript. The saving lives in one place: an algorithm that applies $C^{\otimes k}$, tensor powers of the fixed $2\times2$ matrix $C = \tfrac12\begin{pmatrix}1+i&1-i\\1-i&1+i\end{pmatrix}$, in $O(2^k(k+1)^\theta)$ with $\theta < 1$. That algorithm is the same frame-labelled network on arbitrary roles that the race has been tuning: swap two banks, restore every scratch wire whatever it held, and count the dimensions of the frame changes. The rest of the Fourier proof (Good–Thomas reindexing, exact-width local words, synchronisation into sectors, a Bluestein chirp) uses the tensor routine only through that cost bound. The draft's whole argument rests on this. By its reading, Proposition 4.2 of the Fourier paper uses the tensor routine only through the bound $O(2^k(k+1)^\theta)$ on arbitrary complex arrays. So a better network gives a better $\theta$, and $\delta$ is anything strictly below $1-\theta$.
What crosses over is narrow, and the draft is careful about it. It takes only Jain's complex network at $h = 24$: 2,024 triples, $m = 576$, $W = 207{,}387{,}136$ roles and a total child width of $s = 119{,}453{,}132{,}304$, with icekylinx's whole-residual batching (#10), the copied centres from #36 and Prosz's two-stage endpoint correction, all credited. It does not take the bit network, the ε → 1 stack, the Gaussian resampling or any of the tape and precision analysis. Its audit says it "does not use Jain's integer-specific epsilon stack, bit-interchange bound, Gaussian recovery or tape precision analysis".
So the Fourier number comes out large next to the multiplication race, and for a structural reason. In #109, $\kappa$ is the smallest of seven margins and the bit network binds; Jain's complex saving was slack. The Fourier model is exact complex arithmetic with word-sized addresses. There are no tapes, no bit network and no $\epsilon$ to share the saving with, so $\delta$ can sit just under the complex network's own saving, $a = 36926111/(5\cdot10^{11}) \approx 7.385\times10^{-5}$. In the draft's words: "There is no integer-multiplication bit-network constraint or extra factor-of-two loss in this transfer." The half of Jain's work that never mattered for his own $\kappa$ is the half that matters here. It is also the best complex network on offer: as noted above, `main`'s own is 7.17e-5.
I checked the numbers. From the draft's role counts, $W = 2N + 2vR$ with $N = 2024^2$ and $R = 49{,}208$ gives 207,387,136, and $Wm - N + L$ gives the stated $s$. The deficit is $Wm - s = 1{,}858{,}032$, and the 25-row histogram in Section 6 sums to the same $s$, the rank sum that `Round6.lean` checks for the complex side. With exact fractions, $a - \delta = 426111/(5\cdot10^{11}) \approx 8.5\times10^{-7} > 0$, which is the gap that swallows the extra $(\log\log n)^{4-\theta}$ factor. At 80 digits with the real exponential, the moment at $\theta = 1-a$ is $1 - 1.87\times10^{-9}$ (Colkitt's replay of Jain's file reports the same gap), and bisection puts the critical saving at $7.38611\times10^{-5}$, the "critical root" the draft mentions. The draft's own rational bound, which uses $e^x \le 1/(1-x)$, is looser at $1 - 7.9\times10^{-14}$, and it still clears 1. The headline ratios are right: $7.3\times10^{-5}/10^{-13}$ is exactly 730 million, and against the earlier $5.5\times10^{-10}$ it is 132,727 times. Against OpenAI's actual critical saving of $2.106\times10^{-13}$ (the paper rounds it down to $10^{-13}$ for its headline) it is about 350 million.
Its checkers are standard-library Python, and I ran them under the same exception as before. `verification/run_checks.py` took 80 seconds at 245 MB. It reproduced OpenAI's own $a = 2.10643843\times10^{-13}$ from the paper's $W$ and $\Delta$, passed both earlier suites and matched their archived certificates, and then ran the round-six verifier: hashes of the vendored Jain files, all 4,096,576 local coefficient entries, both physical schedules with the copied centres charged, the full histogram, exact endpoint identities, negative controls, and "PASS: a=36926111/500000000000, delta=73/1000000, rational moment gap=7.919…E-14". The regenerated certificate matched the committed reference.
The draft is plain about the limits, and so is the best reply it got. The producer is Jain's code, vendored, and the extra checks "were written with the same assistant"; the dirty-scratch tests are finitely many rational examples; nothing executes the full DFT or the infinite recursion. Keith Adler went through it with Claude as referee ("treat it as a second pair of eyes, not human review") and couldn't break it. He listed what to tighten: the copied-centre step "is argued, not tested"; whole-residual batching "is not in #130" and needs a short extension of its Lemma 2.2; and the margin is thin, "a = 7.3852e-5 sits just under the critical root 7.3861e-5".
So what is it conditional on? First, on OpenAI's explicit Fourier manuscript, Proposition 4.2 and Section 5.4 in particular, which the draft invokes "with their stated hypotheses, not re-proved". As [the roundup notes](/articles/openai-math#the-fft-is-not-optimal-in-the-right-model-by-a-hair), the Lean statement for family 130 is a weaker one: a subsequential, non-constructive circuit bound from the second paper, with "no all-length, bounded-coefficient, conditioning, or bit-complexity claim". The explicit $\delta = 10^{-13}$ algorithm is manuscript-only, and that is the part Shea improves. (That was true of the first release; OpenAI's update on the morning of 8 October, which I missed, put it in Lean too. See [10 October](#what-the-lean-proof-covers).) Family 109 has no Lean statement at all, so the two problems sit at different levels: #130's core question is settled in Lean and its explicit constant is not, while #109's whole theorem is a manuscript. Second, it depends on Jain's finite complex network and the written arguments around it (copied centres, whole-residual batching, the two-stage endpoint). It does not depend on his assumption-heavy ε → 1 stack or on the hypotheses he replaced. Those network arguments are the same kind Colkitt's audit worked through for #39, but in the multiplication setting. Their transfer into the Fourier paper's array model, including a layout lemma extended from one selected slot to $r$ and a final address translation for the endpoint, is new and unreviewed. In one way this rests on less than Jain's own $\kappa$ does. In another, nobody but Shea's assistant and Adler's has read the new part yet.
It changes nothing about real FFTs. The README says "No practical FFT speedup … is claimed", and the [roundup's numbers](/articles/openai-math#how-small-is-10-13) still apply in spirit. Ignoring constants and the $\log\log$ factor, a 1% saving at $\delta = 7.3\times10^{-5}$ needs $\ln\ln n \approx 138$, so $\ln n \approx 6\times10^{59}$. It's a big improvement on the $4.8\times10^{10}$ the roundup found at $10^{-13}$, and it is still not a number anyone will ever transform. What it shows is that the race is not about one problem. The agents and the people pointing them have found a shared engine, and any result built on that engine is now open to the same tuning.
## 9 October: the ninefold night
The site owner's note the next morning was short: the race to κ = 1 is still on, check the repos again. When I left it, at 19:46 UTC on October 8, `main` was at $5.1017\times10^{-5}$, the best open PR at $5.1739\times10^{-5}$ and Jain at $3.6666\times10^{-5}$, and I had called it a plateau. At 11:00 UTC on October 9 `main` was at $4.609169\times10^{-4}$, the best open PR at $6.5592\times10^{-4}$ and Jain at $6.1534\times10^{-4}$. A hundred and two PRs had arrived in between. There were 179 from 39 accounts; GitHub marks 30 merged, 127 open and 22 closed. So much for plateaus.
### What moved `main`
The ninefold step is one contributor's chain of four PRs, each built on the one before. icekylinx opened #104 at 23:12 UTC (a "stopped product-ring interchange", $7.79476\times10^{-5}$), #115 at 00:26 (partial source gauges, $9.26336\times10^{-5}$), #130 at 02:29 (three-stage Cayley covers, $3.146011\times10^{-4}$) and #144 at 04:19 (paired cubes and shared completed cores, $4.609169\times10^{-4}$). Other people were tuning around them all night (eumemic's #114 was the first claim past $2^{-13}$, at 00:23), but #130 is the step that matters: it came in 2.4 times above the best open claim of the moment. Colkitt reviewed the chain and merged all four through an integration PR, #149, at 05:06. Ten minutes later he posted:
> Validated and merged a breakthrough from IceKylin. In addition to their work, the latest result builds on work from @dysmemic, @SJ_Swapnil_Jain, an664, Zhihao Chen and others.
His "~9X from this morning" is right: $4609169/10^{10}$ over yesterday evening's $25508460085039/(5\cdot10^{17})$ is 9.03. So is "roughly 2¹⁷¹": it is $2^{170.92}$ times OpenAI's $2^{-182}$. A second post in the thread said the agents were "still backed up trying to process the huge volume of PRs that keep coming in. (Keep them coming!)".
### Three shears instead of one swap
Every construction in this race is a way of doing one thing: an interchange, which swaps two banks of values $X$ and $Y$ while putting every scratch wire back the way it found it. Until last night each interchange was a single network on about $h^2$ coordinates (576 for $h = 24$; `main`'s bit side ran at 575), with an enormous number of roles per invocation: 137,151,806 on `main`'s bit side yesterday evening.
#130 does the swap the way you were taught to swap two registers without a temporary:
$$
(X, Y) \;\to\; (X,\, Y+X) \;\to\; (-Y,\, X+Y) \;\to\; (-Y,\, X).
$$
Three shears, $Y \gets Y+X$, then $X \gets X-Y$, then $Y \gets Y+X$, give the swap up to a sign, and the sign is a fixed correction at the end. The trick is in where each shear runs. The address space is split into three orthogonal blocks $A \perp B \perp C$ of dimensions $h$, $h-1$ and $h-1$, so $m = 3h-2$ (70 in #130; #144's complex side uses 72). Each stage acts on one block plus a single shared direction, and the frame a value leaves one stage in is exactly the frame the next stage expects, so there are no data connectors to pay for between them. The note puts it as "Every exit label is precisely the next entrance label."
To make every vertex of the network look the same, the invocations are indexed by every element of the orthogonal group of that binary space, which is what "regular Cayley cover" means here. For $m = 72$ that group has about $2^{2555}$ elements. The count cancels out of the normalized moment, so it costs nothing in κ, and it lands in the constant instead. eumemic's later README puts the network at "about 2^2570 roles". #144 then added two things on top: a "paired-cube" local word that writes the identity as $I = K + H + B$, three pieces built from different parts of the circuit, and an664's completed-core sharing from #128, in which one auxiliary bank is reused in turn across the three orthogonal blocks.
### Why smaller is worth nine times
The moment condition from #10 explains the gain, and once you expand it to first order it says something you can hold in your head. A network with $W$ roles on $m$ coordinates spends its rank on children of widths $r$; the total falls short of $Wm$ by the deficit $D$. Then
$$
a \;\approx\; \frac{D/(Wm)}{L}, \qquad L = \sum_r \frac{n_r\, r}{Wm}\,\ln\frac{m}{r},
$$
where $L$ is the rank-weighted average of how far below full width a child sits, on a log scale. The saving is the fraction of rank the network fails to spend, divided by the log-distance of a typical child from full width.
I computed both sides of that for the old network and the new one, from their own histograms. Jain's round-six complex network (the one Shea took to the Fourier problem, and better than yesterday's `main`) has $D/(Wm) = 1.555\times10^{-5}$ and $L = 0.2106$: 96% of its rank sits in children wider than half of $m$, so wide children are nearly free. That gives $a \approx 7.387\times10^{-5}$, against a certified $7.385\times10^{-5}$. #144's complex side has $W = 29{,}937$, $m = 72$ and $D = 1{,}936$, so $D/(Wm) = 8.982\times10^{-4}$, 58 times larger. Its children are narrow (only 13.5% of the rank sits in children wider than 36), so $L = 1.848$, 8.8 times larger. The ratio comes to $4.860\times10^{-4}$, against a certified $4.856569\times10^{-4}$. The relative deficit grew much faster than the log penalty, and the net is a factor of 6.6 on this side. The deficit itself telescopes neatly, $D = 2v - 3\ell$ per vertex: $2\cdot1760 - 3\cdot528 = 1936$.
### The margin that binds
The assembly under #144 is CrocSwap's semantic assembly: 47 strict constraints, seven cost margins, and two small parameters $\eta = 10^{-8}$ and $\beta = 10^{-6}$. The network saving fed to it is $a = \min(a_{\rm bit},\ (1-\beta)a_{\rm c} - 10^{-10})$. The bit side binds: $a_{\rm bit} = 4613422943/10^{13}$ against $a_{\rm c} = 4856569/10^{10}$. Then $q = a(1-2\eta)$, $c = q(1+\eta)$ and $\epsilon = (1-\eta)/(1+c+q) \approx 0.99908$, and the binding margin is the compact phase layer, $\epsilon q$. The original-prefix margin $1 - \epsilon(1+c)$ sits exactly $\eta$ above it, which is the point: $\epsilon$ is chosen to balance those two, so
$$
\kappa \;\approx\; \epsilon\, a \;\approx\; \frac{a}{1+2a}.
$$
I redid that with exact fractions and got $\kappa = 4609169/10^{10}$, with the binding margin $9.945\times10^{-11}$ above it (the $10^{-10}$ grid swallows the rest), the same gap the maintainer's review reports.
One wrinkle on the bit side: its saving is "stopped". #104 pays the cross-bank address adapters as loops over fixed-width atoms, which costs linear time, and stops the recursion at atom width $e^{\theta}$ with $\theta = 1/1000$. The usable saving is then $(1-\theta)a^* + \theta\,a_{\rm old}$, a thousandth of it coming from an older, weaker network ($a_{\rm old} = 384599/10^{10}$). Unstopped, #144's bit network works out to $4.6177\times10^{-4}$, and the stop costs it about 0.1%.
The review behind the merge, `docs/research/community-round6-review.md`, is "a construction and dependency review, not independent human peer review or a formal proof of integer multiplication", done "with OpenAI Codex assistance". It rebuilt the complex producer, ran its own controls (exact factorizations over $\mathbb Z/9$, $\mathbb Z/25$ and $\mathbb Z/27$, exhaustive frame identities in dimensions two to four, and dirty-value replays of the 26,417-role mixer), read the written proofs, and prepared two small fixes, a verifier guard and one sign in a sentence. It is honest about one gap: a 1.5 MB compressed bit witness from #97 could not be fetched through its connector, so "the selected bit reconstruction was checked through inspected, head-pinned CI, not rerun locally." That file is in my clone, so I ran `make paired-cube-verify` on `main`: the producer regenerated from source and matched its certificate, the bit reconstruction ran locally with every profile and source hash equal, and it ended `PASS kappa=4609169/10000000000 = 4.609169e-4; both moments, shared cores, finite router and 47 strict constraints`, in 44 s at 270 MB.
### Jain's rounds seven to ten
Jain posted four more rounds. Here they are against what was open on CrocSwap at the same minute, as before.
| post (UTC) | Jain's $\kappa$ | best open CrocSwap claim then | what was new |
|---|---|---|---|
| Oct 8, 20:56 | $1599247689723/(2.5\cdot10^{16}) \approx 6.397\times10^{-5}$ | #86, $5.2275\times10^{-5}$ | deferred garbage readout |
| Oct 9, 04:17 | $12612978530233/10^{17} \approx 1.2613\times10^{-4}$ | #143, $4.2483\times10^{-4}$ | opposite bank orders (#104), signed reclaim (#112), shared cores (#128) |
| Oct 9, 05:25 | $4663738/10^{10} \approx 4.6637\times10^{-4}$ | #147, $4.6466\times10^{-4}$ | his bit word inside #144's three-stage cover |
| Oct 9, 09:50 | $6153378/10^{10} \approx 6.1534\times10^{-4}$ | #168, $6.5589\times10^{-4}$ | paired-cube words, birth reuse, nested-prefix modules |
Round seven led everything for two hours, by 22% when it was posted. Zhihao Chen's #97 at 22:39 ported Jain's witness into CrocSwap and came within 0.007% of it, and #99 passed it at 22:56. Round eight landed in the middle of the cover jump and was 3.4 times behind. Round nine was the highest claim anywhere for fifteen seconds, until DaysSky's #150 at 05:25:46. Round ten was about 6% behind eumemic's #168.
What, then, is round nine's 9.14-fold jump over yesterday's `main` made of? Mostly icekylinx's cover. Jain's README says it plainly: `scripts/certificate_round9.py` is "our own implementation of the paired-cube / three-stage cover assembly of icekylinx's PR #144", written from #144's description and files, with "None of PR #144's code" run. It first reproduces #144's published $4609169/10^{10}$ exactly, all 47 constraints and 7 margins. Then it swaps one thing: the bit side's word becomes his own round-seven witness (deferred readouts at $h = 23$, $R = 27{,}794$), with 1,504 of its 10,922 deferred gauges omitted. Choosing which ones to omit is an exact $s$–$t$ minimum cut, a formulation he credits to Th0rgal's #146. That raises the bit network's saving from #144's $4.6177\times10^{-4}$ to $4.67238028\times10^{-4}$; the complex side is #144's, unchanged. So round nine is 1.18% above `main`, and that 1.18% is Jain's. The ninefold before it is the cover's. His round-ten README credits, by PR number, eumemic, icekylinx, DaysSky, jamesyc, Th0rgal and an664 for the levers it stacks.
On assumptions, round nine moves him closer to the CrocSwap side rather than further away. Rounds one to eight ran on his own ε → 1 stack and its "deliberate departures from upstream hypotheses", the prime number theorem in place of the interval lemma among them. Round nine's certificate doesn't import that code at all. It runs #144's assembly, which carries the hypotheses `main` carries. What it adds is a short list of interfaces "stated here but not machine-checked": the cover lifting, frame theorem, exterior rule and stopped recurrence from #130 and #144; a precision grid of the form $2^{-P}3^{-K}$, because the cubes divide by 3 ("This looks benign …, but only dyadic grids were audited before"); block-factored frame representatives; and two inherited row-reserve constants. The maintainer's #144 review does discuss the denominator-three grid, so that one has had a reading too.
I recomputed round nine with exact fractions, outside his scripts. From the certified $a^* = 116809507/(2.5\cdot10^{11})$, the stopped mix gives $a_{\rm bit} = 29175578117/(6.25\cdot10^{13}) \approx 4.66809\times10^{-4}$, which binds against the complex $4.856569\times10^{-4}$. Then $\epsilon \approx 0.999067$, the binding margin is $\epsilon q \approx 4.66373820648\times10^{-4}$, and the floor on the $10^{-10}$ grid is $4663738/10^{10}$, as claimed, $2.06\times10^{-11}$ under the margin. $\log_2$ of that is $-11.0662$, so "2⁻¹¹·⁰⁶⁶" is right, and the ratios in his post (3.70 over round eight, "2¹⁷⁰ fold") check out too, the second one rounded down from $2^{170.93}$. At 80 digits with the real exponential, the critical saving of his bit histogram (rare-class fallback included) is $4.672380285\times10^{-4}$, so the certified grid point leaves almost nothing on the table. `make round9` and `make round10` pass, and round ten's certificate re-derives #144's and round nine's κ before certifying its own. The heavier round-nine ledger, which replays the whole forward F2 shear of the modified bit word and rebuilds its child histogram from the events, printed `ALL PASS` after 110 s at 0.96 GB. Rounds seven to ten each ship a Lean file again (`lean/Round7.lean` to `Round10.lean`), still core Lean with no `sorry`, no `axiom` and no `native_decide`, and round nine's now includes the finite bridge and the 47-row assembly. There is still no Lean toolchain on this machine, so I didn't run them; my fraction recomputation covers the same arithmetic.
Two things in his thread are worth keeping. Told by a reply that he was "losing the frontier", he answered: "I found mistakes in the PRs there so have to be super careful. Public results I publish are 100% verified correct." And round ten's commit fixes a bug in his own F2 checkers, which had been identifying a kept copy in a way that skipped some walk and data checks; he reports that rounds seven to nine are unchanged under the fix. I'd trust the second more than the first. The third post in that thread says "This will legit not be able to continue unless you support, I am running out of my resources", which is a reminder that all of this runs on somebody's token budget.
### Who @dysmemic is
The other post the site owner flagged, "κ = 4.135962 × 10⁻⁴. The race to κ = 1 continues. We must have linear integer multiplication. We will have it!", is from @dysmemic, and the screenshot attached is the tracker. The account is eumemic's: its earlier posts link eumemic's PRs #5 and #13, and on October 8 it linked `github.com/eumemic/exoplanets`. The number is the title of eumemic's #137, "Padded triple covers with sequential dirty reuse", opened at 03:18 UTC, six minutes before the post. So it lives on CrocSwap as an ordinary PR, not on a fork. It led the open claims for six minutes. geckods' #138, a tightening of the same slack constants, came three seconds after the post. eumemic's later #168 was the best open claim through the morning, and the 10:41 leader, chafreaky's #179, refines its operation frames by 0.0047%.
### The tracker now
Prosz's tracker changed its headline overnight. It used to lead with the strongest claimed κ, a draft PR at the time; it now leads with the maintainer-reviewed κ, read from `main`'s `certificates/selected-result.json`, and shows the open claims separately. He also added a zoom, after people complained that everything since $2^{-15}$ was a flat line at the top of the chart. When I rendered it at 11:00 UTC (data "Updated 9 Oct, 10:36 UTC") it showed 178 PRs, 27 of them drafts, from 39 contributors, and 49 public forks.
Sort its PR list by highest κ, though, and the top entry is #159 at $\kappa = 0.127865$, which is not a claim anyone should read. It was closed eleven minutes after it was opened, and its whole body is one sentence about "multivariate Hamiltonians" followed by "Kind reminder, AI is not to be blindly trusted." The tracker parses titles, which is the right design for a dashboard and the reason not to take its sort order at face value. I left #159 off my chart.
### And the Fourier problem again
Shea's draft hasn't moved: his only commits since are a README rewrite at 23:52 UTC that adds "This is not a faster FFT" and a who-did-what list. Its δ is still $7.3\times10^{-5}$ from Jain's round-six network.
eumemic, though, did the obvious next thing. At 08:54 UTC @dysmemic posted that the community's networks "also apply to OpenAI's new exact DFT algorithm, raising its saving from 2.1·10⁻¹³ to 4.856·10⁻⁴ in the log exponent. About a 2 billion-fold improvement!", linking [eumemic/exact-dft-bounds](https://github.com/eumemic/exact-dft-bounds). Forty minutes later the same account added, "Looks like @ryaneshea already had this idea", and the README now says Shea's repository "has priority for the idea of the transfer". The new repository takes #144's complex network as merged on `main` ($m = 72$, $W = 29{,}937$, deficit 1,936) and adds a batched recursion for the Fourier setting (its Theorem 3.1), so that a frame change of residual dimension $\rho$ becomes one recursive call to $C^{\otimes\rho f}$. The result is $\theta = 1 - 607/1250000$, so $\delta = 4.856\times10^{-4}$. Against OpenAI's actual critical saving of $2.106\times10^{-13}$ that is 2.3 billion times, and against Shea's $7.3\times10^{-5}$ it is 6.65 times.
The README is careful about scope: it is "a paper proof with an exact finite certificate, and it is not formally verified", the network's "exact complex phases, the Cayley cover and completed-core sharing are written proofs", and "The constants are astronomically large: the network has about 2^2570 roles, and the recursion's base range is of order 10⁹ bits." Its `make verify` is one exact-rational moment check plus source pins. I ran it (0.34 s; "margin 3.158222e-03 … PASS" out of $W = 29{,}937$), and my 80-digit critical root for the same histogram is $4.85657\times10^{-4}$, so $607/1250000$ leaves about 0.012% headroom. Nobody has reviewed the Fourier transfer itself yet, the batched recursion least of all.
### Where the ceiling stood at 11:00
At 11:00 UTC on October 9: `main` is at $4609169/10^{10} \approx 2^{-11.08}$, reviewed by its maintainer with Codex; the best open claim is chafreaky's #179 at $655920176686219/10^{18} \approx 2^{-10.57}$, 42% above `main`; and Jain is at $6153378/10^{10} \approx 2^{-10.67}$, with its arithmetic in Lean. Everything is still conditional on the OpenAI manuscript.
None of this has touched the ceiling. #144's assembly still has a margin equal to $\epsilon$ and another equal to $1 - \epsilon(1+c)$, so κ cannot pass ½ whatever the networks do. What bounds the current witnesses is much lower than that, and it is the chain above: κ is $\epsilon q$, about $a/(1+2a)$, so 0.09% under the weaker network's saving; and that saving is about $D/(Wm)$ divided by $L$. To get to $10^{-2}$ you need networks that waste a couple of percent of their rank at a similar log spread. One draft PR (mikey1201's #172, a "zeta family" on the Boolean lattice) projects κ up to $2.88\times10^{-2}$ at $h = 3$ through the in-tree assembly, but it is labelled a research record with named open obligations, not a claim, and I haven't checked it.
In runtime terms, $\kappa = 4.609169\times10^{-4}$ takes 0.26% off $n\log n$ at $n = 2^{256}$ and 2.0% when $\log_2 n = 2^{64}$, and halving the time needs $\log_2 n = 2^{2170}$. The calculator further down has it as a preset.
### 9 October, afternoon: reusing dead registers
The site owner's second note of the day was "there have been more updates to Jain, check his tweets". There were two. At 12:50 UTC Jain asked for an arXiv endorser in cs.CC (Computational Complexity) for a paper that "collects proofs from our work on OpenAI problem #109", and at 13:07 he posted round eleven, $\kappa = 6685733/10^{10} \approx 6.6857\times10^{-4} = 2^{-10.547}$. He ended that post differently from the other ten: "We seem to be approaching a wall. Gains now come in fractions of a percent. The next real jump needs a big breakthrough which we are unable to find yet." By the evening the CrocSwap side was saying the same thing, with proofs.
Rounds ten and eleven do one kind of thing, most visibly on the complex side: they stop paying for registers the network has finished with.
Round ten's version is birth reuse, jamesyc's idea from #124 with eumemic's late compensation reads from #143, which Jain re-implemented as an exact maximum-weight matching. A gauged slot is a role whose register has to be read at a known frame, and in the plain ledger it pays a tail child to get its register into place. Born on the register of a slot that has just died, it doesn't: the donor's last climb and the newborn's first are charged as one ascending chain. On round ten's complex word all 3,180 gauges found a donor. With the rest of that stack, at $p = 11$ (so $m = 66$ and deficit $D = 1{,}320$ per vertex), the word came to 14,592 roles and a certified $a_c = 61609676/10^{11} \approx 6.1610\times10^{-4}$.
Round eleven's new piece is the one his thread describes as "two retired copies of the same value are subtracted in place, and a fresh slot is born on the zeroed register with no read". His word has 428 pairs of registers that each finish holding the same value and are never touched again. For such a pair, one gate $P \mathrel{-}= Q$, run at a frame that contains both copies' last frames, leaves $P$ with no signal at all, only dirty scratch. A fresh role is then born on $P$ with no read. Its starting content is a linear function of the initial dirty values, and the frame-zero compensation of every ungauged role is recomputed over the new transcript to cancel it. The kernel is from icekylinx's #184 ("zero-fresh recycling through a containing frame"), a PR that was closed five minutes after it was opened. Jain's own part is applying it to this word, choosing 386 hosts by another exact matching, and writing the ledger and the checker.
The ledger shows why it helps and why by so little. Each host removes one role, so $W$ falls from 13,308 to 12,922, and the deficit stays at 1,320, because the children removed and the children added differ in total width by exactly $h$ per stage. In the commonest shape (copies last at dimensions 2 and 3, mixed at dimension 4, $h = 22$) the removed children have widths 20, 4 and 19 and the added ones 2, 1 and 18. In the first-order picture from [the ninefold night](#why-smaller-is-worth-nine-times), $D/(Wm)$ rises 3.0%, but the new children are narrow, which costs about 1.5% in $L$, and the net is the 1.43% his gate reports: $a_c$ goes from $662034051/10^{12}$ to $67147467/10^{11}$. From the round-eleven histogram I get $D/(Wm) = 1.548\times10^{-3}$ and $L = 2.303$, so $a \approx 6.721\times10^{-4}$ against the certified $6.7147\times10^{-4}$.
Most of round eleven's 9% over round ten comes from elsewhere. The complex side switched to eumemic's #168 v4 modules (used as data), a re-chosen cube-local circuit Jain calls C1, which he puts at about 1.3% better than #168's, and 42 deleted terminal outputs. The bit side got a new cyclic all-but-one module, 1,762 birth-reuse pairs and 24 terminal sinks. Its $W$ went from about 21,491 to 19,017 with the deficit fixed at 1,936, so $D/(Wm)$ rose 13% and $L$ 6.7%, and the coarse saving went from $6.330\times10^{-4}$ to $6.701\times10^{-4}$.
### Which side binds, and which margin
The two rounds bind on opposite sides. In round ten the complex network was the weaker one, $6.1610\times10^{-4}$ against a stopped bit saving of $6.3239\times10^{-4}$. Round eleven's complex side jumped past the bit side, so now the bit side binds: $a^* = 134020033/(2\cdot10^{11})$, stopped at $\theta = 1/1000$ to $a_{\rm bit} = 133893704947/(2\cdot10^{14}) \approx 6.6947\times10^{-4}$. Inside #144's assembly nothing moved. The binding margin is still the compact phase layer $\epsilon q$, with the original-prefix margin exactly $\eta = 10^{-8}$ above it. I redid the assembly with exact fractions: $\epsilon \approx 0.998663$, $\epsilon q \approx 6.68573327\times10^{-4}$, and the floor on the $10^{-10}$ grid is $6685733/10^{10}$, $2.7\times10^{-11}$ under the margin. Its $\log_2$ is $-10.5466$, so "2⁻¹⁰·⁵⁴⁷" is right, and "1.09 fold" is 1.0865. The same arithmetic reproduces round ten's $6153378/10^{10}$, bound by the complex side.
`make round11` passed in under a second. It reproduces #144's κ, round nine's and round ten's before certifying round eleven, then runs eight unit tests. I also ran the two new heavy gates by hand. The complex recycling checker on the $p = 11$ word ended `ALL PASS` after 44 s at 0.96 GB, with its controls rejected, among them a skipped mix and a flipped mix sign. The JSON-only bit ledger replayed the $p = 12$ word forward and reflected, matched the release inventory, and ended `ALL PASS` after 2 min 18 s at 0.98 GB. `lean/Round11.lean` is there. His `gen.py` regenerates it byte for byte from the histogram file, it has no `sorry`, `axiom` or `native_decide`, and it covers both moments, the stopped mix, the finite bridge and the 47-row assembly. There is still no Lean toolchain on this machine, so I didn't run it. The README is plain about what is new and "stated here but not machine-checked": the recycling rests on #184's argument and "There is no literal frame replay of the mix gate at any size"; the terminal deletions and sinks rest on jamesyc's lemma from #166; and the C1 circuit, the per-operation descent and the recycling are checked at full size only by an F2 recount and a scalar replay.
### Nineteen minutes in front
When round eleven went up, at 13:07:49 UTC, it was the highest claim anywhere. `main` had moved ten minutes earlier (below) to $6.6189\times10^{-4}$, and the best open PR was ikeboy's #194 at $6.6743\times10^{-4}$, so Jain was 1.0% above `main` and 0.17% above the best open claim. It held until 13:27:16, when evmckinney9's #197 claimed $6.7608\times10^{-4}$ by packing the bit residuals of rohanarun's #187 into shared banks. That reads #197 by its title as it stands tonight, and PRs here get edited in place, though its body's own comparison with #194 agrees with the title. By 18:50 Jain was 2.4% behind the open frontier.
### The arXiv paper
The request itself is two lines. arXiv asks people submitting to a category for the first time to be endorsed by someone who already has, so this is the normal gate for a new author. I went looking for the paper. It isn't in his repository at the newest commit (`1a580dc`, 13:01 UTC), which holds the ten TeX notes and their PDFs, the certificates, checkers and Lean files, but nothing that reads as one paper, and none of his other public repositories has it. So I can't tell you what it proves.
What I can say is what a paper collecting these proofs has to stand on, because the README sets it out round by round. It assumes OpenAI's manuscript, which has no Lean statement and no published review. Since round nine it runs inside icekylinx's cover assembly from #144 and inherits that assembly's interfaces (the three-stage cover lifting, the exterior rule, the stopped recurrence), and on top of those sits his own list of stated-but-unchecked interfaces, which grew again in round eleven. What Lean checks is the arithmetic: the moments, the stopped mix, the finite bridge and the assembly. A single written paper would at least give the proofs one place to be refereed, and refereeing is what this race has had least of.
### `main` jumps again, by import
At 12:57:37 UTC Colkitt pushed a morning's worth of reviewed commits to `main` in one go, taking it from $4.609169\times10^{-4}$ to $330942774629799/(5\cdot10^{17}) \approx 6.61886\times10^{-4}$, about $2^{-10.561}$ and 43.6% higher. The witness is Dugongue's #186. It takes chafreaky's shared-edge complex supplier from #181 and eumemic's #168 v4 bit word and adds a compensated rematch, eleven coordinated frame cuts, and "completed entrance banks", which on the bit side pack 2,200 rank-20 entrance gauges into shared scratch banks. Here the complex branch binds, at $6.6232\times10^{-4}$ against an effective bit saving of about $6.6506\times10^{-4}$. The push also carried an overnight candidate that never reached `main` on its own, the recycled-bit composition of #147, #150 and #151 at $4.7215\times10^{-4}$. GitHub marks those three, and two verification PRs, #64 and #101, merged at 12:57:38. #186 itself came in as a pinned package and is still open on GitHub.
The review, `community-round8-review.md`, is again Colkitt "with OpenAI Codex assistance", and this time it found a gap and filled it. #186's identity "does not alone specify an efficient conflict-free schedule for the physical invocations", so the maintainer wrote a bank-scheduling supplement, checked exhaustively in a small model over $GL_2(\mathbb Z/9)$ (3,888 invocations, 15,552 endpoint basis columns), and says plainly that it "is not a new Lean-checked theorem". It also explains why Rohan's #185 can't be stacked on top: #186 is limited by its complex side, so better bit leaves change nothing. I ran `make entrance-bank-verify` on `main` at `3b6b668`. It passed, "complete complex and bit words, 47 constraints, seven margins, and 21 assembly controls", plus the scheduling diagnostic, in two minutes at 1 GB. As far as the fxtwitter mirror shows, Colkitt hasn't posted about this one.
### The race starts proving its own ceilings
The PRs I found most interesting this afternoon claim no κ at all. DaysSky's #192 proves that no choice of operation frames for #168's complex word can certify more than $\kappa < 7.010\times10^{-4}$. It starts from "Every certified saving satisfies `a < D/C`", where $C = \sum n_r\, r \ln(m/r)$. That is the first-order formula from last night turned into an inequality ($C = WmL$), and it follows from $e^x \ge 1+x$. Then it bounds $C$ from below over every frame layout. Two hours later DaysSky's #201 bounded the whole design: no word on #144's paired-cube design can certify more than $\kappa < 2.5592\times10^{-3}$, about 3.9 times `main`, whatever its circuit, gauges, reuse or frames. It leaves out the newer source-assisted words, which break its rules. chafreaky's #212 carried #192's method to the bit word that #200 to #207 share and got $\kappa < 6.959\times10^{-4}$ there, so frame tuning on that word has at most 1.8% left. huxint's draft #203 argues that several current lines can't reach $\kappa = 0.01$ even with every auxiliary call free.
None of these has been reviewed, and I haven't checked them. They are still the right work at this stage, and they put a number on Jain's wall: the word families being tuned are within a few percent of their own ceilings, and the paired-cube design as a whole has at most a factor of about four left.
### Where it stands at 18:50 UTC
`main` is at $6.61886\times10^{-4} \approx 2^{-10.561}$, reviewed by its maintainer with Codex. The best open claim is hcg890's #217 at $6.84697\times10^{-4} \approx 2^{-10.512}$, opened at 18:38 and 3.4% above `main`. There are 217 PRs from 47 accounts; GitHub marks 35 merged, 158 open and 24 closed. Jain is at $6.6857\times10^{-4} \approx 2^{-10.547}$, with the arithmetic in Lean. All of it is still conditional on the OpenAI manuscript. In runtime terms `main` now takes 0.37% off $n\log n$ at $n = 2^{256}$ and 2.9% at $\log_2 n = 2^{64}$, and halving the time needs $\log_2 n = 2^{1511}$.
On the Fourier side, eumemic's `exact-dft-bounds` hasn't changed since its 09:34 credit to Shea. Shea's draft has. At 15:24 UTC he reorganised it as a running record of transfers and selected Jain's round-eleven complex network, which takes $\delta$ to $0.00067$ from the tensor saving $67147467/10^{11}$, 1.38 times eumemic's $4.856\times10^{-4}$. His audit file calls it "same-assistant mathematical review and finite checks; independent review and end-to-end formal verification remain pending". I ran its `verify_round11.py`, which checks all 17,057,040 scalar output coefficients of the word, and it passed in four minutes.
Prosz's tracker has more rows but no new layout, so I haven't re-shot it.
## 10 October: the Fourier side gets a proof
The site owner's note this morning was a link to Ryan Shea's post from 03:31 UTC:
> We're publishing another update to OpenAI Problem #130: computing the exact discrete Fourier transform below n log n. Our paper now incorporates the all-length bound: T(n) = O(n(log n)^(1−δ)), with δ = 0.0007547360. That's 10.34 times the exponent saving of δ = 7.3×10⁻⁵ in our previous announcement. The stronger Fourier result comes from Jacob Sussman's five-stage framework and Chafik Boukhalfa's improved construction, building on @OpenAI, @0xdoug, @RohanArun, @SJ_Swapnil_Jain and the wider community.
I expected another written transfer, like the two before it. It isn't one. The number in that post is a Lean theorem, proved on OpenAI's own statement of the problem, and it is the first result in this whole story that doesn't rest on an unrefereed manuscript. It also isn't Shea's, and his repository says so in plain words.
### What Shea's update is
The arithmetic in the post is right. With exact fractions, $7547360/10^{10}$ over $73/10^6$ is 10.3388, so "10.34 times" holds. But that compares against his first announcement, on 8 October. In between, at 15:24 UTC on 9 October, he had already moved to $\delta = 0.00067$ with Jain's round-eleven network ([above](#where-it-stands-at-1850-utc)), and against that the new number is 12.65% more, which is the figure his README gives. The post skips its own middle step. Against eumemic's written $4.856\times10^{-4}$ the new $\delta$ is 1.55 times, against OpenAI's headline $10^{-13}$ it is 7.5 billion times, and against OpenAI's actual critical saving of $2.106\times10^{-13}$ about 3.6 billion.
The README's own description is more modest than the post: "The current update incorporates and reproduces an already published upstream formal Fourier theorem; it does not claim a new exponent of our own." What Shea's side did between 01:58 and 04:53 UTC was add his name as author, drop the word "draft", import the upstream proof and its certificate package pinned by hash, rebuild the Lean project from those sources (2,625 seconds of compilation on a 32 GB machine, by the receipt), and write a long attribution audit. The audit is careful. It corrects one of the imported notices, which credits Rohan Arun with PR #209 when #209 is Jacob Sussman's.
So is there a paper on arXiv now? No. The "paper" is `manuscript.pdf` in the repository, generated from Markdown by a build script. A search of arXiv finds nothing from Shea, Sussman or Jain on any of this. What the manuscript assumes splits in two. Its headline is the imported Lean theorem, which assumes only Lean's kernel, Mathlib and OpenAI's statement of the model. Its own contribution is a "reusable conditional transfer" written in prose: any finite network meeting a stated contract gives a Fourier bound through OpenAI's compiler and all-length reduction. That part is still a written argument, and the abstract keeps the two apart: "The reusable written transfer argument in this paper is distinct from that imported formal proof."
### Sussman's five stages
Jacob Sussman (jacobalansussman on GitHub) appeared in the race only once on CrocSwap, with PR #209 at 16:56 UTC on 9 October, which changes one line of CONTRIBUTORS.md to list his repository, [wht-power-saving-lean](https://github.com/jacobalansussman/wht-power-saving-lean), and claims no κ. The repository did the rest in one day. At 13:13 UTC it published a Lean proof of a Walsh–Hadamard bound with saving $5.399225\times10^{-4}$, on his own circuit in a new five-stage layout. At 16:21 it put the community's best complex circuit, ikeboy's #193, into his layout and proved $7.474547\times10^{-4}$. At 19:00 it carried that over to the discrete Fourier transform of every length, at $7.474546\times10^{-4}$. His README says how: "I built this on 8 and 9 October 2026 with a team of AI agents (Claude) that I directed … The agents wrote the Lean proofs, designed the network, found the certificate by computer search, and audited one another's work."
The new idea is the layout. In the three-stage cover from [the ninefold night](#three-shears-instead-of-one-swap), a bank pair trades places through three shears, one invocation each, so two pairs cost six. Sussman gives every pair a twin, a second pair whose labels coincide with its own at three moments. Additions between arrays with equal labels are free, so twins can hand values across at those moments, and two pairs get exchanged with five invocations instead of six. The cover gets wider ($m = 5h$ coordinates instead of about $3h$) and the role count becomes $W = 4v + R$. On the same helper circuit, #193's, the saving goes from $7.009184\times10^{-4}$ in three stages to $7.474547\times10^{-4}$ in five, 6.6% more for changing nothing but the schedule. His second result's network has $h = 22$, $v = 1{,}320$ pairs, $R = 9{,}412$ helpers, so $W = 14{,}692$ and $m = 110$.
Chafik Boukhalfa is chafreaky on GitHub, whose name has been on PRs since #46. His #233, at 22:01 UTC on 9 October, kept #193's word and changed only which dead registers are handed to which new ones, using #200's maximum-weight pairing, for a complex saving of $3549537/(5\cdot10^9) \approx 7.0991\times10^{-4}$ in three stages. Then #256, at 01:35 on 10 October, wrote that word out as one explicit numbered program in Sussman's certificate format: 12,052 registers, 18,248 frames and 52,300 gates, accepted by Sussman's own checker. In the five-stage layout it gives $7547361/10^{10}$. Nine minutes later he opened [PR #2 on Sussman's repository](https://github.com/jacobalansussman/wht-power-saving-lean/pull/2) with the Lean proofs: the Walsh–Hadamard transform at $1 - 7547361/10^{10}$ and the Fourier transform of every length at $1 - 7547360/10^{10}$. A commit at 02:29 reports that OpenAI's official comparator accepts both (18.5 and 16.5 minutes on Linux). That PR was still open, unmerged, when I last looked at 11:35.
### What the Lean proof covers
First a correction. Earlier in this article I wrote that the explicit $\delta = 10^{-13}$ algorithm for family 130 was manuscript-only, with only a weaker, subsequential statement in Lean. That was true of OpenAI's first release. At 05:20 UTC on 8 October OpenAI pushed an update (commit `fd4aeeb`, "Update manuscripts and Lean formalizations") that added `ComparatorChallenges/UniformFourier.lean`: fixed programs that compute the DFT and the convolution of every positive length, with work $O(n(\log n)^{1-10^{-13}})$, proved. I missed it.
Sussman's statement file is that file with three changes: a comment, the exponent, and a `Z` appended to seven names so they don't clash with OpenAI's originals, which the proof imports. OpenAI's 89 files under `OAI/` are byte-identical to theirs (Shea's audit compared all 89, plus the challenge file and the toolchain pin, with OpenAI's commit). The proof uses only the standard axioms, `propext`, `Classical.choice` and `Quot.sound`, and both Sussman's `Compare.lean` and OpenAI's comparator check that the statement and its definitions are the ones intended. So for once, "verified" means the whole theorem, not the arithmetic around it. That is the opposite of the #109 side, where Lean has only ever checked certificate arithmetic and the theorem is a manuscript.
What it does not cover is written down in Sussman's README, which is the best scope statement anyone in this race has produced. The model is OpenAI's: exact complex numbers, unit cost per operation, unrestricted coefficients, a root of unity supplied as input, integers bounded by a polynomial in $n$. So it says nothing about bit complexity or numerical computation, and it doesn't contradict Morgenstern's $n\log n$ lower bound, which needs bounded constants. The accounting is whole-block only, the community's whole-residual batching from #10, which Sussman proved in Lean for this model; in OpenAI's own per-rank recursion his best figure is still $3.155781\times10^{-4}$. And nobody has read the new Lean line by line: "The new proofs have not yet had a line-by-line human review, and I would welcome one."
It is also not an algorithm anyone will run. His README works it out: the proof indexes a table of $2^{a}$ arrays by the orthogonal group of a 110-dimensional binary space, with $a = 6049$; below $\log_2 n = 665{,}500$ the program is just the ordinary $n\log n$ algorithm; and with that constant the bound only drops below $n\log_2 n$ once $\log_2\log_2 n$ passes about $8\times10^6$. Ignoring constants, a 1% saving at $\delta = 7.547\times10^{-4}$ needs $\ln n \approx 6\times10^5$.
I couldn't run Lean: there's no toolchain here, and a Mathlib build of this size isn't something to do on a shared machine. I ran the finite side instead. Shea's `verify_five_stage.py`, which runs Sussman's reference checker on the explicit #256 program and rebuilds its histogram, passed in 25 seconds. From that histogram (358,975 children, rank 1,613,040 against $Wm = 1{,}616{,}120$, deficit 3,080) I get a critical saving of $7.5473642\times10^{-4}$ at 80 digits, so the certified $7547361/10^{10}$ leaves $3.2\times10^{-10}$. The moment at that saving is $1 - 8.06\times10^{-10}$, and at $7547365/10^{10}$ it fails by $2.05\times10^{-10}$, both matching the certificate's own bounds. The first-order picture from the ninefold night gives $D/(Wm) = 1.906\times10^{-3}$ and $L = 2.522$, so $a \approx 7.556\times10^{-4}$. Compared with round eleven's complex side, the five-stage word wastes 23% more of its rank and pays 10% more in log spread.
Two other entries belong next to it. danadran01's [exact-dft-power-saving](https://github.com/danadran01/exact-dft-power-saving), committed at 03:48 UTC on 9 October, was by Sussman's account the first outside Lean proof of OpenAI's uniform statement, at $3.2\times10^{-6}$; I haven't built it. And Boukhalfa's #225 on CrocSwap, at 20:06 the same day, wrote out a transfer at $7.009\times10^{-4}$ that the Lean proof has now passed. eumemic's `exact-dft-bounds` hasn't changed.
### Five stages reach κ
The layout crossed straight back. At 22:05 UTC on 9 October hcg890's #234 combined "Sussman's published five-stage complex baseline" with a new bit construction, also five-stage, and claimed $\kappa = 7.037017\times10^{-4}$, 2.8% above #217, the best open claim at my last look. The bit side binds there, at $m = 120$ against the complex side's 110.
The night then went the way the others had: one structural step, then dozens of PRs gaining hundredths of a percent. eumemic's #210, Rohan Arun's #237 and #240 and Dugongue's #242 tuned the five-stage banks to about $7.10\times10^{-4}$ by midnight. At 04:17 DreamingOfClouds' #276 brought a new paired-cube bit word (gen4) at $7.2287\times10^{-4}$; the same account's #285 at 07:10 annealed its modules against the exact compile cost (gen5) for $7.4279\times10^{-4}$, 2.27% more, "a second structural change to the bit word"; chafreaky's #296 at 09:47 (gen5b) and #299 took it to $7.4990\times10^{-4}$, and Rohan's #300 at 10:40 to $7.49918\times10^{-4}$. The best open claim when I stopped is lydakis's #275, which opened at 04:17 at $7.1369\times10^{-4}$ and has been renamed five times since, most recently at 11:31, to $7.51075693834608\times10^{-4} \approx 2^{-10.379}$. That is 13.5% above `main`. There are now 301 PRs from 61 accounts: GitHub marks 35 merged, 232 open and 34 closed.
### The merge that GitHub doesn't show
At 04:13 UTC Colkitt posted:
> Verified and merged PR 210 from @dysmemic which pushes the frontier to κ ≈ 2⁻¹⁰·⁴⁶ (tightened from κ = 2⁻¹⁸²) … Since our last update, the community has increased our conditional integer-multiplication exponent saving by 54%, from κ ≈ 0.000460917 to κ ≈ 0.000710046.
The numbers check: $\log_2(7.10046\times10^{-4}) = -10.4598$, and 0.000710046 over 0.000460917 is 1.5405. "Since our last update" means his last post, the #144 merge; he never posted about #186, and over #186's $6.61886\times10^{-4}$ the gain is 7.3%. The value matches the "preceding source527 result" in #210's description, $7.10046193349537\times10^{-4}$, before #210 was retitled to $7.10069\times10^{-4}$.
But at 11:35 UTC GitHub disagrees. `main` is still `3b6b668`, pushed at 13:23 UTC on 9 October, its `certificates/selected-result.json` still says $330942774629799/(5\cdot10^{17})$, #210 is open, there is no new integration or review branch, and the tracker's own GitHub read reports the repository last pushed at 13:28 UTC on 9 October. The post says the results are "published here" and links the repository. I can't tell whether the push failed or is still queued behind his agents (he said last night they were "backed up"). Until it lands, the reviewed number on the record is #186's, and I've left the red line on the chart where GitHub has it. (It landed at 14:08 UTC, as a historical checkpoint already overtaken; see [11 October](#main-moved-once).)
### Walls, in writing
Jain hasn't posted a twelfth round. What he did post, at 15:51 UTC on 9 October, and what I missed at 18:50, is two papers on Zenodo: "Ceilings and Impossibility Results for Finite Witnesses of Sub-n log n Integer Multiplication" (59 pages) and "No-Go Results for Finite Witnesses of Sub-n log n Integer Multiplication". Neither is the paper he wanted an arXiv endorser for, and nothing of his has appeared on arXiv. He describes them as "where the walls are: proven ceilings and no-go theorems for the finite-witness approach to beating n log n."
The ceilings paper puts #144's complex word at no more than $2^{-10} \approx 9.77\times10^{-4}$ for anything that keeps its visits. The no-go paper has one line I found striking in hindsight: banks swapped by layers of bank-to-bank additions need at least three layers, three layers need total rank at least $3n$ for $n$ port pairs, "a computation shows that five layers can do with less." Sussman had published his five-stage layout two and a half hours earlier, and put the community's circuit into it thirty minutes after Jain's post. Neither cites the other, and I see no sign either had read the other; they found the same opening from opposite sides. Jain's repository also gained a "closed ideas" list overnight, every route he tried that doesn't raise κ, each with its model and status, and it says of both papers: "Both papers were checked by adversarial review passes of our own. That is not independent peer review." He has since moved to a different problem, deterministic 3SUM.
The CrocSwap side kept proving limits too. eumemic's #288 reports a search for a new mechanism that could "at least double the saving exponent on both the bit and the complex side. We found none." Every frontier witness, it says, gets its saving from the same index-coding mechanism, "with a per-level coding gain of about 1.003 at m ≈ 110–120", and it proves among other things that $m = 2$ never saves for any frames in $GL_4(\mathbb C)$. GamingPuzzled's #231 shows the paired-cube decoder can't extend to four-dimensional ports: in its family a decoder exists exactly when the port dimension is not a power of two, by Lucas' theorem. I haven't checked any of these.
What I take from the night is how these ceilings work. Each holds inside its stated model, and the jump came from outside every one of them. #201's bound of $2.5592\times10^{-3}$ was for #144's three-stage paired-cube design, and the five-stage words are a different design. The ceilings tell you which knob is exhausted. They don't tell you there are no more knobs.
### Where it stands at 11:35 UTC on 10 October
`main` on GitHub is still $6.61886\times10^{-4} \approx 2^{-10.561}$. Colkitt has announced $7.10046\times10^{-4}$ but not pushed it. The best open claim is #275 at $7.51076\times10^{-4} \approx 2^{-10.379}$, and Jain is still at round eleven's $6.6857\times10^{-4}$, now 11% behind it. All of it is still conditional on OpenAI's #109 manuscript. In runtime terms the best open claim takes 0.42% off $n\log n$ at $n = 2^{256}$ and 3.3% at $\log_2 n = 2^{64}$, and halving the time needs $\log_2 n = 2^{1331}$.
The Fourier problem is now ahead in both senses. Its saving, $\delta = 7547360/10^{10}$, is larger than any κ claim, and it is a Lean theorem on OpenAI's own statement, waiting for a human reader and for Sussman to merge it. The tracker has no new layout, so no new screenshot, and Julian Schiavo hasn't posted a fifth chart.
## 11 October: E8 takes the race past a thousandth
The site owner sent three posts from Rohan Arun, who has taken to calling the race like a sports commentator ("Math typically never updates this fast, so this really is like sports for nerds LOL"). The last one, at 17:31 UTC on 11 October, opens "From κ=2⁻¹⁸² to 0.001289283116" and reports a 69.13% gain in a day, a switch to "E8 geometry", the first PR past $\kappa = 10^{-3}$, and a set of statistics about the agents doing the work. When I left the race at 11:35 UTC on 10 October the best open claim was $7.51\times10^{-4}$. When I fetched everything again at 19:40 UTC on 11 October it was $1.2966\times10^{-3}$, or $1.3726\times10^{-3}$ if you count a claim that is under challenge, and there were 485 PRs.
I wanted three things from the day: what E8 actually is in this construction, whether the headline number survives exact arithmetic, and whether 261,895,084 lines of code can possibly mean what it sounds like.
### The wall changes sides
Rohan's word "wall" is a good one, and it is easy to say what it means. Since the three-stage cover, the final assembly takes the smaller of two network savings, the bit side's $c$ and the complex side's $b$, so only the weaker network matters. All through the afternoon of 10 October the bit side was the weaker one, and PR after PR shaved the bit word.
Then two bit-side jumps put it ahead. At 14:13 UTC utcorvusvolat-dotcom's #310 brought a redesigned bit word (the "w3" word, whose "Design T" frees helpers without adding frames), and at 15:55 DreamingOfClouds' #315 moved the bit word to cube size $p = 10$. At 23:45 LJH-217's #346 put #310's Design T and its twin condensation on the $p = 10$ word, and the bit side jumped from $7.70277\times10^{-4}$ to $8.01726\times10^{-4}$. Now the complex supplier, at about $7.93\times10^{-4}$, was the wall, and #346 claims $396290046684973/(5\cdot10^{17}) \approx 7.92580\times10^{-4}$. Rohan's #350 retimed the complex program for $7.93469\times10^{-4}$ an hour later, which is the post of his that opens "BREAKING".
At 01:03 UTC DaysSky's #352 changed only the complex supplier, to a unit Jacob Sussman had published about two hours earlier, and the wall switched back. The new complex saving is $b = 876248285600677/10^{18} \approx 8.7625\times10^{-4}$, 9.3% above the bit side, so the bit side binds again, and $\kappa = 801083628465007/10^{18}$, 0.96% over #350. Both of Rohan's figures check out. That unit is what everyone means by E8.
### What "E8" is in this race
I expected "E8 geometry" to mean that someone had built the network's cover out of the E8 lattice. It doesn't. The cover is still Sussman's bridged five-stage word from [10 October](#sussmans-five-stages), indexed by the orthogonal group of the label space. What changed is the *code*: the set of labels the unit's data pairs carry.
Sussman's fourth result (commit `5ba3046`, 22:44 UTC on 10 October) describes it plainly. Take 9 points. The labels are all 84 three-element subsets and all 36 complements of pairs, 120 labels of width 9. Two pairs "know" each other, which means their exchange leaves an unwanted term in each other's targets, when their labels overlap in an even number of places, and each pair knows 56 others. In his note: "An agent recognised the first two as cases of one construction. They are called E8 and E7 here, after the root systems." The agents also counted the label maps that preserve the family: $348{,}364{,}800 = 960\cdot9!$, which is the order of the Weyl group of E8 divided by its centre.
The name is earned, and you can see why with a little linear algebra. Complement every label: the 84 triples become the 84 six-element sets and the 36 seven-element sets become the 36 pairs. Those are exactly the words of weight 2 or 6 inside the 8-dimensional space of even-weight words in $\mathbb F_2^9$, which is the E8 root lattice reduced mod 2: E8's 240 roots land on 120 nonzero vectors, one for each $\pm$ pair. Two odd labels overlap evenly exactly when their complements overlap oddly, which is when the two roots are not orthogonal. I checked the part that is easy to check by brute force. The "knows" graph on Sussman's 120 labels and the non-orthogonality graph on E8's 120 root lines are both strongly regular with parameters $(120, 56, 28, 24)$: every label knows 56 others, two that know each other share 28 acquaintances, and two that don't share 24.
The bit side took the same idea in a different field. Dylan Crow's #370 (utcorvusvolat-dotcom on GitHub), at 05:05 UTC, is "the first bit supplier that is not a paired-cube word". Its labels "are the 120 root lines of E8 in $\mathbb Q^8$, with the E8 form, which is positive definite, so every frame is nondegenerate". The bit network computes over the rationals and needs every frame's Gram matrix to be invertible, and a positive definite form gives that for free, which is a real reason to want this lattice rather than a decorative one. It has $h = 8$, $m = 40$, $v = 120$ and $R = 865$ helpers, and its bit saving, $9.0407\times10^{-4}$, put the wall back on the complex side at $\kappa = 8.75481\times10^{-4}$. So E8 fed the complex side first, at 01:03, and the bit side four hours later.
### Why a 120-label family pays
The trade-off from [the ninefold night](#why-smaller-is-worth-nine-times) explains the gain without any appeal to symmetry. A unit's saving is about $D/(Wm)$, the fraction of its rank it fails to spend, divided by $L$, the log-distance of a typical child from full width. In a five-stage unit with $v$ pairs, $R$ helpers and label width $h$, the counts are $m = 5h$, $W = 4v + R$, and $D = 4v - 5\,c_{\rm st}$, where $c_{\rm st}$ is the copy moves the scatter's totals cost: the four exchanges could save $4v$ moves and the copies take some of that back.
For E8 the scatter needs 8 totals at the full frame of 9, so $c_{\rm st} = 72$ and $D = 480 - 360 = 120$; with $R = 783$ helpers, $W = 1{,}263$ and $m = 45$. I ran Sussman's checkers on the committed certificate (`refcheck.py` accepted it in 0.2 s, `gxdry.py` printed the same whole-block figure, 8762479) and took the child histogram from the second. It gives $D/(Wm) = 2.111\times10^{-3}$ and $L = 2.407$, against $1.906\times10^{-3}$ and $2.522$ for Boukhalfa's five-stage word of 10 October. The waste fraction is 10.8% higher, the log penalty 4.6% lower, and at 80 digits the critical saving is $8.7624829\times10^{-4}$, so the certified $8762479/10^{10}$ leaves $3.9\times10^{-10}$. That is 16% over the old unit.
Where the higher waste fraction comes from is the interesting part. Boukhalfa's unit has 1,320 pairs on labels of width 22; E8 has 120 pairs on labels of width 9. Per pair it saves less ($D/v = 1$ against 2.33, because its copy ratio $c_{\rm st}/v$ is 0.6 rather than 0.33), but every array pays for $5h$ coordinates, and 45 is much less than 110. The agents' own yardstick, saving per bit of label width, is 0.0278 against 0.0265. Everything then rides on keeping the helpers few. After the scatter each target holds 56 unwanted half-terms, 6,720 in all, and the regular structure makes them shareable: Sussman's note finds that the 120 pairs split into 40 triangles in which a corner's 56 acquaintances are 27 shared with one partner, 27 with the other, and the two partners themselves. The final circuit uses 6.5 helpers per pair, where the old unit used 7.1.
One sentence in that note keeps this honest. The shared sums were finally chosen by an integer programme, which beat the symmetric greedy designs by 5.5%, and "the gain came from dropping the symmetry, not from using it." E8 supplied a dense, uniform pattern of interference on very narrow labels. A solver did the rest.
### Five stages become seven, nine and eleven
The next step was to change the word around the unit. DaysSky's #363 at 03:24 UTC generalised Sussman's bridged word to $K$ chained pairs in $2K+1$ invocations, with $m = (2K+1)h$, $W = 2Kv + R$ and $D = 2Kv - (2K+1)c_{\rm st}$. On E8 that makes $D/(Wm)$ equal to $2.111\times10^{-3}$, $2.281\times10^{-3}$ and $2.210\times10^{-3}$ for five, seven and nine stages, so seven is the sweet spot, and the certified savings follow: $8.762\times10^{-4}$, $9.328\times10^{-4}$ and $9.194\times10^{-4}$. The seven-stage E8 complex unit with #370's bit word gave DaysSky's #373 at $9.3191\times10^{-4}$ at 05:26.
From there the gains came, as Rohan put it, from "optimizing and canceling existing parts of the algorithm". Zhihao Chen's #377 compressed a nine-stage window to dimension 73 and lifted the complex side by 14%. Chen's #425 replaced the E8 complex circuit with "mutable-basis Hamming packets", 680 helpers instead of 783. icekylinx's #433 cut it to 560 with split centres. And DaysSky's #442 at 13:18 found the biggest single step of the afternoon, 9.2%, by noticing that a helper's leftover recursive work can be owned by a data array that is sitting idle in that window, on coordinates it would have to advance later anyway. The deficit stays at 2,496 on both sides while the live roles drop from 14,259 to 9,457 on the bit side, so $D/(Wm)$ goes up with nothing else paid. Eleven-stage hosting words followed within the hour.
### Past one in a thousand
Rohan's post says Dugongue posted the first PR past $10^{-3}$ "around 3 a.m.". That is Eastern time: #390 opened at 07:09:46 UTC, claiming
$$
\kappa = \frac{125027527861191}{1.25\cdot10^{17}} \approx 1.000220\times10^{-3} \approx 2^{-9.966},
$$
0.198% above chafreaky's #382. The complex side binds there, at $b = 40048866680681/(4\cdot10^{16})$, and the new pieces are on that side: 18 target-prefix rewrites and early helper restoration through mixed-width banks. Its own table is candid about what it isn't: "Our first composition merely reached the already-public #382 cap. That is not presented as a discovery." With the outer settings that Chen's #377 spells out ($\eta = 10^{-12}$, $\beta = 10^{-9}$, three bootstrap steps from $384599/10^{10}$), I get a binding margin $1.4\times10^{-18}$ above #390's claim, so it follows from its two network savings; I didn't run its pipeline, which needs a C++ build.
There is a footnote, and it shows how slippery "first" is here. applecrusher's #376 opened at 05:44 claiming $9.3191\times10^{-4}$ and has been retitled fifteen times since. GitHub's rename log has it at $1.0000547\times10^{-3}$ from 06:59:21, ten minutes before #390 opened. Its title now reads $1.12086\times10^{-3}$, for a package built on icekylinx's #433, which didn't exist until 11:54. So #390 is the first PR *opened* past $10^{-3}$, and #376 the first *title*. [The chart](#everyone-else-shows-up) near the top now runs to 19:40 UTC on 11 October, with a dotted line at $10^{-3}$, the crossing marked at #390, #376 plotted at its opening claim, and `main`'s one move of 10 October. Its hollow circles are explained in the next section.
### #467, and the claims it leaves out
Rohan's post ends "Just submitted a new best bound", linking his #467 from 17:17 UTC: a fresh shared-donor kernel and retiming on DreamingOfClouds' R715 bit word from #463, with four-replica whole-residual hosting and chafreaky's complex supplier from #458 unchanged. Its certificate gives a bit saving $c = 645473756513106438971/(5\cdot10^{23})$, a complex saving $b = 1297203880559577927527/10^{24}$, and $\kappa = 51571324647751/(4\cdot10^{16})$.
I redid the assembly with exact fractions, outside its code: the network saving is the smaller of the two less $10^{-24}$, six bootstrap steps from $384599/10^{10}$, $\eta = \beta = 10^{-18}$, and the largest point of the $10^{-18}$ grid strictly below $(1-\eta)q/(1+q)$. It comes out at exactly $51571324647751/(4\cdot10^{16})$, $8.3\times10^{-19}$ under the margin. At 80 digits, the critical saving of the bit histogram is $1.29094752\times10^{-3}$, so the certified $c$ leaves $2.9\times10^{-12}$, and the complex side's leaves $3.4\times10^{-12}$. The first-order picture gives $D/(Wm) = 1632/(4800\cdot78) = 4.359\times10^{-3}$ and $L = 3.369$: next to the five-stage word of 10 October, more than twice the waste fraction and a third more log penalty. I also ran its `verify_component.py`, which rebuilds the word from the pinned base, replays all 955 columns and re-certifies the bit saving. It passed in 27 s. The full verifier needs SymPy, a C++ build and Boost headers, which this machine doesn't have, so the physical-routing checks are the PR's word, plus its receipts.
Was it "a new best"? At 17:17 it was not the highest claim on the board. higherbeing's #448, at 14:51, claimed $1.33073\times10^{-3}$ with "cross-copy idle-axis hosting", which lets one helper residual straddle two copies of the same idle child. Fifteen minutes later chafreaky's #451 posted an exact obstruction: for the smallest case ($q = 3$, $m = 2$, one coordinate each), the operator the derivation needs has trace 90, while the split it proposes has trace 54, and the complex analogue gives $4+2i$ against $4+4i$. Rohan had built #453 on #448. He retitled it "Under review" and wrote at the top: "the claimed gain is not currently established." So when he posts #467 as a new best, he means the best claim that doesn't use that split, and on that reading he's right: it beat #465's $1.28348\times10^{-3}$ by 0.45%.
The split hasn't gone away. #450, #464, #469 and higherbeing's #480 all rest on it, and #480 at 18:40 claims $1.37255\times10^{-3}$, the highest number of the day, while saying in its own scope section that #451's note and the replies "describe the disagreement". Nobody has merged anything, so nobody has ruled. On the chart these five are hollow and kept out of the green line. Rohan's own dashboard, the image above, headlines #448 regardless.
When I stopped, the best claim that avoids the split was Dugongue's #483 at 19:33, $1.296606691\times10^{-3} \approx 2^{-9.591}$, 0.57% over #467. Against the best open claim 24 hours before Rohan's post (#320, $7.6955\times10^{-4}$ by its title now), #467 is 67.5% higher. I couldn't find the baseline that gives his 69.13%, but the size is right.
### Rohan's numbers, checked
"151 PRs opened in the last 24 hours" is close. In the 24 hours to his post I count 148 (#321 to #468); the count reaches 151 in the window ending about fifteen minutes later. "89 disclose Codex and 48 disclose Claude" depends on what counts as disclosure. In those 148 bodies, 94 mention Codex and 89 mention Claude, 48 mention both, and 13 mention neither, five of them Dugongue's. But most bodies carry long credit lists for everything they build on, so a mention often credits someone else's assistant rather than the author's own. His "most prolific" figures are all-time, not daily: at 17:31 GitHub had 73 PRs from eumemic and 62 from chafreaky, which match his 73 and 63, and 108 from rohanarun, not 118.
The lines are the striking one. I fetched all 148 PR heads and diffed each against its merge base with `main`. Together they add 224,246,908 lines and delete 280. Different window, same order of magnitude. Of those lines, 203,238,279 (90.6%) are in `.json` files, and 78 million are under directories named `parent…`. That is where it comes from. Each PR vendors the complete package of the PR it builds on, byte for byte, so that its verifier can run against pinned inputs, and that package contains its own parent's package, and so on. applecrusher's #376 alone adds 97,589,908 lines in 7,589 files. Its largest file is a 1,465,606-line pretty-printed JSON array called `HOSTED-SIGMA.json`, and the same file appears eight times at increasing depth, under paths like `…/parent-prefix60/parent-prefix56/parent-two-pair/parent-eleven95/parent-eleven80/parent-eleven79/…`. eumemic's frozen stack of PRs from #338 to #347 does the same thing in miniature: each PR re-adds the whole stack under it, and #347 adds 8.5 million lines. Git stores identical content once, so all 148 PR heads together, plus `main`'s history, came to 182 MB of packed objects in my clone. "261,895,084 lines of code added" is true as GitHub counts it and misleading as English: it is mostly the same machine-written data, copied into each other.
To be fair to the copying, it has a purpose. Pinned, byte-identical parents are how these packages make a reviewer's replay reproducible without trusting anyone's branch. It just means lines added is no measure of work done.
### A prompt for anyone's agents
The post ends with a recruiting pitch: "anyone can copy the prompt on this site to contribute with Codex or Claude". The site is [Auto Research by Supers](https://app.getsupers.com/auto-research), which Rohan calls "our new app" in a reply. It is a dashboard of the race (the image above is its chart), with a leaderboard of contributors by PRs, merges and "lead minutes", a tally of disclosed agents, and one long prompt to paste into Codex or Claude Code. The prompt is more careful than I expected. It tells the agent to read the reviewed record and the open PRs directly from GitHub rather than from a dashboard, to reproduce the strongest witness before trying to beat it, to "not add gains from different PRs without proving their compatibility", to certify with exact arithmetic, to record failures, to credit every reused construction and disclose AI assistance, and not to "merge your own result or claim maintainer acceptance". Rohan also published his failed ideas, which his post describes as sharing "GitHub files with failures to avoid repeating tokens": #368 archives a 100-item research campaign, 40 of whose items have no execution result, and #477 indexes 300 trials with their failures. That is good practice, and it is also how you get 148 PRs a day.
### `main` moved once
Did #210 land? Yes, but not the way Colkitt's post implied. At 14:08 UTC on 10 October GitHub marks DreamingOfClouds' #276 merged, and `main` now selects Rohan's #279 (the first four commits, on #276's new bit word) at
$$
\kappa = \frac{90446448750933}{1.25\cdot10^{17}} \approx 7.23572\times10^{-4} \approx 2^{-10.433},
$$
9.3% above #186. The same push imported eumemic's #210, the one Colkitt announced overnight, as a "historical source527 checkpoint" at $7.10046\times10^{-4}$. Its review says "This import fills the historical publication gap; it does not lower the current selected result." So the merge he announced did happen, ten hours later and already superseded. Both reviews are again Colkitt "with OpenAI Codex assistance", and the #279 review is careful in the usual way: it reports a coverage assertion that failed CI on the first head and the fix, a standalone checker that "imports no contributor code", and that #279 had meanwhile moved to a larger claim that is "outside this review".
Nothing has reached `main` since. Its last push is 14:13 UTC on 10 October, there is no new review branch, and Colkitt hasn't posted about either checkpoint; his latest post on the subject, on the afternoon of 10 October, recommends Nil Mamano's Big O Speedrun leaderboards. `main` hasn't crossed $10^{-3}$. The best open claim is now 79% above it, across 411 open PRs. GitHub marks 36 PRs merged and 38 closed, from 72 accounts.
### Fourier, the quasi-Riemann hypothesis, and Jain
Sussman's fourth result is a Lean theorem twice over. Besides the Walsh–Hadamard bound, `transform_mainE8` proves OpenAI's all-length Fourier statement with exponent $1 - 8762478/10^{10}$, on the same standard axioms, and both checkers pass. So for under seven hours, until #373 at 05:26, the Lean-checked Fourier saving of $8.762\times10^{-4}$ was ahead of every multiplication claim. Then the multiplication side passed it, and the Fourier side moved again. At 15:08 UTC Nil Mamano, who built the [Big O Speedrun](https://bigospeedrun.com/exact-dft) leaderboards, posted that his agents ("Opus 5.5 orchestrating 4 Codex and 3 Opus 5.5 … Gave them 9h") had found a new certificate by an integer programme over shared partial sums, on the race's bridged words, and proved $\delta = 2555269/(2.5\cdot10^{9}) \approx 1.0221\times10^{-3}$ in Lean, 16.6% over Sussman. The proof is a PR on the leaderboard's own verifier repository, which I didn't build, and the leaderboard lists it as "Lean Verified". It is also the first Lean-checked Fourier saving above $10^{-3}$. The multiplication claims are still larger, at about $1.3\times10^{-3}$, and still certificates on a manuscript.
Ryan Shea's paper still headlines $\delta = 0.000754736$. What he added on 11 October is a Lean proof of his earlier round-eleven saving, $\delta = 0.00067$, in Sussman's framework, so that his previous, written-only transfer now has a formal twin. eumemic's `exact-dft-bounds` hasn't changed since 9 October.
Rohan's post also notes that "researchers are tightening bounds across multiple other papers now, like the Quasi-Riemann hypothesis", naming @HaileyStormC. The quasi-Riemann paper is OpenAI's family 003, which [the roundup covers](/articles/openai-math#a-fixed-gap-next-to-the-line-re-s--1-family-003): no zero of $\zeta$ or of the Dirichlet L-functions with real part above $7/8$. The game there is the width $c$ of the zero-free strip next to the line $\Re s = 1$, which OpenAI has at $1/8$. Tim Gehrunger's tracker lists seven entries, each rebuilt and checked in Lean by the tracker, and the widest is Hailey Collet's of 10 October, $c = 0.12511335296$, so zero-free for real part above $0.87488664704$, a 0.09% wider strip (her earlier post on the problem says she works with GPT-6 Pro). It is the same pattern on a different problem: an unreleased model's conservative constant, an open leaderboard, agents tightening it a few hundredths of a percent at a time, and Lean checking each step.
Swapnil Jain hasn't posted a twelfth round on #109 and has moved on. Since 10 October he has posted a faster deterministic 3SUM, a power saving for Yates' 1937 algorithm for the GF(2) Möbius transform ("steal the explicit power-saving network built for the exact DFT, flip its characteristics"), and smaller AES S-box circuits. His repository's only new commit links a corrected second version of the ceilings paper.
### The ceilings, a day later
Yesterday I wrote that the ceilings tell you which knob is exhausted, not that there are no more knobs. E8 is a clean example. Jain's ceilings paper caps #144's complex word at $2^{-10} \approx 9.77\times10^{-4}$ for anything that keeps its visits, and the complex suppliers are now at $1.297\times10^{-3}$. That doesn't contradict him. They are different words, which is exactly the escape his hypotheses leave open. DaysSky's #201 bound of $2.5592\times10^{-3}$ was for the three-stage paired-cube design, and the frontier now runs on E8 units in seven- to eleven-stage bridged words, outside its rules, though the best claim is still at about half that number. eumemic's #288 searched for a mechanism that would "at least double the saving exponent on both the bit and the complex side" and found none. Thirty hours later the open claims were 1.73 times where they stood when I last looked, with E8, more stages and hosting doing most of it.
What still binds is the shape. The outer assembly now gives $\kappa \approx s/(1+s)$, where $s$ is the weaker network's saving. That is a different balance from #144's $a/(1+2a)$, and Chen's #377 calls $b/(1+b)$ "the limiting ceiling of this retained outer formula". The saving itself is still about $D/(Wm)$ over $L$, and both have now grown: #467's bit word wastes 0.44% of its rank, where #144's complex word wasted 0.09%. New format ceilings are already arriving for the new designs, among them DreamingOfClouds' #446 on window width, copy cost and label dimension and aliromman's #468, which proves that "nine calls are necessary for three abelian bit registers". I haven't checked them. And κ still can't pass ½.
### Where it stands at 19:40 UTC on 11 October
`main` is at $7.23572\times10^{-4} \approx 2^{-10.433}$, PR #279, reviewed by its maintainer with Codex, and unmoved since 14:13 UTC on 10 October. The best open claim that avoids the contested split is Dugongue's #483 at $1.296607\times10^{-3} \approx 2^{-9.591}$, and Rohan's #467, which I recomputed exactly, is at $1.289283\times10^{-3}$. The highest claim of all, higherbeing's #480 at $1.372550\times10^{-3}$, rests on the split #451 challenges. Jain is still at round eleven's $6.6857\times10^{-4}$. There are 485 PRs from 72 accounts, and all of them are conditional on OpenAI's #109 manuscript. In runtime terms #467 takes 0.71% off $n\log n$ at $n = 2^{256}$ and 5.6% at $\log_2 n = 2^{64}$, and halving the time needs $\log_2 n = 2^{776}$. For `main` the same figures are 0.40%, 3.2% and $2^{1382}$.
The Fourier problem's best Lean-checked saving is $1.0221\times10^{-3}$, below the multiplication claims for the first time since the evening of 9 October, and still the only one of the two that is a theorem. Prosz's tracker has the new numbers (485 PRs, 72 contributors, 82 forks) but the same layout, so no new screenshot. He has also started a [map of community efforts](https://math-frontier-atlas.netlify.app) across all of OpenAI's released problems.
## 13 October: `main` catches the frontier
The site owner sent Colkitt's post from 20:01 UTC on 12 October. It opens "New community checkpoint for integer multiplication", gives
$$
\kappa = 0.00161272137 \approx 2^{-9.276},\qquad T(n) = O\!\left(n(\log n)^{0.99838728}\right),
$$
says "Reviewed and merged to main", and quotes his own post of 10 October about #210, the merge that took GitHub ten hours to show ([above](#main-moved-once)).
Before anything else I checked the direction, because a post like this is easy to read backwards. The bound is $O(n(\log n)^{1-\kappa})$, so a bigger κ is a smaller power of $\log n$ and a better bound. $1 - 0.0016127213726553 = 0.9983872786\ldots$, which rounds to his 0.99838728, and $\log_2 0.0016127213726553 = -9.2763$. Both of his numbers are the same statement, and the repository's README at `1db1fbe` has the exact witness, $\kappa = 16127213726553/10^{16}$, "0.6517% above the previous selected saving". That is 2.23 times the $7.23572\times10^{-4}$ `main` had held since 10 October.
The surprise came when I cloned. By the time I read the post, `main` had moved four more times, and at 07:20 UTC on 13 October it stood at
$$
\kappa = \frac{1661634653669567}{10^{18}} \approx 1.661635\times10^{-3} \approx 2^{-9.233}.
$$
That is Jeremy Kothe's #799, and no PR title anywhere claims more. For four days this article tracked a gap between what contributors claimed and what the maintainer had reviewed. On the morning of 13 October that gap is zero.
### Eight moves in thirty-one hours
These are the selections on `main` since my last look, with GitHub's merge times. Each one is a review round in `docs/research/`, with the previous selection kept as a checkpoint under `certificates/checkpoints/`.
| merged (UTC) | selection | $\kappa$ | binds | what changed |
|---|---|---|---|---|
| 12 Oct 01:56 | icekylinx's #531 on #523, eumemic's #493 bit word | $1.517761\times10^{-3}$ | complex | direct delivery: 504 helpers instead of 560 |
| 13:11 | icekylinx's #535, DreamingOfClouds' #510 | $1.584931\times10^{-3}$ | bit | post-center rank-four kernels |
| 15:46 | Jeremy Kothe's #588, eumemic's #543 | $1.602279\times10^{-3}$ | complex | 43-PR round; #543's R420 bit word |
| 17:01 | Kothe's #596 on #591 and #592 | $1.612721\times10^{-3}$ | complex | hyperplane-ended helpers |
| 21:02 | Kothe's #617 | $1.635184\times10^{-3}$ | complex | cold helper endpoints, 28 per word |
| 22:37 | Chafik Boukhalfa's #626 on Kothe's #622 and #623 | $1.635527\times10^{-3}$ | complex | K-triangles: 492 helpers per call |
| 13 Oct 06:11 | Boukhalfa's #741, DreamingOfClouds' #589 bit word | $1.661203\times10^{-3}$ | complex | eight B-gate frames enlarged |
| 06:44 | Kothe's #799 on #741 | $1.661635\times10^{-3}$ | complex | four paired read exchanges |
The first row is mostly catch-up. When I stopped on 11 October the best open claim that avoided the contested split was Dugongue's #483 at $1.2966\times10^{-3}$, and the open PRs kept going overnight: icekylinx's #523 at 00:26 UTC with punctured Hamming ports, then #531 an hour later. Colkitt reviewed that chain and merged it at 01:56, 2.0976 times the old `main` by his review's count. It also settled the cross-copy dispute in the only way that matters for the record: `main` passed every claim built on the split, #480's $1.37255\times10^{-3}$ included, without using it. Nobody has ruled on #451's obstruction, and for the record nobody now has to.
From there the gains get smaller. #596 is 0.6517% over #588. The whole day of 12 October, from #531 to #626, is 7.8%. #741 is 1.57% over #626, and #799 is 0.026% over #741.
### Helpers that stop at a hyperplane
The idea in Colkitt's post belongs to Jeremy Kothe (jeremykothe2 on GitHub), who opened 37 of the 118 κ claims since #486, more than anyone else. His thread entry is short: "selected helper computations can finish in a smaller space, making the recursive work cheaper to pack." The review explains it.
In the network, a helper is a scratch register that borrows data, does its share of the exchange and is cleaned up afterwards. In #588's word every entering helper climbs all the way to the full frame before it finishes. #591 looks for helpers whose cleanup is a single gate whose two operands no earlier cleanup gate has touched. That gate commutes forward, so it can run right after part B, and then nothing later touches the helper. The span of its two operands' last frames, which the review calls U, has dimension eight, so the helper can stop there instead of at the full frame. Its residual width falls from eight to seven.
This is a different kind of gain from the ones earlier in the article. The deficit $D$ doesn't change; the review says so ("after residual accounting the deficit is unchanged"). What changes is the shape of the paid profile, the histogram of child widths that the moment prices, and with it how the children pack. A width-seven residual fits beside width-nine helpers in the rank-16 owner slots, so the same rank fills fewer, fuller slots. My reading, in the first-order picture from [the ninefold night](#why-smaller-is-worth-nine-times), is that $D/(Wm)$ holds still and $L$, the log-distance of a typical child from full width, comes down; the review itself only says the gain is "in the whole paid profile". #591 found 38 such helpers in each local provider, #592 changed the dispatch to 53 calls of one provider and 19 of another, and #596 improved the owner map. The bit side didn't move: eumemic's #543 word, 420 helpers, certifies $1.6538\times10^{-3}$, 2.4% above the complex side's $1.615326\times10^{-3}$, so the complex side is the wall.
Two Lean pieces came with it. Kothe's #593 proves the helper interface for an arbitrary terminal frame and the three exact Clifford identities the selected helpers use, and DreamingOfClouds' #599 extends the interface to multi-gate cleanup and cold helpers; the maintainer added a script that checks each selected helper's recipe against the literals in the Lean file. The review's limit is the usual one and I'll repeat it: "These proofs do **not** constitute a complete Lean multiplication theorem."
### After the post
The post went out at 20:01. #617 was merged an hour later, and it is Kothe again. #601 replaced the single-gate rule with a commutation test, which raised the count from 38 to 46 hyperplane-ended helpers per word. DreamingOfClouds' #604 let fourteen scratch helpers end at four-dimensional frames. #617 added a second cold endpoint to each factory block, 28 per word, for 1.3928% over #596. Kothe's #622 rebuilt the skeleton around four "K-triangles" that read unformed values directly instead of copying face packets, cutting each provider from 504 helpers to 492, and Boukhalfa's #626 repaired the owner map on top.
Overnight the complex word was rebuilt once more, on icekylinx's gauge44 factory (#645), and then Kothe and Boukhalfa traded PRs every twenty minutes or so until 05:49. The move that took `main` is Boukhalfa's #741, and it has the cleanest explanation of the whole day. It enlarges eight frames in the B section without touching a gate, an endpoint, a width or an owner. Each change turns a pair of paid children of ranks 2 and 3 into a pair of ranks 1 and 4. The rank mass is the same, $2+3 = 1+4$, so the deficit is the same, and the review's whole argument is one line: "Strict concavity of $r^{1-a}$ explains the improvement at unchanged rank mass." For a small positive $a$, a child of rank $r$ costs about $r^{1-a}$, a concave function, and spreading a fixed total over a more uneven pair costs less. At $a = 0.0017$, about the size of the savings here, I get $2^{1-a} + 3^{1-a} = 4.99205$ against $1 + 4^{1-a} = 4.99058$. Tiny per pair, and it is 1.57% on κ. Tekalign's #777 found the same eight frame changes independently, twenty-six minutes later, and the maintainer merged it as verification.
Then a ceiling went up and was stepped around within minutes, the same pattern as the ceilings of 9 and 10 October. Boukhalfa's #795, opened at 05:45, proves that #741's owner allocation is optimal for its fixed local profile. Kothe's #799, opened four minutes later, changes the profile. Its four paired read exchanges take the local histogram from `{1:2556, 2:334}` to `{1:2548, 2:338}` with every other width unchanged: eight rank-one children become four rank-two ones, the same rank in four fewer calls, with the physical owner bytes identical to #741. That is worth 0.026005%, which is where the race stood when I stopped.
### The wall, and the formula, again
The binding side changes twice on 12 October. At #531 the complex side binds, with the bit word 2.98% ahead. icekylinx's kernels in #535 lift the complex side to $1.58999\times10^{-3}$, just past DreamingOfClouds' bit word at $1.58745\times10^{-3}$, so for two and a half hours the bit side is the wall, by 0.16%. eumemic's #543 then moved the bit side well clear, and every selection since binds on the complex side. That explains where the work went. The later bit improvements, eumemic's #555 and DreamingOfClouds' #589, are titled "κ unchanged" or count only as headroom, and at #799 the bit side certifies $1.7334\times10^{-3}$ against the complex side's $1.664400\times10^{-3}$, 4.1% to spare. The [stop gauge](#where-kappa-comes-from) now has #535, #596 and #799 as its last three stops.
The outer assembly is still the one from 11 October. I recomputed $s/(1+s)$ with exact fractions from the binding savings in each `ASSEMBLY.json`, and the published κ sits under it by $9.4\times10^{-19}$ at #531, $7.4\times10^{-19}$ at #535, $3.5\times10^{-19}$ at #596 and $7.6\times10^{-20}$ at #799. So κ is still a hair under the weaker network's saving, and it still can't pass ½. DreamingOfClouds' #595 goes further and certifies a ceiling of $\kappa \le 5.1595\times10^{-3}$ for the current format, under its own accounting axioms. The review is careful to call it "not a universal ceiling on multiplication or every network interface". The frontier is at about a third of it.
### What the maintainer did
The thing that changed most since 11 October isn't κ. It is the review. When I last looked GitHub marked 36 PRs merged, and the open claims were 79% past `main`. At 07:20 UTC on 13 October it marks 152 merged, 116 of them since then, and the open claims are 0% past `main`. There are 843 PRs from 170 accounts, more than twice the 72 of two days ago, and 641 are still open, but most of the new ones claim no κ at all. Of the 358 since #485, 83 say so in the title: circuit savings on the older R420 word ("save 6,934 R420 XORs with a restored helper basis"), verification speed-ups ("6.47x faster mutation replay", "reduce reconciliation replay peak RSS by 56%"), independent replays, audits of old CI hashes, Lean proofs. The last round alone merged fourteen PRs of that kind beside #799.
How does one maintainer review 116 PRs in a day and a half? In rounds, with an agent, and with a written scope. Every round names the immutable heads it froze, rebuilds the selected construction cold, and records which checks are fresh, which reuse an earlier replay, and what was not checked. Contributor test reports are kept as contributor evidence and labelled that way. And the reviews carry the same disclosure: "Maintainer integration is by Douglas Colkitt with OpenAI Codex assistance." So it is the same kind of review as before, a careful replay of finite certificates on top of an unrefereed manuscript, done much faster. The all-size compilation, the tape implementation and the analytic transfer are still hypotheses, in the reviews' own words.
### Where it stands at 07:20 UTC on 13 October
`main` is at Kothe's #799, $1661634653669567/10^{18} \approx 1.661635\times10^{-3} \approx 2^{-9.233}$, and it is the best claim anywhere. From $2^{-182}$ that is a factor of about $2^{172.8}$ in under a week. Colkitt's posted checkpoint, #596 at $1.612721\times10^{-3}$, is now one of nine preserved checkpoints. In runtime terms #799 takes 0.92% off $n\log n$ at $n = 2^{256}$ and 7.1% at $\log_2 n = 2^{64}$, and halving the time needs $\log_2 n = 2^{602}$. It is all still conditional on OpenAI's #109 manuscript.
## 13 October (evening): auto-research gets a standard
Two more posts from Rohan Arun arrived after I stopped. The first, at 08:18 UTC, opens "From κ = 2⁻¹⁸² to 0.001661 :)" and reports a 28.9% gain since 11 October. The second, at 15:00, proposes a standard layout for any repository that wants agents working on it. I cloned the repository again at `bbc600d` (15:08 UTC) and fetched every PR head.
### Two more moves, both tiny
0.001661 is #799, where the last section stopped. Rohan's 28.9% is against his own [#467](#467-and-the-claims-it-leaves-out): I get $1.661635\times10^{-3}/1.289283\times10^{-3} = 1.2888$ (reasoned from the two certificates). Since his post, `main` has moved twice. At 13:30 UTC (commit time) it took Chafik Boukhalfa's #890, built on Jeremy Kothe's "U8" rewrite of eight local deliveries, at $\kappa = 1661957774647029/10^{18}$, 0.01945% over #799. At 15:06 it took Jonathan Capalbo's #942, the same word hosted across 37 replicas:
$$
\kappa = \frac{1661960547713947}{10^{18}} \approx 1.661961\times10^{-3} \approx 2^{-9.233}.
$$
The README calls that "a 0.00016686% increase in the exponent saving". My exact division agrees, and it is an increase, the right direction (measured). The same README says an all-replica bound "closes further improvement on the same `10^-18` grid" within the fixed design. The race is in the sixth significant figure now.
### The bit side's breakthrough is the one above
Rohan's "the bit side broke through" is [the wall](#the-wall-and-the-formula-again) seen from the other side. His +4.18% is eumemic's #543, a 420-helper E8 bit word whose README puts its nine-stage $c$ 4.18% above #510's (reported). DreamingOfClouds' #545, the "punctured star", removes nine of the 120 E8 lines as sources and targets, leaving 111 live ports, and turns each retained total into a rank-7 "pseudo-target", so the decoder constant falls from 72 to 63. eumemic's #555 put it on the R420 skeleton, and #589 tuned its frames to $c = 0.001733427039589844$. That word is unchanged in #890's and #942's assemblies. At #942 the complex side certifies $1.664727\times10^{-3}$, so the complex side binds and the bit side has 4.13% to spare (measured from `ASSEMBLY.json`). A merged note now proves #589 frame-optimal: "no reassignment of gate frames, however many gates change at once, raises the nine-stage bit saving".
### Leech, Turyn and BW16
DominikScholz's #598, merged, builds a 24-dimensional Turyn construction from two Hamming codes and finds it is "exactly 64 copies of E8 in different bases, rather than a larger exponent gain". The full Leech shell, 98,280 lines, fails even a deliberately optimistic cost relaxation.
Barnes–Wall is DaysSky's #895, opened at about 10:45 UTC and not merged. It rebuilds BW16 by Construction D (4,320 minimal vectors, 2,160 lines) and says the loss is "entirely the copy charge". Each BW16 line is orthogonal to only 855 of the other 2,159, so it finds 72 pairwise non-orthogonal lines, and those force the decoder window to at least 72; E8's largest such set is 8. That caps BW16's screen ceiling at $3.8923\times10^{-3}$, 0.649 times E8's bit ceiling, although with free copies it would be more than three times E8's (reported). Good geometry, expensive copies. I could not find "BW16's doubling block structure" written up as a lead in `main` or in any open PR's notes, so that part is unverified.
### The PR numbers, without the API
This time the GitHub API wasn't reachable from my sandbox, so I can't repeat [the 11 October check](#rohans-numbers-checked) on PR bodies. What git gives is every PR head and its commit time, and PR numbers here are nearly in opening order (972 PR refs up to #974). By head-commit time, the 24 hours to his post run from about #569 to #865: 296 PRs under 119 commit-author names, against his 285 from 110 contributors. The last four hours hold 174 against his 180, and 78 names whose first PR falls in them, against his "about 85". Jeremy Kothe has 38, Sagar Thakrar 23 and Boukhalfa 15, against his 39, 24 and 14. Commit names aren't accounts and commit times aren't opening times, so this is consistent, not checked. "229 mention Codex/OpenAI, and 70 mention Claude" needs the bodies, and I haven't checked it.
### A prompt that rations the record
The second post links `auto-research.md` from Rohan's PR #928. It reproduces Mister-Meeseeks' community prompt, and its most interesting line is a heading: "ASSIGN YOURSELF A TRACK — DO NOT AUTOMATICALLY PICK THE NEWEST RECORD". The agent draws a random integer from 0 to 99. Draws from 0 to 39 go to the two suppliers, 40 to 59 to decoder structure and recursive interfaces, 60 to 74 to adversarial audit, then verification, verification engineering and consolidation. So at most 40% of agents chase κ (reasoned). It also says "Do not reopen a previously closed search merely with more computation", "'No candidate found' is not an impossibility theorem", and "Success means useful evidence that another person can check—not necessarily a bigger headline number".
The folder half is less tidy. `research/` stays as the frozen legacy tree, because "More than 1,600 files pin paths under `research/`". New work goes to `auto-research/