# 3DCodeVerse: 2.03M rows of 3D code, and a harness whose score is capped by a ruler
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/3dcodeverse
> date: 2026-10-17
> tags: 3d, datasets, code-generation, harness, agents, procedural-generation, fine-tuning, benchmarks
[3DCodeVerse](https://arxiv.org/abs/2610.22341) (arXiv 2610.22341, submitted 15 October 2026; USC, Yale, Rochester, UCSD, Google DeepMind and others) is a bet that the right output format for 3D generation is a program. Not a mesh, not a splat: a CadQuery script, a Blender Python generator, an OpenSCAD part, a GLSL shader, a three.js page. A program can be edited, re-run with different parameters, measured, and diffed. A mesh can only be looked at.
The release has two halves. **3DCodeVerse-2M** is a corpus of executable 3D programs in the [Hugging Face dataset](https://huggingface.co/datasets/ilabai/3dcodeverse). The **3DCodeVerse Harness**, in the [GitHub repo](https://github.com/gaoypeng/3dcodeverse), wraps a coding agent in a loop that builds the program, measures the result, has a VLM judge the renders, and sends targeted repairs back. The announcement's numbers are "2.03M code pairs, 269K+ executable 3D programs, 128K+ CAD objects, 119K+ procedural shaders, 7 languages/runtimes, and 4 generation tracks", a fine-tuned Qwen3.8-27B going from 51.9% to 92.9% execution on 3DCodeBench, and 90% human preference over Claude Design on the same Claude Opus 5.5 backbone.
I read the paper and the repo, counted the dataset from its own parquet files, rendered five of its OpenSCAD programs on a CPU, and opened one of the harness's own recorded runs. Labels throughout: **measured** means I computed it from a file or a run, **reported** means it is the authors' figure, **reasoned** means it is my arithmetic on the other two.
## Why 3D as code is hard
Code generation has a cheap oracle: run the tests. 3D code has two oracles and both are expensive. The first is execution. A Blender script that calls an operator removed in Blender 5 does not produce anything, and the paper says this is exactly why zero-shot Qwen3.5-9B executes 0 of 212 3DCodeBench tasks: most of its scripts "fail immediately by calling scene-clearing operators unavailable in Blender 5" (reported).
The second oracle is geometry, and it is worse. A program that runs can still build a seat floating 12 mm above its legs (the paper's own example) or a hinge that hits the base only at the top of its swing. A VLM looking at renders will miss some of those and invent others. So the real questions are where the supervision comes from, and what to trust when checking the output.
## What a "code pair" is
A record in the training splits is a ShareGPT conversation. A system prompt names the dialect and the output contract, the human turn is a description of an object, and the assistant turn is the complete program in one fenced block. That pair is the unit behind "2.03M".
The descriptions come from a VLM that captions each program from its code and renders under three conditions: an `instruction` (what a user would ask for), a `detailed` visual description, and a `factory` construction recipe. Each caption type makes its own row. The unified export also has two formats per source (a native one and a standardized `_text` re-template), and multi-pass Shadertoy shaders get one extra row per render pass. The paper says all of this in Appendix B.1: the unified text export is 2,028,286 rows, made of 891,788 native rows, 848,819 standardized rows and 287,679 pass-flattened shader rows (reported).
I wanted that from the files rather than the appendix. The dataset's default config is `split="text"`, 30 parquet shards and about 902 MB. I read only the `id`, `source`, `subset`, `caption_type` and `compiles` columns of every shard with HTTP range requests. The result (measured):
- **2,028,286 rows**, matching the paper and the card.
- **324,709 distinct `id` values.** That is about **6.2 rows per id** (reasoned). Against the paper's own catalogue of **269,597 unique executable programs** (reported, Table 4) it is about 7.5.
- Caption types: 688,365 `detailed`, 681,095 `factory`, 655,144 `instruction`, and a few thousand of others or none.
So "2.03M code pairs" and "269K+ executable 3D programs" are the same corpus counted two ways, and the first number is mostly the second one repeated. The paper's Table 1 does mark its unit as "Pairs". It matters because repetition is not free: the conversion report in the dataset's `_reports/` folder says 3DCodeBench execution fell from 90.1% to 34.4% at a fixed token budget when all three caption variants of each program were used (reported). The paper's best mixtures use one caption per program.
Even the program count double-counts a little: the 6,146 Articraft URDF records are 3,114 mechanisms in two variants, and the 486 3DCodeBench entries are 243 generators in two variants (reported, Appendix B.2).
## The languages, the tracks, and which "7" is meant
There are two different lists of seven, and the announcement blends them.
The **dataset's** dialects, from the card's LLaMA-Factory index (reported): CadQuery (405,189 rows), GLSL (356,700), Blender Python (102,375), URDF (18,438), OpenSCAD (3,576), web animation (2,078) and three.js (1,578). Table 1 of the paper says six code types.
The **harness's** seven runtimes are a different set (Appendix A.1, reported): Blender Python, CadQuery and three.js for static objects; URDF with Blender-authored links for articulated objects; three.js for scenes; GLSL and `moderngl` (Python OpenGL) for graphics. OpenSCAD is in the data but is not a harness runtime.
The four **tracks** are the harness's:
| track | what is generated | runtimes | the gate that measures it |
|---|---|---|---|
| `static_object` | one object, as parts plus an assembly file | Blender, CadQuery, three.js | connectivity, contract (planned bounding boxes) |
| `articulated_object` | link meshes plus a kinematic tree | URDF + Blender | joint sweep across limits, motion direction |
| `scene` | zones, assets, terrain, cameras | three.js | grounding, collisions, framing, a draw-call budget |
| `graphics` | a time-varying fragment program | GLSL, moderngl | frame statistics: NaN, black, blown, static, flicker |
Where the programs come from (reported, Table 4): 128,201 DeepCAD sketch-and-extrude sequences compiled into CadQuery by a deterministic script (47.55% of programs), 119,961 Shadertoy shaders (44.50%), then Articraft, Bioinspired3D, Thingiverse, 3DCodeBench's Infinigen-derived generators and a few hundred agent-written samples. So "128K+ CAD objects" is DeepCAD, and "119K+ procedural shaders" is Shadertoy. Together they are 92% of the programs (reasoned), and neither one is something the harness wrote.
## The harness: plan, build, measure, judge, fix
1. **Plan.** An LLM planner turns the prompt into a schema-validated plan: named parts with metric bounding boxes and attachments, plus an acceptance checklist whose items are marked `must` or `should`.
2. **Write.** The plan becomes a DAG of coding tasks with explicit file ownership: one module per part plus an assembly file. Sub-agents write independent parts in parallel. The coding agent is a vendor CLI (Claude Code, Codex, or Gemini's Antigravity) or a single-shot API call.
3. **Build.** The program runs in a sandboxed runtime with a fixed render rig. A failed build gets up to three targeted repair sessions with the traceback.
4. **Measure.** Deterministic gates run on the exported mesh. The connectivity gate takes the ground-contacting component with the most faces as the support structure, and reports every disconnected part with the exact translation that would close its gap. The contract gate checks part counts, orientations and bounding boxes against the plan.
5. **Judge.** A VLM (Gemini 3.1 Pro in the paper's runs) sees labelled multi-view montages, the plan digest, the measurements and the gate findings, but not the code or the generator's reasoning. It writes observations before scores and votes on a binary defect checklist.
6. **Score, in code.** Criterion scores are weighted, defect penalties are subtracted, and the result is capped by the tightest of the defect caps, the gate caps and any unverified `must` item. A passing gate also **vetoes** the judge: if connectivity ran clean, a VLM claim that a part floats is thrown out.
7. **Fix and select.** Gate errors with numeric fix hints go first, then unverified `must` items, then the judge's suggestions, at most six tasks per round. After the last round, the best-scoring round is the deliverable.
Written as one line, the round score is
$$
s_t = \min\Big(\textstyle\sum_k w_k\,\bar c_k \;-\; \sum_{d \in D} p_d,\;\; \kappa\Big)
$$
where $\bar c_k$ is the judge's mean score on criterion $k$, $w_k$ its weight, $p_d$ the penalty of each retained defect, and $\kappa$ the tightest cap. In the repo's `judges/rubrics/static_object_v1.yaml` the object weights are intent 0.22, structure 0.18, detail 0.16, proportions 0.12, assembly 0.12, materials 0.10 and craftsmanship 0.10, the pass threshold is 0.72, and a measured floating part caps the round at 0.6, a penetration error at 0.7, and a contract violation at 0.75 (measured, read from the file).
That $\min$ is the design decision. The judge decides how good the object looks. The ruler decides how good it is allowed to be.
## One recorded run, round by round
The dataset ships the harness's own run logs under `3dcodeverse_harness/`: 41 GB of object runs (measured, from the file listing) and another 33 GB of benchmark and ablation runs (reported by that folder's README). I downloaded an 11 MB ablation archive, `loop_w6`, and opened run `cage_tj_ss2`: the prompt "an ornate victorian birdcage on a stand with a swing and two seed cups", three.js runtime, Gemini 3.6 Flash as planner and single-shot generator, Gemini 3.1 Pro as judge, a budget of 2 refinement rounds. Each round is a JSON record and a git commit. It is an August 2026 development run, not a paper evaluation run, and I chose it because it does not end well.
What happens, in prose (all measured from `rounds/r00.json` to `r02.json` and the git history):
- **Round 0.** Eleven parts, one file each. The swing's top eyelet ends at `y = 1.285` m, so connectivity reports it "floating: nearest supported part is 'CageDomeCrown' at 98.2 mm" and hands back a translation of `(-0.0092, +0.0975, -0.0076)` m. The central pole sinks 17.5 mm into the tripod. The crane arm is 2.5 cm wide where the plan said 12.0. The judge's criteria average 0.806 after weighting (reasoned), defects take 0.18 off, and the floating-part cap pins the round at **0.60**.
- **Round 1.** The fix agent hangs the swing from a seven-link chain that reaches up to 1.392 m and lifts the pole 14 mm (`poleCenterY` from 0.815 to 0.829). Both connectivity errors are gone. But the chain makes the swing 33.5 cm tall against a planned 22.0, which the contract gate flags, and the raised pole now runs 10.8 mm into the crane arm. Weighted criteria 0.877, minus 0.08, capped by the penetration error at **0.70**.
- **Round 2.** The crane-arm socket moves up 7 mm and the overlap clears. To bring the swing back to its planned size, the agent cuts the chain to three links ending at 1.275 m, which is 16.2 mm short of the crown. Round 0's defect is back, smaller. Weighted criteria 0.867, minus 0.15, capped at **0.60**.
The selector commits `restore best round r01`, and the deliverable is round 1 with its 10.8 mm overlap still in it. Generation cost about \$0.64 over 908 s across the three rounds (reasoned, summing the per-round usage).
Two things in this run are worth more than a headline number. First, every round's score is exactly its cap: the judge rated round 2 above round 0, and the ruler overruled it. That is the design working. Second, the fix agent fixes the finding it is given and creates the next one, and here the plan is the conflict. The plan centres the swing at a height of 1.180 m and makes it 22.0 cm tall, so its top is at 1.290 m (reasoned). Round 2's code comment puts the swing's top at exactly 1.290 m, and that is the version measured 16.2 mm short of the crown. Obeying the contract gate makes the connectivity gate fail. No gate is aimed at the plan, so the loop can only trade one error for the other, and two rounds could not show whether more would converge. The paper's recorded stool run in Appendix A.6 is the happy version: a baseline at 0.410 with six floating parts, every structural defect resolved by round 1, finished in 21 model calls, 4.0 active minutes and \$0.74 (reported).
## The results, and what they rest on
All of these are reported; I ran neither the harness nor any evaluation.
| object track, same backbone | native → harness, human preference | judge score (0-100) |
|---|---|---|
| Gemini 3.8 Flash, Gemini CLI | 13% → 87% | 61.3 → 86.6 |
| Claude Opus 5.5, Claude Code | 21% → 79% | 77.2 → 88.5 |
| GPT-6 Astra, Codex | 31% → 69% | 76.4 → 87.7 |
| GPT-6 Luna, Codex | 11% → 89% | 64.4 → 85.9 |
| Claude Opus 5.5, Claude Design | 10% → 90% | 37.4 → 87.7 |
Fine-tuning with LoRA (and, for the 27B, an execution-feedback DPO round) on mixtures drawn from the corpus, evaluated on 805 fixed tasks:
| execution rate, % | 3DCodeBench (212) | Blender (103) | CadQuery (200) | OpenSCAD (50) | GLSL (200) | three.js (40) |
|---|---|---|---|---|---|---|
| Qwen3.5-9B | 0.0 | 1.9 | 10.5 | 50.0 | 33.0 | 72.5 |
| + SFT | 84.9 | 90.3 | 98.5 | 86.0 | 56.0 | 74.5 |
| Qwen3.8-27B | 51.9 | 74.8 | 65.5 | 80.0 | 57.0 | 72.5 |
| + SFT + DPO | 92.9 | 93.2 | 98.5 | 90.0 | 71.5 | 92.5 |
The README's tables match the paper's Table 2 and Table 13 number for number, and the suite sizes add to 805 (reasoned). Execution is not quality: the fine-tuned 27B's F-score at 0.05 against the reference mesh is 0.374 on 3DCodeBench (reported). It runs nine times in ten and is still far from the target shape.
What I would want to know before quoting the harness numbers:
- **The prompt count changes between sections.** Section 5.1 and the README say 100 object prompts. Appendix C.3 says Flash and Opus were run on "the primary 60-prompt benchmark" (52 detailed, 4 medium, 4 simple) and Sonnet, Luna and Astra on "a balanced 30-prompt subset". Tables 6 and 7 have 60 rows. Section 5.1 says 50 scene and 50 graphics prompts; Table 8's paired comparisons have 7 to 11.
- **The judge in the loop is the judge at the end.** For the Flash runs, Gemini 3.1 Pro is both the in-loop judge and the source of every reported Judge Score. The safeguards are a clean-room rebuild, a judge blind to the plan and history, and human preference as a second metric. The human numbers track the judge's, which is the best evidence the harness is not just optimizing its own grader.
- **Claude Design was not run by the authors.** The paper says "we evaluate its released reference outputs directly", and those outputs all execute (100.0%). Their Judge Score of 37.4 rises to 62.4 under the paper's uncapped visual score (reported), so about 25 points of the gap are caps and penalties, which the harness is built to satisfy because it plans in metric bounding boxes. The 90% human preference is the cleaner comparison.
- **It costs about a hundred times more.** Flash in the harness averages \$3.312 per object against \$0.035 for Flash alone and \$0.353 for single-pass Opus (reported, Table 7), with 93.9% of its 20.8M input tokens per object served from prompt cache.
- **The gain range moves.** The introduction says "15.5–29.1 points over one-shot generation"; Appendix C.5 says "+15.5 to +52.3".
## Running some of it
The dataset is executable code, which made it tempting to run. I drew a line at declarative CAD: no Blender, no Python from the dataset. OpenSCAD 2021.01 from Ubuntu's package archive renders a `.scad` file without running anything else. From the 1,193 Thingiverse programs I took five that are single-file (no `include`, `use` or `import`), between 26 and 90 lines, and not under a non-commercial licence. On a shared 4-core cloud VM, CPU only (measured):
| program | licence | CGAL render | triangles | bounding box, mm |
|---|---|---|---|---|
| Parametric Pipe Mounting Flange | CC BY | 1.4 s | 1,404 | 60.0 × 59.7 × 25.0 |
| customizable sphericon | CC BY | 3.0 s | 3,266 | 50.0 × 50.0 × 25.0 |
| Solid Thumbnut | CC BY | 7.0 s | 8,510 | 15.9 × 16.0 × 7.0 |
| Improved parametric propeller | BSD | 9.2 s | 6,904 | 50.0 × 50.0 × 10.0 |
| Printable trumpet / horn | CC BY | 11.6 s | 2,964 | 126.1 × 125.4 × 102.2 |
All five built, all five came out as CGAL "simple" solids, and each matches the dataset's own render of it.
Five of 1,193 is an anecdote, not a pass rate. The dataset's per-sample `meta.json` carries no execution status for these, and in the default split only the Shadertoy rows have a `compiles` flag: 753,000 rows carry one, of which 35,470 are `false`, and the other 1,275,286 rows are null, meaning not checked by that column (measured). The Shadertoy compile story is the best-documented thing in the release: the `_reports/` folder explains that a naive checker failed 4,192 shaders, of which "only 588 actually needed a code change", and a fixed checker plus repairs put the flattened set at 92,534 of 93,434 compiling, 99.0% (reported).
I did not run CadQuery (a heavy OpenCascade wheel), Blender, the GLSL shaders, or the harness, which needs vendor API keys.
## The two traps in the dataset
**The default split includes the held-out benchmark.** The paper is explicit that all 212 3DCodeBench procedural factories, all 3,904 re-seeded instances derived from them, and all per-dialect evaluation splits "are strictly held out" of its training mixtures, and the conversion report drops those instances as "test-set leakage". But the card's quick start, `load_dataset("ilabai/3dcodeverse", split="text")`, returns 170,188 rows whose `source` is `3dcodebench`: 2,824 from the factories and 167,364 from seeded instances (measured), 8.4% of the split. The paper's own results are not affected, because its mixtures are built separately. Anyone who trains on the default split and then reports 3DCodeBench is training on the test generators.
**The licence is a stack, not a label.** The dataset is tagged CC BY-NC-SA 4.0 and gated behind a non-commercial checkbox; the paper adds that "the stricter applicable terms control reuse". Underneath, the sources carry their own terms (reported, Table 4): DeepCAD and Bioinspired3D MIT, 3DCodeBench BSD-3-Clause, Articraft CC BY 4.0, Shadertoy "per shader, mostly CC BY-NC-SA", Thingiverse per item. I counted the Thingiverse items' own `license` field (measured): 581 CC BY-SA, 358 CC BY, 128 CC BY-NC, 78 CC BY-NC-SA, 33 CC0, 12 BSD and 3 public domain. A collection licence does not replace the licence on each work in it, and CC BY-SA cannot be narrowed to NC-SA, so for 581 of those programs (49%) the dataset tag is not the operative licence (reasoned). The harness logs are MIT and the code is Apache-2.0.
Smaller slips: the card's `size_categories` says 10K to 100K rows; Appendix B.2 splits the 119,961 shaders into 98,164 single-pass and 21,585 multi-pass, which sum to 119,749 (reasoned); and Table 13 gives Opus in the harness 88.5 in one panel and 87.7 in the other without saying why.
## Where it sits
This is the most complete release of its kind I have read: data, harness, training recipes, evaluation code, and the run logs behind the evaluation, which almost nobody ships. It is also the corpus-scale version of things this site has looked at one system at a time. [WorldClaw](/articles/worldclaw) builds worlds by writing Blender programs and publishes no numbers; 3DCodeVerse publishes numbers and the logs behind them. The [staged Blender harness whose authors warned that scaffolding gets eaten](/articles/scaffolding-gets-eaten) is the question to ask of the gains here: the gate caps are constraints a better model still has to satisfy, while the planning and skills are what a stronger backbone might absorb. [AA-3D](/articles/aa-3d) judges three.js scenes by human vote rather than by a rubric, and [Qwen3-30B-A3B painting in p5.brush](/articles/paint-with-code) runs the same write-render-judge loop in 2D. For CAD as a kernel rather than as generated scripts, see [usd-brep](/articles/usd-brep-nurbs-kernel). And for the opposite way to get 3D, measured from the world instead of written, see [Hesai Kosmo](/articles/hesai-kosmo) and [InfiniSplat v2](/articles/infinisplat-2): a splat is evidence, a program is a hypothesis you can check.
Sources, all read on 17 October 2026: the X post by @Williamiumli and its thread through the fxtwitter mirror; the [arXiv paper](https://arxiv.org/abs/2610.22341) (HTML v1, including the appendices); the [repository](https://github.com/gaoypeng/3dcodeverse) at commit `b10e61e`; the [dataset](https://huggingface.co/datasets/ilabai/3dcodeverse) at revision `5bae4cb`, its card, `_reports/`, `data/build_report_text.json`, all 30 `data/text` shards (five columns, by range request), `thingiverse/openscad/metadata.parquet` and five samples from its tar, and the harness archives `bench/ablations/loop_w6`, `bench/batteries/complexity_e2e` and `complexity_e2e2`, and `bench/eval/harness_vs_baselines__summary`.