BlenderWorld: 223 Blender briefs, graded by an agent that opens the .blend

BlenderWorld (arXiv 2610.23986, dated 16 October 2026; Zhejiang University, Alibaba's Qwen team, CUHK, Fudan, NTU) is a benchmark for agents that build 3D scenes in Blender. Each of its 223 tasks is a production brief: recreate this data-centre aisle with a cold-aisle enclosure, model this lion cub with fur and a rig, build a Chesterfield sofa to these dimensions and render it at 4K. The agent gets the brief, any reference images or video, and any spec files. It hands back an editable .blend plus whatever renders and exports the brief asked for.

Qwen's announcement puts the motivation in one line: "Looking right in 2D doesn't really mean the scene holds together in 3D." So BlenderWorld does not score a hero render. It renders the submitted project from eight views it chooses itself, and it sends a second agent into the .blend to check what the pixels cannot show.

I read the paper (the PDF ships in the GitHub repo), the evaluator code at commit dd4f357, and the Hugging Face dataset. I downloaded all 223 task records and rubrics (6.3 MB), installed Blender as the bpy Python module on a shared 4-core cloud VM with no GPU, built two toy submissions, and ran the evaluator's own scene-inspection helper and scoring functions on them. I made no hosted model calls, so I did not run the judges themselves. Labels throughout: measured means I computed it from a file or a run, reported means it is the authors' figure, reasoned means it is my arithmetic on the other two.

Why 3D output is hard to grade

Code benchmarks have tests. 2D design benchmarks can compare a screenshot to a target. A 3D brief has neither.

There is usually no ground-truth asset. "Recreate this room" from one photograph admits many correct scenes, and "design a booth in this style" admits more. A single render is also a weak witness. It sees one side of every object, and an agent that knows it is being judged on one picture can build only what that picture sees. The paper's own failure gallery has exactly this: a farm scene where the two people are photographs mapped onto flat quads, present in the submitted camera and gone from the side.

And some requirements are not visual at all. "Keep the .blend editable", "the wooden leg is 150 mm", "under 3,000 faces", "UVs do not overlap": none can be read off pixels. BlenderWorld routes each requirement to the evidence that can settle it.

The tasks

Every number in this section is measured from the 223 task.json and rubric.json files in the Hugging Face release (revision e67837a), and each one matches the paper's own figure.

The 59 attachments are 28 Markdown, 20 YAML and 11 JSON files (reported, Appendix A). They carry dimensions, material lists, camera settings and polygon budgets. The paper says 500 criteria, 18.9%, come from attachments (reported).

The tasks sit in nine categories. Architecture and interiors dominate with 72; then art and animation 30, products and manufacturing 28, characters and animals 25, nature and landscapes 21, rendering and simulation 17, vehicles and machinery 11, procedural and game assets 10, and science and visualization 9. Some briefs are in Chinese: the sofa task, bw_0160, asks in Chinese for a 2200 by 950 by 750 mm three-seat sofa with 150 mm turned wooden legs on brass casters, 85 mm three-quarter camera, 3840 by 2160 output, and a .blend kept because, in my translation of the brief, "the client will most likely want changes".

expand

What the agent works with

The agent runs in a Linux container with one NVIDIA GPU, Blender 4.2.10, a shell, file tools, image viewing and the Blender MCP server (reported, Appendix D). The MCP tools are execute_blender_code (run Python inside Blender), get_scene_info and get_object_info, get_viewport_screenshot, plus Poly Haven and Sketchfab search and download. So the interface is both: an agent can drive a live Blender over MCP or write scripts and launch Blender headless from the shell. The two GPT models ran under Codex CLI 0.147.0; the other eleven under Claude Code 2.1.220. The Harbor packaging gives each task an 8-hour generation budget (reported, dataset card).

The deliverable is one ZIP per task with final.blend at its root, and optional textures/ and outputs/. A ZIP with no .blend scores zero and stays in the denominator. Evaluation runs in a separate container, and the rubrics are hidden from the agent during generation.

The rubric: 2,651 compound pass/fail criteria

Each task carries between 6 and 22 binary criteria, 11.9 on average, 2,651 in all (measured). Each criterion has a category, an importance weight, an evidence route, and a written pass and fail condition.

• Weights: 2,208 criteria are major (weight 3), 396 secondary (2), 47 detail (1). That is 83.3% at weight 3 (measured), so in practice the weighted pass rate is close to a plain pass rate.
• Routes: 1,544 criteria are scene_summary, which means the scene-inspection agent judges them, and 1,107 are render, judged by a vision model (measured). More than half the benchmark, 58.2%, is decided by opening the file (reasoned: 1,544 / 2,651).
• Categories: function 660, geometry 446, composition 310, material 291, lighting 203, space layout 157, holistic 156, layout 125, physics 124, scale 69, presence 52, count 32, color 26 (measured).

A criterion is not a single fact. The paper calls each one an "acceptance unit": related conditions merged so the criterion passes only when all of them hold. The sofa's R02 reads, in full: every leg "has a recognizably rotationally turned wooden profile, its wooden leg component is nominally 150 mm high, and it is fitted beneath with a connected caster assembly whose wheel and mounting hardware visibly and materially read as brass." Four requirements, one bit. That keeps the bar high and makes a single failed criterion hard to interpret, as the paper admits.

Editability is graded explicitly. 301 criteria in 214 of the 223 tasks mention it (measured, a text search for "editab" in criterion and pass condition). The sofa's R10 wants the .blend to retain "editable sofa geometry, assigned materials, the author-final camera, lights, HDRI world, and render configuration ... rather than only flattened content". A reply under the post asked whether a scene that renders right but is one joined mesh would pass. The rubric does not say; it is left to the judge's reading of "editable", and I could not test it without running the judge.

The rubrics were written by GPT-5.6-Sol, piloted with GPT-5.6-Sol as the judge, and cross-checked by at least two of GPT-5.6-Sol, Gemini 3.1 Pro Preview and Claude Opus 4.8, with human review "complementing" (reported, Appendix B).

How a submission is scored

expand

There are three model roles, and the paper uses GPT-5.6-terra for all three (reported).

1. The render agent picks eight views. A coding agent (the Claude Code CLI, pointed at whatever Anthropic-compatible endpoint you configure) opens the .blend, reads the brief and the rubric, and renders eight 1024 by 1024 views in Cycles at 64 samples. It starts from the submitted camera, then picks wide views and at least two detail views relevant to the rubric. It may move cameras and change render settings. It may not touch geometry, materials, lighting, animation or colour management, and it may not repair defects. Each view gets at most two corrective re-renders, enforced by a Blender startup hook that refuses a fourth attempt at the same output path (measured from render_agent.py and the evaluation guide).

2. The scene inspector opens the file. For scene_summary criteria, a second coding agent copies the .blend and runs Blender headless with a probe script it writes itself. The prompt in judge_blender_agent.py asks it to pull object lists, triangle counts, modifiers and Geometry Nodes inputs, shape keys, drivers, bounding-box dimensions in millimetres, unit settings, render engine and samples, material node graphs, UV overlap, physics settings and embedded scripts, and to save the probe and its raw output as an audit trail.

The rule that makes this more than a script runner: for each criterion the agent returns two sub-verdicts, bpy_check (does it hold in the file) and visual_check (does it hold in its own renders), each pass, fail or not applicable. The harness, not the model, then sets the score to 1 only if every applicable sub-verdict passed. A red Principled BSDF base colour that renders grey fails. A dimension that looks right but measures wrong fails.

3. The vision judge scores the render criteria from the eight views, the agent's own exported images and up to sixteen frames from exported video, with original exports taking precedence when evidence conflicts. A separate blind pass writes prompt-free descriptions of the submission and reference images for grounding. The same judge rates eight dimensions from 1 to 10: scene fidelity, physical plausibility, style and colour, realism, spatial layout, proportion and scale, lighting and materials, completeness.

From evaluation.py, with $w_j$ the weight and $\hat y_j \in \{0,1\}$ the verdict of criterion $j$, and $d_k$ the eight ratings:

The benchmark score is the mean of $S$ over all 223 tasks, with failed evaluations as zero. The public evaluator refuses a result unless every criterion has an integer 0 or 1, all eight ratings are integers from 1 to 10, and the fused score matches this formula to within a millionth.

The sofa task below is real: its eleven criteria, their weights and routes come straight from bw_0160/rubric.json. The verdicts are yours to set.

Nine of the sofa's eleven criteria go to the scene inspector. Weights sum to 32 (ten at weight 3, one at 2), so one missed major criterion costs 9.375 rubric points and half that, 4.6875, on the task score (reasoned). The dimension score moves in steps of 1.25 per rating point.

What I ran

The exception for this piece let me install Blender as a Python module if it fit in a gigabyte. The bpy 4.2.0 wheel installs to 1.5 GB, so I used bpy 4.5.0 instead, which came to 888 MB in a venv (measured). That is not the evaluator's pinned 4.2.10, and the real evaluator would abort on the version mismatch before any model call. The helper I ran uses only APIs present in both.

I wrote a builder that makes a toy version of the sofa: a box body sized to 2200 by 950 by 750 mm, four cylinder legs on brass caster wheels with mounts, a camera, Cycles at 64 samples and 3840 by 2160. Two variants:

• drift: 102 mm of timber on a 48 mm caster, 50 mm lens. These are the leg numbers from the paper's Opus-5 failure case (Figure 15c below).
• spec: 150 mm of timber on the same caster, 85 mm lens.

Then I opened each final.blend read-only and ran the evaluator's own helper, evaluators/evidence_checks.py, which the inspector is told to use, followed by the kind of measurement the inspector prompt asks it to write.

                              drift          spec
scene_facts().engine          CYCLES         CYCLES
scene_facts().cameras.lens_mm 50.0           85.0
camera down pitch (deg)       8.27           8.27
UV overlap, SofaBody / Leg_0  none / none    none / none
timber leg height (mm)        102.0 x 4      150.0 x 4
floor to frame (mm)           150.0          198.0
triangles, 13 objects         924            924
probe wall time               0.023 s        0.020 s

All of it measured, CPU only. Two things fall out.

The fixed helper is narrow. scene_facts() reports the Blender version, the render engine, frame rate, each camera's lens, depth of field and its downward pitch from the horizontal, rigid-body substeps, and a triangle-by-triangle UV overlap test. It does not report object dimensions, materials, modifiers or counts. Those, which are most of what scene_summary criteria ask about, come from a probe the judging model writes on the spot. "Scene inspection" in BlenderWorld means an agent with bpy, not a deterministic checker, and the paper lists deterministic dimension and tolerance scripts as future work. The probe and its output are saved next to every verdict as the audit trail.

The wrong quantity measures correctly. In the drift build, floor to frame is exactly 150.0 mm, because 102 plus 48 is 150. That is the check Opus-5 wrote for itself in the paper's case study, and it passed its own verifier. The brief asked for 150 mm of wood. A scene inspector that measures the timber object gets 102 mm and fails R02. Whether it measures the right thing depends on the probe it writes.

Then I fed verdict sets through the repository's build_score_summary() and the inspector's normalize_agent_rubrics() (measured):

• All eleven pass: rubric 100.0.
• Drift build, R02 and R08 fail: rubric 81.25. Losing only the weight-2 R04 instead gives 93.75.
• A model that self-reports score: 1 but says bpy_check: pass, visual_check: fail gets 0. The harness overrides it and records cross_check_applied: true. A criterion where both sub-verdicts are not applicable falls back to the model's own score.

Results

All reported (paper Table 2; one run per agent per task, highest available reasoning effort). The table is internally consistent (reasoned): for every row, the mean of Rubric and Dimension equals Score to within 0.05; the 78-to-145 weighted mean of Text and MM equals Score to within 0.07; and Perfect times 223 lands on a whole number of tasks (94 for GPT-6-Astra, 57 for Opus-5, 35, 26, 23, 14, 15, 12, 8, 3, 4, 1, 1), which match the counts in the paper's Figures 9 and 10.

GPT-6-Astra leads every column, with fewer output tokens per task (84.5K) than Opus-5 (241K) or DeepSeek-V4.1-Flash (209K). Its lead is bigger on image and video tasks (80.7 against Opus-5's 72.1) than on text (78.6 against 74.8). And full success is rare: the best agent passes every criterion on 94 of 223 tasks; the paper's Appendix E.2 says 143 of its outputs clear 90% of criterion weight, so 49 miss by a small margin (reported).

The 50:50 fusion is a choice, and it barely matters for the ranking. Because the benchmark score is a mean of a linear combination, the leaderboard at any weight is the same combination of the two column means, so I can re-weight it exactly from Table 2.

Across every weight from all-visual to all-rubric, exactly one pair of agents swaps: Grok-4.6 and Opus-4.8, whose lines cross at a rubric weight of about 0.455, just under the paper's 0.5 (reasoned; it matches the paper's Figure 12b). Gemini-3.8-Flash and MiniMax-M3 are the only agents whose visual ratings beat their rubric scores.

expand

Can the judge be trusted?

The paper checks its judge against people on 150 pairs of outputs for the same input, three human raters per pair, 147 pairs after dropping three-way splits (reported, Table 6). With GPT-5.6-terra as judge, the fused score picks the human-preferred output 82.99% of the time. On the 70 pairs whose average scores are within 10 points, that drops to 68.57%. The rubric score alone does worse: 73.47% overall and 51.43% on close pairs, which is barely better than a coin. The dimension score, the eight 1-to-10 ratings, agrees with people more than the rubric does in every judge tested.

Read that next to the leaderboard. A gap of a few points between neighbours, like Grok-4.6 and Opus-4.8, is inside the range where the judge and people disagree a third of the time on individual outputs. Means over 223 tasks are steadier than single pairs, but the paper reports one run per agent and no variance, so I cannot say how much steadier (reasoned).

Two more things a reader should hold in mind, neither of which the paper tests:

• Family overlap. The judge (GPT-5.6-terra), the rubric author and pilot judge (GPT-5.6-Sol), and the first- and third-placed agents (GPT-6-Astra, GPT-5.6-Sol) all come from one vendor. The human-agreement study does not break results down by which model produced the outputs, so self-preference is neither shown nor ruled out.
• The rubrics are public. Every task folder in the Hugging Face release carries its rubric.json under tests/. That is necessary for anyone to run the evaluator, and it means a future model trained on the web can have seen the answers' checklist. Generation-time isolation only protects the runs in the paper.

How agents fail

expand

The paper's failure analysis (reported) names four patterns: dynamic interactions that never happen, scenes that only exist from one camera, distorted fine anatomy, and objects that do not touch what they should rest on. The trajectory traces in Appendix E.8 are better than the gallery, because they show the cause.

expand

Each of the three is an agent that verified something, carefully, and verified the wrong thing. GPT-5.6-Sol held its forced-perspective alignment error under one millionth and left the body floating. Opus-5 rewrote "150 mm wooden legs with brass casters" into "150 mm total including casters" in its own task list, built to that, and confirmed it. That is the same pattern I reproduced with the toy sofa above, and it is the strongest argument for the benchmark's design: the judge has to re-derive the requirement from the brief, not from the agent's notes.

The tool-use numbers point the same way. Seed2.1pro-0915 and DeepSeek-V4.1-Flash look at images the most, 33.9 and 31.3 times per task, and score in the middle; GPT-6-Astra looks 15.0 times and leads (reported, Figure 6). Looking is not the bottleneck. Knowing what to check is.

What it costs to run

The evaluator needs Blender 4.2.10, Node.js 22.17.0, ffmpeg, the Claude Code 2.1.220 CLI and a vision Chat Completions endpoint. Every task runs two coding agents and a vision judge, so a full pass is 223 agentic sessions on your bill; the Harbor package asks for one GPU and an 8-hour timeout per phase (reported). bw evaluate --dry-run checks the submission inventory without calling anything.

Where it sits

BlenderWorld is the third agentic 3D benchmark on this site in a few weeks, and they measure different things. AA-3D has agents write three.js pages and lets a human panel vote on blind pairs. 3DCodeVerse treats the program as the artifact and scores renders and measured gaps in a repair loop. SKY-001 compared frontier models each writing one Blender script for a whole car. BlenderWorld's contribution is the .blend as evidence: an editable project, opened and probed, with the render as a veto. If you are building a Blender agent, the Blender 5.3 splat importer and WorldClaw, which builds whole worlds as Blender programs, are the neighbouring pieces.

What I take from it: the rubric-plus-inspection design is right, and the paper's own trajectories show why. The thing to watch is that the inspector is itself a model writing bpy. Until the dimension and tolerance checks are deterministic scripts, which the authors list as future work, a BlenderWorld score is two layers of agent judgement, and its close calls should be read that way.

Unverified

• I did not run the three model roles, so no score here reproduces Table 2, and my bpy was 4.5.0, not the pinned 4.2.10.
• Whether a single joined mesh passes the editability criteria is the judge's call; the rubric text does not settle it.