# Which half of the scaffolding gets eaten

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/scaffolding-gets-eaten
> date: 2026-09-22
> tags: explainer, agents, architecture, llm, systems
The authors of a 3D coding harness went back to their own system with a stronger
base model and published what they found:

<Callout type="note">
**"We built our 3D coding harness earlier this year, back when none of the models we tested could really hit the bar. Revisiting it now with stronger models like Astra, we're seeing a sharp threshold: below it, the harness helps a lot. Above it, almost not at all. This is mostly the same across all model providers. A small warning for everyone building 'agents': all of today's scaffolding will eventually get eaten by the base model."**
</Callout>

That is a claim against their own contribution, which is the kind of claim worth
reading carefully. It also cuts against this site. Thirty-two of the articles here
name a harness, a decision layer, a scaffold or an enumerated option set in
their own title or description. If scaffolding is a bridge you build only until
the base model can cross on its own, most of that is a depreciating asset and I
have been writing it up as though it were a fixed one.

So: read the paper, work out what it measured, work out what the revisit adds,
and then test the one distinction that decides how much of the warning applies.
The distinction is between scaffolding that **substitutes** for a capability the
model lacks and scaffolding that **constrains** what the model is allowed to do.
I did not want to assert it. I wanted to take this harness apart and see whether
its parts actually sort that way.

They mostly do. There is a third bucket the binary hides, and the constraint
half is in worse shape than the split implies.

## The paper

[*Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language
Models*](https://arxiv.org/abs/2606.02580) (He, Luo, Ma, Averbuch-Elor — Cornell,
arXiv:2606.02580, 1 June 2026) asks whether a pretrained VLM can reconstruct a
scene from a single image as an *editable Blender program* — no differentiable
rendering, no multi-view supervision, and deliberately no specialist 2D or 3D
foundation models. The output is Python that builds a scene, so the reconstruction
can be relit, rearranged and dropped into a physics engine afterwards.

Their observation is that a VLM asked to do all of that at once fails, and their
fix is decomposition. SEIG splits reconstruction into the order a human artist
works in — **geometry, then material, then composition, then lighting** — and
closes each stage with its own generator–verifier loop before the next one starts.

<Figure
  src="/articles/scaffolding-gets-eaten/fig1.png"
  alt="A left-to-right pipeline. A reference photograph feeds an initializer that produces four candidate scene rollouts; a rollout selector picks one. That scaffold then passes through four boxed stages in sequence — Geometry Refinement, Material Refinement, Composition Refinement, Lighting Refinement — each drawn as a pair of boxes with circular arrows between a Refinement step and a Verifier step. A scene graph runs along the bottom and feeds into every verifier. The rendered scene at each stage grows progressively closer to the reference, ending in a final prediction of a mushroom house."
  caption="SEIG: four coarse rollouts, a selector, then four generator–verifier stages that each commit before the next begins; the scene graph feeds every verifier (He et al., 2026, Figure 2)."
/>

The mechanics worth keeping in mind for later:

- **Initialization samples four** independent scene graphs and coarse scaffolds,
  and a rollout selector keeps the one with the most complete object coverage.
- **Every object gets a stable Blender name** at initialization and keeps it
  through every later stage, so refinement can be addressed at a node.
- **Each stage has a round budget** — five for geometry, three each for material
  and composition, two for lighting — because "refinement effectiveness" degrades
  as context accumulates.
- **The verifier returns a checklist, not a critique.** Free-form feedback was
  "noisy across attempts, giving the generator inconsistent targets"; an explicit
  approval checklist is injected into the generator's next context.
- **Stages are scoped by the tools they get.** Material refinement runs through
  "a material-only tool that permits only material-related edits." Composition
  "is not allowed to edit object geometry or materials." Lighting optimises
  illumination "while keeping object shape, appearance, layout, and camera fixed."

One implementation detail does more work than anything else in this article:
**every reported result uses Claude Opus 4.7**, as both generator and verifier,
at every stage, with no fine-tuning. The paper says why — "any observed difference
in reconstruction quality between methods can be attributed to harness design
rather than the underlying model." That is a good experimental decision and it has
a consequence: there is no model-by-model table in this paper. Not a bar chart, not
a row. One backbone, start to finish.

## There is no bar

"None of the models we tested could really hit the bar" implies a threshold. The
paper does not define one. It reports six continuous similarity scores between the
reconstructed render and the reference image — PSNR and SSIM at the pixel level,
LPIPS and DreamSim as learned perceptual distances, and cosine similarities from a
DINOv2 ViT-L/14 `[CLS]` feature and a CLIP ViT-B/32 embedding. Where reference
meshes exist, each reconstruction is first registered to the reference with NDP and
ICP (whichever gives the smaller Chamfer distance) and re-rendered from the
reference camera, so the camera estimate does not contaminate the score.

The evaluation set is two collections: five renders from each of seven NeRF synthetic
scenes — the `materials` scene with its specular metal spheres is excluded, leaving 35
images — plus 13 object-centric scenes from Edit3D.

| NeRF synthetic | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DreamSim ↓ | DINO ↑ | CLIP ↑ |
|---|---|---|---|---|---|---|
| VIGA (VLM-only) | 12.33 | **0.7122** | 0.3506 | 0.3693 | 0.6221 | 0.8451 |
| VIGA (full) | 11.18 | 0.6647 | 0.3944 | 0.3624 | 0.5545 | 0.7986 |
| SEIG | **13.58** | 0.6881 | **0.3493** | **0.3021** | **0.7188** | **0.8830** |

| Edit3D | PSNR ↑ | SSIM ↑ | LPIPS ↓ | DreamSim ↓ | DINO ↑ | CLIP ↑ |
|---|---|---|---|---|---|---|
| VIGA (VLM-only) | 11.52 | **0.6776** | 0.3931 | 0.3847 | 0.5606 | 0.8366 |
| VIGA (full) | 12.48 | 0.6743 | 0.4466 | 0.4441 | 0.4832 | 0.7883 |
| SEIG | **12.65** | 0.6737 | **0.3823** | **0.3433** | **0.6293** | **0.8446** |

Nothing in those twelve columns is a pass mark. And the absolute level is worth
saying out loud, because it supports the authors rather than undercutting them: a
PSNR of 13.58 dB between a render and its reference is a poor pixel match by any
standard. These numbers are not directly comparable to novel-view-synthesis
figures — different task, post-registration, single input view — but at this level
nobody has hit a bar you would recognise as one. When the authors say the models
could not really hit it, the tables agree.

## The one number the threshold claim is about

Strip the paper to the quantity that is actually a property of the scaffolding and
you get one comparison. **SEIG minus VIGA (VLM-only)**: same backbone, same tasks,
same metrics, both VLM-only, one staged and one monolithic. That difference *is*
the staging, and it is the thing the revisit says goes to zero.

<MeasuredAndAsserted />

At Opus 4.7 the staging is worth **+1.25 dB PSNR and +15.5% DINO on NeRF synthetic**,
**+1.13 dB and +12.3% on Edit3D**. Two things in that panel deserve more attention
than the paper gives them. **SSIM goes the wrong way on both sets** — the unstaged
baseline is the best of the three, and on Edit3D SEIG is third. And the LPIPS win
on NeRF synthetic is 0.3506 against 0.3493: a **0.4% improvement**, which is one of
the "five out of six metrics" the paper claims. The staging is a real effect with a
clear signature: the semantic scores move most (DINO, CLIP, DreamSim), PSNR moves with
them, LPIPS barely registers on one set, and SSIM — the pixel-structure metric — goes
backwards on both. Four convincing columns on NeRF synthetic, three on Edit3D. Not six.

There is also a row the paper does not have, and its absence matters. **There is no
no-harness baseline.** VIGA VLM-only is itself an agentic write–render–compare–revise
loop. Nobody measured "prompt the VLM once for a Blender program." So the claim
"below the threshold the harness helps a lot" is, in this paper, a claim about one
harness beating another harness — not about scaffolding beating none.

## What the revisit does and does not establish

The revisit is a social-media post. That is not a knock; practitioners' reports about
their own systems are useful and this one is being made against interest. But it should
be read as a different kind of object from the paper, and four things follow.

1. **No number.** "Helps a lot" and "almost not at all" describe a shape. The paper
   has six metrics and two datasets; the revisit reports none of them. The right-hand
   panel above is the point: a step, a knee and a straight line all fit those words.
2. **No bar.** The threshold is stated as a property of models ("below it", "above it")
   but the quantity being thresholded was never defined.
3. **The original was not a model sweep.** "None of the models we tested" refers to
   work that is not in the paper. The published record contains exactly one backbone.
   So the before-state of the comparison is unpublished, and so is the after-state.
4. **"Mostly the same across all model providers"** rests on no provider list and no
   per-provider figure. The paper's cross-provider evidence is zero rows.

Now the part that cuts the other way, and it is stronger than any of the four above.
**The paper already located the bottleneck in the model.** Discussing its own result,
it says the pattern is "consistent with the finding from BlenderGym and IR3D-Bench
that **visual precision, not tool orchestration, is the dominant bottleneck** in
current agentic 3D pipelines."

If the binding constraint is the model's visual precision, then better visual
precision dissolves the gap. That is not a reversal — it is the paper's own diagnosis
running forward a few months. The authors wrote down the mechanism that would make
their harness redundant, in the paper that introduced it, and then reported that it
happened. Whatever else is missing, they are not guessing.

## Scaffolding that was already worth less than nothing

Before any threshold argument, this paper contains a cleaner result about scaffolding
that almost nobody quotes. **VIGA full** is VIGA plus SAM and SAM-3D — specialist
segmentation and single-image 3D reconstruction models, bolted on to pre-segment and
pre-reconstruct objects before the agent loop runs. **VIGA VLM-only** is the same
system with those models switched off. Same backbone in both.

<BenchBars
  title="DINOv2 semantic similarity to the reference, NeRF synthetic (higher is better)"
  unit=""
  max={0.8}
  bars={[
    { label: "SEIG (staged)", value: 0.7188, highlight: true },
    { label: "VIGA, VLM only", value: 0.6221 },
    { label: "VIGA + SAM + SAM-3D", value: 0.5545 },
  ]}
/>

Adding the specialist models makes it **worse on five of six metrics on NeRF
synthetic and five of six on Edit3D**. The paper gives the mechanism: the VLM agent
"may overwrite the texture of the 3D objects generated by SAM-3D," and single-view
lifting drags its own failure modes in. The paper points at the humanoid in Figure 3,
where VIGA full exhibits the classic Janus failure — frontal facial features duplicated
onto the back of the head — which neither VLM-only configuration produces, because both
build the figure compositionally out of primitives instead of lifting a single-view mesh.

<Figure
  src="/articles/scaffolding-gets-eaten/fig3.png"
  alt="A five-row comparison grid. Columns are Reference, VIGA (VLM-Only), VIGA (full) and Ours, and each row shows an input view above two novel views. Scenes are a toy xylophone with mallet, an ice-cream parlour on a green base, a sailing ship in a circular water dish, a child in a pink inflatable ring, and a checked cloth basket of bread. The Ours column tracks the reference most closely on materials and layout; VIGA VLM-only loses small elements and surface detail; VIGA full produces flat, mis-coloured and fragmented shapes, with the bread basket collapsing into a draped sheet."
  caption="Same backbone in all three columns. Adding SAM and SAM-3D (VIGA full) costs five of six metrics on both datasets — scaffolding with negative value before any capability threshold is reached (He et al., 2026, Figure 3)."
/>

Hold that next to the warning. This is scaffolding that did not need to be eaten by a
stronger model, because it was already worth less than nothing at the model they had.
"All of today's scaffolding will eventually get eaten" is the optimistic version.

## Substitute, supply, constrain

Here is the distinction I want to test. Some scaffolding exists because the model
cannot do something. Retry loops, output parsers, chain-of-thought prompting, JSON-mode
wrappers, ReAct templates — every one of those was an answer to a deficiency, and every
one of them has been absorbed. That kind gets eaten, and the warning is simply right
about it.

Other scaffolding exists because *I need a guarantee the model cannot give me*. A finite
action space, a typed schema, a validator that runs before anything ships. Its value was
never "the model cannot do this." A better base model does not make an unbounded action
space safe.

Taking SEIG apart, the binary turns out to be missing a bucket. A render is neither. When
the verifier executes the Blender code and looks at the output, it is not compensating for
a weakness and it is not forbidding anything — it is **supplying** a fact the model could
not have had from the inside. Test results, compiler errors and tool responses are all this
third thing, and they are most of what people point at when they say "the harness."

<HarnessLedger />

The honest count: **three of eight are pure substitutes, and they include the mechanism the
paper credits for its entire result.** The staged decomposition exists because "pretrained
VLMs struggle to reconstruct all scene factors simultaneously" — that is the paper's own
sentence, and it is a capability claim about a checkpoint. The four rollouts are best-of-n
bought against variance. The round budget is a cap on context degradation. Every one of
those premises is a thing that scaling fixes.

So on this harness, the warning is right about the part that matters most. I would have
preferred a different answer.

## The part that is a permission

Two of the eight rows are pure constraints, and one of them is worth drawing, because it is
the clearest example I have found in a paper of a rule enforced on the model rather than
read by it.

<StagePermissions />

Read the material row. Material refinement does not merely *tend* to leave the geometry
alone; the agent is handed a tool that can only express material edits. The guarantee is not
in a prompt the model reads, it is in the surface the model can reach — which is the same
placement [agent-harness](/articles/agent-harness) argued for from the other direction: a
permission check the agent can edit is not a permission check.

What this buys is visible in the pipeline's intermediate output. Because no stage may undo
an earlier one, every stage boundary is a committed, coherent, editable Blender program.

<Figure
  src="/articles/scaffolding-gets-eaten/fig2.png"
  alt="Three worked examples, each a row of renders progressing left to right. A coffee cup and saucer on a wooden table, a brown teddy bear on a white rug, and a cartoon house with a tree. Each row starts from the reference photograph and passes through Initialization, Geometry, Material, Composition and Lighting, with red dashed insets magnifying the detail that changes at each step — a saucer appearing, a wood grain arriving, a logo resolving on the bear's chest. Below each row, further renders show novel viewpoints, object rearrangement and relighting of the finished scene."
  caption="Each stage commits before the next begins, so every intermediate is a coherent editable scene — which is what makes the rearrangement and relighting panels on the bottom rows possible at all (He et al., 2026, Figure 7)."
/>

That property is not a capability substitute. Relighting works because lights are a separate
committed stage output; per-object editing works because objects were built independently and
only later assembled; physics works because the result is "a structured collection of
separately addressable meshes" rather than a fused implicit surface, needing no remeshing or
watertighting. None of that is waiting on a better model. A stronger VLM emitting one
entangled blob is still an entangled blob.

## Constraints do not get eaten. They get repriced.

This is where the two-bucket defence of the corpus stops working, and the paper hands me the
evidence.

A constraint's value is the consequence of the event it forbids, multiplied by how often that
event would otherwise happen. A stronger base model moves exactly one of those terms. It
overwrites an accepted stage less often; it does nothing at all to what happens when it does.
Frequency is a model property. Consequence is a deployment property. So the expected value of
the guarantee falls, slowly, and never reaches zero.

But the constraint has a second term, and that one moves the wrong way. **What you gave up to
get the guarantee is worth more when the model is better.** SEIG's limitations section is the
receipt, and it describes its own permission matrix:

<Callout type="warning">
**"Errors introduced in early stages may propagate throughout the pipeline, leading to local minima from which later stages cannot easily recover. For example, inaccurate geometric reconstruction may constrain subsequent material, lighting, or compositional reasoning."**
</Callout>

That failure exists *because* the composition stage is not allowed to edit geometry. The
ten frozen cells in the matrix above are the guarantee and the failure mode, and they are the
same ten cells. The paper even names the fix — "additional global refinement passes
that revisit and jointly optimize earlier scene factors" — and its cost, "substantially
increased computational cost and inference time."

Now run the base model forward. A model good enough to hold all four factors at once is
precisely the model that would use a global pass well. The ban costs nothing when the model
cannot do the thing anyway; it costs a great deal when the model could. So the constraint tier
is not safe. It does not get absorbed — it gets **outvoted**, when the opportunity cost of the
ban rises past the expected cost of the event it prevents.

Which is a real concession to the warning, arrived at from the other side. Substitutes get
eaten. Constraints get audited. Both roads end with less scaffolding than you have now; only
the reasoning differs, and the reasoning is what tells you which ones to keep.

The ones that survive an audit are the ones where the forbidden event is expensive in a way
capability does not touch: money moved, data deleted, a claim published. There, frequency
falling from 1% to 0.001% changes the arithmetic and not the decision.

## Running the test on this site

<CorpusSplit />

Three of this site's own arguments, cut in half. The pattern in that figure is the finding:
each piece welded a capability argument to a guarantee and shipped them as one claim. The left
column is depreciating in every row. The right column is not, and in every row this site stated
it *less* clearly than the left.

The `webmcp-windtunnel` line I keep coming back to is the cleanest case. A tool refused a bad
argument and answered with its four legal values; the next step used one. The article's own
gloss: that recovery is "a property of the code on the website, not of either model." That is
exactly right and it is exactly the kind of thing a better base model does not supply. But the
same article's headline was a cost multiplier between two checkpoints, which is the most
perishable number in it.

And there is a measured result in this corpus that contradicts the warning's universal
quantifier outright. [The harness effect](/articles/harness-effect) held the model fixed and
swapped only the orchestration layer across six models: cost per task fell **for every single
one**, by roughly a third to 61%, and the paper's mechanism list is cache-shape discipline, incremental
compaction, context offload, durable execution and failure-spend governance. Not one of those
is a capability the base model can absorb, because none of them is about the model. Prompt
caching pays out when your prefix is byte-stable across turns; that is a fact about the bytes
you send and the provider's billing, and a smarter model does not make your prefix stable.

Set that beside the other edge. [Agent harnesses](/articles/agent-harness) records STOP
improving results on GPT-4 and *degrading* them on GPT-3.5 and Mixtral — Weng's reading being
that "the base model must be capable enough to improve the mechanism." So substitute
scaffolding helps inside a window: too weak and the model cannot use it, too strong and it does
not need it. Both edges move right over time. A harness built on a substitute has a shelf life
equal to the width of that window, and this paper's revisit is a report of the upper edge
arriving.

## The take

The warning is right, and it is right about more of this paper than I expected before I
sorted the parts. SEIG's headline mechanism is a capability substitute in its authors' own
words, its bottleneck is a model property by its own diagnosis, and the gains it measures are
concentrated in the semantic metrics that a better visual model would be expected to move.
Reporting that against your own contribution is good practice and I would rather have this
finding than not.

What the revisit does not have is a measurement. The staging gain is a concrete, published,
two-dataset number at one backbone; the claim that it goes to zero at a stronger one is two
adjectives. The experiment that settles it is the same evaluation set, the same six metrics, a
different backbone — a weekend's work for anyone with the code, which is not released.

The distinction survives contact, with one correction and one caveat. The correction: it is
three buckets, not two, and the middle one — execution results, tool responses, the render —
is most of what people actually mean by a harness and is the one nobody argues about. The
caveat is the one that costs this site something. Constraints are not a safe place to have put
your engineering. They do not get eaten, but the thing you traded away to get them appreciates
with every checkpoint, and a constraint whose price you never wrote down will get removed by
someone who has only ever seen its cost.

So the rule I would actually apply, going forward: for every bounded action space, typed
schema and validation gate, write the sentence json-render wrote in its README — what this
makes impossible. If that sentence names something a stronger model would have done well, the
guarantee is on a clock. If it names something you never wanted to happen at any capability
level, it is not scaffolding. It is the specification.

<ChangeMyMind>

<Falsifier claim="Scaffolding splits into substitutes (eaten), supply (shrinks) and constraints (repriced, not eaten).">
This is the article's load-bearing claim and it is testable on this exact system, now that the revisit says a strong enough backbone exists. Rerun SEIG on the same evaluation set with two separate ablations, not one. **(a)** Remove the staging — one monolithic generator–verifier loop over all four factors, same total round budget. **(b)** Keep the staging and remove only the stage-scoped edit permissions, so every stage may write every factor, same budgets. The split predicts that (a)'s delta collapses toward zero at the stronger model while (b)'s mean metrics also collapse — but that the *rate at which a later stage regresses an accepted earlier output* stays above zero, and the committed-intermediate property is lost at every model. If (b) shows zero regressions across a few hundred stage transitions with permissions removed, the permission was substituting for a capability after all, the constraint bucket is empty on this system, and the split is a distinction without a difference.
</Falsifier>

<Falsifier claim="The threshold claim is about one measurable quantity: SEIG minus VIGA (VLM-only), and nobody has measured it twice.">
Run both configurations on the same evaluation set with a stronger backbone and report the same six metrics. If the staging gain holds near +1.25 dB PSNR and +15.5% DINO on NeRF synthetic, the threshold claim is wrong on the harness it was made about. If it lands near zero, it is right and this article's substitute classification of the staging is confirmed from the outside. The obstacle is not difficulty, it is that no code or project page was released, so only the authors can run it cheaply — which is worth saying, because a claim that only its makers can check is a claim with a structural problem.
</Falsifier>

<Falsifier claim="The paper defines no bar, so 'hit the bar' is a construct introduced in the revisit.">
I searched the paper for a pass threshold on any of the six metrics and found none; the evaluation is entirely continuous scores with no acceptance criterion. If the authors publish the bar they were working to — a PSNR floor, a DreamSim ceiling, a human-judgement rate — then the before-state is well defined after all and this objection drops. I would still want the after-state measured against it.
</Falsifier>

<Falsifier claim="Constraints get repriced: the ban costs more as the model improves.">
Implement the global refinement pass the limitations section proposes — one pass, after lighting, allowed to edit any factor — and measure it at two backbones. My prediction is that it recovers little at Opus 4.7 and substantially more at a stronger model, which is what "the opportunity cost of the ban rises" means in numbers. If the recovery is flat across backbones, the frozen cells cost the same at every capability level, the ban never gets outvoted, and the reprice argument is decoration.
</Falsifier>

<Falsifier claim="Some scaffolding cannot be eaten because it is not about the model at all.">
The harness effect's efficiency result is the counterexample to the universal quantifier: cache-shape discipline and its four companions cut cost for all six models tested, by roughly a third to 61%, and prompt-cache hit rate is a property of prefix byte-stability, not of intelligence. If a provider ships server-side caching that achieves a high hit rate regardless of how the client assembles its prompt, that mechanism does get eaten — not by the base model, but by the platform, which would make my category boundary the wrong one and "eaten by something" the right framing.
</Falsifier>

<Falsifier claim="Adding specialist models to the loop made results worse on five of six metrics on both datasets.">
Computed from Tables 1 and 2 as published, comparing VIGA full against VIGA VLM-only. The obvious confound is that VIGA full was not tuned by its own authors for this evaluation — an ablation run by a competing method is a weak measurement of that method. If VIGA's authors rerun their full configuration on the same evaluation set and it beats VLM-only, the negative-value reading is an artifact of the reimplementation and I should retract that section, keeping only the Janus-head observation, which the paper reads off its own figure and which does not depend on the table.
</Falsifier>

</ChangeMyMind>

---

*Source: [Thinking in Blender: Staged Executable Inverse Graphics with Vision-Language
Models](https://arxiv.org/abs/2606.02580) (Guangzhao He, Rundong Luo, Wei-Chiu Ma, Hadar
Averbuch-Elor — Cornell University), arXiv:2606.02580, 1 June 2026. Figures 2, 3 and 7 are
reproduced from the paper for commentary; all other diagrams are mine, computed from the
paper's published tables or drawn from its method text. There is no project page, no code
release and no video; the quoted revisit is a social-media post by the authors, not a paper,
and is treated as such throughout. Table values are quoted as published; the relative gains
are my arithmetic on them.*