# Qwen3.5-4B-3D-Coder: a 4B model, 4,146 teacher traces, and the scenes that actually run

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/qwen35-4b-3d-coder
> date: 2026-11-01
> tags: qwen, threejs, fine-tuning, small-models, datasets, 3d

A [post by @dextune](https://x.com/dextune/status/2116164031488868393) says a 4B model can generate interactive 3D scenes with Three.js, and that on 29 October the full FP16 weights, the training dataset and a technical report became public. Training took about 60 hours, the post says, and the scenes come out as HTML and JavaScript in chat mode, not through an agent harness. Specific generation settings and a Three.js system prompt are required.

That is a lot of artefacts for one thread, so I read them instead of the thread. The model is [`ccore/Qwen3.5-4B-3D-Coder`](https://huggingface.co/ccore/Qwen3.5-4B-3D-Coder), an independent fine-tune of [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) that its own card says is "not an official Qwen release". The data is [`ccore/ThreeJS-3D-Scenes`](https://huggingface.co/datasets/ccore/ThreeJS-3D-Scenes). Every claim below carries one of three labels. **Measured** means I ran code against a file and this is what it printed. **Reported** means the author says so and I did not check it. **Reasoned** means arithmetic on numbers of either kind.

<ModelCard repo="ccore/Qwen3.5-4B-3D-Coder" />

What I could not do is run the model. A 4B checkpoint in int8 is about 5 GB, and the sandbox I work in shares its memory across jobs. More to the point, one answer from this model is tens of thousands of tokens long, which I come back to. What I could do is run the thing the model was taught from: all 213 scenes in the validation split, in a real browser engine. That turns out to say something useful about the gap between a teacher's output and a student's.

## What was released

| Artefact | What it is | Licence |
|---|---|---|
| FP16 weights | four safetensors shards, 4,539,265,536 parameters | Apache-2.0 |
| GGUF | Q4_K_M and Q8_0, plus an optional BF16 vision projector | Apache-2.0 |
| Dataset | 4,146 rows: 3,933 train, 213 validation | MIT |
| Technical report | 10-page PDF dated 28 October 2026 | in the model repo |

All four are reported by the repos; the licence files I read (the model repo ships the Apache-2.0 text). The dataset's rows were generated with MiniMax M3.1 Flash, so the scenes and the reasoning traces are a teacher model's work, and the dataset card says so. The report states that the data-generation and curation pipeline "remains separate from the public release". You get the output of the pipeline, not the pipeline.

One of the card's lines is worth quoting because it is the whole usage contract: the model is meant to be run with a fixed system prompt, which tells it to import only the Three.js core module from one cdnjs URL, to import no addons, and to write its own camera controls. I compared that against the dataset. **Measured:** 212 of the 213 validation rows carry a system message, and it is that same 224-character string, character for character. The model was trained to answer under that prompt. Without it, nothing in the training data says what happens.

## The model: a Qwen3.5-4B with its prediction head removed

I read `config.json` and the four safetensors headers by HTTP range request, without downloading any weights, and diffed them against the base model's.

**Measured.** The 4,539,265,536 parameters split three ways: 3,570,052,096 in the 32 language layers, 635,699,200 in the tied token embedding (248,320 by 2,560), and 333,514,240 in the 24-layer vision tower. All 723 tensors are F16. The layer pattern is the one the base model has: 24 `linear_attention` layers and 8 `full_attention` layers, every fourth layer full. `layer_types` is identical to the base's. Only dtype, token-id and vision rope fields differ in the config, which is what you expect from a checkpoint re-saved by a newer Transformers (5.20.2, against the base's 4.57.0.dev0).

The one structural difference is in the weight index. The base [Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) lists 738 tensors; this checkpoint lists 723. The 15 that are missing are exactly the 15 whose names start with `mtp.`, the multi-token-prediction head, which the config still declares (`mtp_num_hidden_layers: 1`). That is why the card says to turn MTP off in LM Studio, and why the report mentions "a layer/MTP metadata adjustment" during GGUF conversion. **Reasoned:** the fine-tune updated the language parameters, and the head did not make the trip, so any speculative-decoding path that expects it has nothing to read. That costs decode speed, not quality, and I have not measured either.

The hybrid layout matters for this task for a reason that is arithmetic. Only the eight full-attention layers keep a key and a value for every past token. With four KV heads of width 256 in F16, that is 2 × 8 × 4 × 256 × 2 = 32,768 bytes per token. The report evaluates at 65,536 tokens of context, so the cache is 2.00 GiB. A 32-layer model that kept a cache in every layer would need 8.00 GiB for the same window. The 24 linear layers add a fixed state, which I make 48 MiB.

<KvBudget />

The linear-state figure is the weakest number here. I assumed the gated delta-rule layout of Qwen3-Next (32 value heads, a 128 by 128 matrix each, fp32) because the config's `linear_*` fields and `mamba_ssm_dtype` match it. I did not read the modelling code for 3.5. Even if it is off by a factor of four, it is under 200 MiB against a 9 GB checkpoint.

## The recipe (reported) and what it implies (reasoned)

The report is candid about its own status: "an exploratory study", with a protocol that "included retries and different stopping rules across versions". Its recipe is plain supervised fine-tuning, with no RL stage.

- Full fine-tuning of the language parameters with the vision tower frozen, in checkpoints 1 to 4.
- One LoRA epoch (r = 16, alpha = 32, constant 5 × 10⁻⁵) on 178 examples of at most 16,384 tokens, merged into the weights as checkpoint 5.
- A final full run from checkpoint 5 with a fresh Adam state: microbatch 1, accumulation 2, peak learning rate 10⁻⁵ decaying to 2 × 10⁻⁶ on a cosine, 1,956 updates, checkpoint 8 at the end. About 59 h 55 min of training sit in the final weights, 16 h 01 min 37 s of it in that last run.

<Figure
  src="https://media.thesatyajit.com/ai/articles/qwen35-4b-3d-coder/fig1.png?v=6fe59c6a8caeb3f6"
  alt="Boxes joined by arrows: the official Qwen3.5-4B, then checkpoint 1 with 717 examples and 3 epochs, then checkpoints 2 and 3 on 1,915 to 3,933 examples, then checkpoint 4, then checkpoint 5 (a short LoRA merged in), then the final full fine-tuning that makes checkpoints 6, 7 and 8. A dashed arrow from checkpoint 5 leads to discarded 24K and 34K LoRA experiments."
  caption="The lineage of the released weights; the dashed branch is the LoRA runs the author discarded. (technical report, Figure 1)."
/>

Two things about the recipe are easy to read past. First, the data was grown in three steps, 797 then 2,128 then 4,146 rows, and the model was re-trained as it grew, so there is no single "trained on 4,146 examples for N epochs" number. Second, the report says discarded LoRA branches "showed lower scene quality and more repeated variable declarations". The card's one known issue is "duplicate variable names". Those are the same defect in two documents.

**Reasoned, from the report's totals.** The final run processed 112,903,599 input tokens in 57,697 seconds, which is 1,957 tokens a second on one RTX PRO 6000 Blackwell. With 4.206 billion language parameters (layers plus the tied embedding, which acts as the output head), the usual 6 × parameters × tokens gives 2.85 × 10¹⁸ FLOPs, or about 49 TFLOP/s sustained. That ignores attention and the FP8 layers, so treat it as an order of magnitude. It says the run was ordinary, not heroic.

Then the supervised-target count. The report gives 112,431,961 supervised targets against 112,903,599 input tokens. The difference, 471,638, is 120.6 tokens a row, which is about the size of a prompt (I measure 113 tokens of system plus user text per validation row, plus a few template tokens). **Reasoned:** the loss is taken on everything the assistant says, the reasoning trace and the code, and on none of the prompt.

<Figure
  src="https://media.thesatyajit.com/ai/articles/qwen35-4b-3d-coder/fig6.png?v=29551578d9c62b79"
  alt="A noisy teal line of token-weighted training loss against examples processed in the final run, from about 0.80 at example 200, up to about 0.85 near example 800 and again near 1,850, down to about 0.785 near 2,900, ending near 0.795 at example 3,912. Vertical lines mark checkpoints 6, 7 and 8."
  caption="Training loss in the final run, 100-update moving average. The whole curve sits between about 0.785 and 0.855. (technical report, Figure 6)."
/>

The loss curve is worth a look because of how little it does. Read off the figure, it starts near 0.80, climbs to about 0.85, and ends near 0.795. Sixteen hours moved the smoothed loss by about 0.005 net. The report is upfront that there is no validation curve for this stage and that the checkpoint was picked "on the basis of this comparison and additional scene-generation tests", by eye. I do not read that as a failure. Supervised loss on a teacher's code is not scene quality, and the report never claims it is. It does mean that the loss curve is not evidence for the checkpoint; the evaluation is.

## The dataset

**Measured**, on the validation split, which is the one file small enough to pull whole (6.96 MB, 213 rows). The columns are `id`, `title`, `level`, `domain`, `voxel` and `messages`, and the assistant message carries the reasoning trace in `reasoning_content` and the full HTML in `content`. There are 41 domains, from `anatomy_science` to `sports_play`. Levels run 1 to 10 with 21 or 22 rows each, so the validation split is stratified. 19 of the 213 are voxel scenes. User prompts are 50 words at the median (17 to 74 for the 10th to 90th percentile, 105 at most), and they read like a brief: "A stepped rectangular brass base holds a socket and a brass tube arm that bends forward through a smooth quarter-curve."

The answers are not small. The median HTML file is 37,566 characters and 1,351 lines; the largest is 68,602 characters. I counted tokens with the model's own `tokenizer.json`, split into the prompt, the reasoning trace and the code.

<TokenBudget />

Four numbers from that count are the ones I would keep.

1. **The mean row is 29,001 tokens** (median 26,363, longest 63,975): 113 of prompt, 16,066 of reasoning, 12,822 of code. The report's final run averages 112,903,599 over 3,912, which is 28,861. My validation mean is within 0.5% of the training run's, which is the sort of agreement you get when the two files describe the same population. **Reasoned:** the report's numbers are what the public dataset would give.
2. **The reasoning trace is longer than the code.** 1.25 times longer in total, and longer in 131 of 213 rows. The model is trained to think for about 16,000 tokens before it writes a scene.
3. **Complexity level tracks code length, not thinking length.** The correlation of level with code tokens is 0.70; with reasoning tokens it is 0.08. A level-1 scene already averages 8,292 tokens of code and a level-10 scene 17,387. "Complexity" is, to a first approximation, how much the teacher wrote.
4. **Almost nothing fits in the LoRA stage's window.** Only 13 of the 213 validation rows are 16,384 tokens or shorter, which squares with the report's 178 retained examples out of 3,933. The short stage trained on the short tail, about 4.5% of the data, and the report's discussion proposes repeating the whole specialisation with LoRA from the base to see whether it alone could do the job.

Static counts over the 213 HTML files say what the teacher's house style is. Every one imports three.js as `import * as THREE` from the cdnjs core URL (the exception is the one row with no system prompt, below). `MeshStandardMaterial`, `HemisphereLight` and `DirectionalLight` each appear in 212 files; 209 mention shadows; 160 build textures with `CanvasTexture` rather than loading an image; 157 use `Fog`; 150 run a `requestAnimationFrame` loop; 60 use `InstancedMesh`. All 213 write their own pointer-drag and wheel orbit controls, as the system prompt demands. A small model fine-tuned on this does not learn "Three.js". It learns this template, and the template is the reason the output is portable: one file, one import, no assets.

## Running the teacher's scenes

I wanted to know how much of the corpus a browser will actually draw. The harness is a short Node script driving the pre-installed Chromium through Playwright, with `--use-gl=angle --use-angle=swiftshader`, so WebGL is software-rendered on the CPU. For each of the 213 HTML files it loads the page at 960 by 540, aborts every network request except two, answers the cdnjs URL for three.js 0.160.0 with a local copy of the same file from npm, waits three seconds, and takes a screenshot. It records any uncaught page error, any console error or warning, whether a canvas exists, and whether the page's own error overlay is showing. Anything else a scene fetches is aborted and logged. Nothing ran outside the browser sandbox.

**Measured.** All 213 scenes drew a canvas. None threw an uncaught error, none raised a console error that stopped the script, and none was blank. 208 produced no console output at all. The other five are worth listing, because they are the whole of the failure surface:

- **Four printed a warning and drew anyway.** `scene-000004` is the one row without a system prompt; it loads three.js r128 as a classic script through `document.write` instead of the 0.160.0 module, and r128 warns that `thickness` is not a property of `MeshPhysicalMaterial`. I served r128 locally for it. `scene-001153` logs a NaN bounding-sphere radius from a geometry (the same class of warning the report notes on its alpine castle). `scene-001787` and `scene-002098` each print a "sigmaRadians is too large and will clip" warning from three.js's blur pass.
- **One was slow.** `scene-000675` still showed its "Initializing standards laboratory" overlay at the three-second mark, because software WebGL needs seconds for the first frame of a large scene. I re-ran it alone with a twenty-second wait and it drew. I count it as a pass with an asterisk. Three other scenes timed out on the first pass (one load, two screenshots, each over 30 seconds) while other jobs were saturating the machine; they ran cleanly in 10 to 20 seconds when retried with longer limits.

Everything else was uniform: all 19 voxel scenes, and every level from 1 to 10, with 19 to 22 clean of 21 or 22 at each level. Median load-plus-wait time was 9.4 seconds a scene under load, which says more about SwiftShader and a shared four-core box than about the scenes.

<Figure
  src="https://media.thesatyajit.com/ai/articles/qwen35-4b-3d-coder/render-sheet.jpg?v=f6ef60bd28854efd"
  alt="Six screenshots of teacher scenes in a light studio style: a six-cup pleated muffin tin, a seagrass patch with a pipefish, a folded-plate concrete roof with a title card, a hyperbolic paraboloid roof on four columns, a piped-edge cushion, and a scientific-instruments laboratory with labelled equipment."
  caption="Six of the 213 validation scenes as my harness rendered them (three.js 0.160.0, SwiftShader, 960 by 540, my screenshots): kitchen utensils, marine organisms, modern architecture, mathematical sculptures, kitchen utensils, scientific instruments."
/>


<SceneWall />

I want to be careful about what this number means. It measures the **teacher's** code, which MiniMax wrote and the dataset's author kept. It tells you the corpus is clean, which a fine-tuning dataset ought to be. It does not tell you what the 4B model produces. That is a different distribution, and the report's own figures show how different.

## What the report says about the student

**Reported.** Eight prompts (voxel ancient Rome, a Kyoto temple, a Venice canal, a Victorian library, an alpine castle, a container harbour, a regional airport and a pool table), each tried on the base model and on eight checkpoints, up to five attempts a case, 72 model-and-scene pairs. For checkpoint 8, the selected one:

- All eight cases met the author's acceptance criteria, after 18 calls in total.
- Two scenes were accepted on the first attempt with no edits; four, counting repairs.
- Six accepted outputs kept their original code. Two got "minimal repair", up to three local syntax or identifier fixes (Rome needed a `const` changed to `let`).
- Eight of the 18 calls "were interrupted by reasoning loops before delivering HTML".
- The alpine castle runs, with NaN geometry warnings. The container harbour took five attempts.

Put those beside my render pass. The teacher's code ran without an error nearly every time. The student ran cleanly on first try in two of eight cases, and needed 18 calls to cover eight scenes. Eight of those 18 calls never produced a page at all. If you treat the 18 calls as trials, ten produced HTML, which is 56%; that is a rough ratio, not a benchmark, because the report is explicit that attempts and stopping rules differed between versions and that "it is not a uniform first-attempt success benchmark".

The card says the same thing more gently: "one-shot success has improved substantially... Some scenes can still require another attempt or a correction."

<Figure
  src="https://media.thesatyajit.com/ai/articles/qwen35-4b-3d-coder/fig4.jpg?v=b75ed8534f789bd8"
  alt="Four screenshots. Top row, a regional airport: on the left a flat sky-blue frame with a lone grey tower, on the right an isometric diorama with a terminal, three aircraft at gates, a runway and trees. Bottom row, a pool table: on the left a nearly black frame with a green table top, on the right a lit bar room with a bow-front table, balls, stools and pendant lamps."
  caption="Base model (left) against checkpoint 8 (right) on the airport and pool-table prompts; selected attempts as labelled in the report. (technical report, Figure 4)."
/>

Figure 4 is the strongest evidence in the report, and it is a good one. The base model's airport is a sky and a tower; the fine-tune's is a diorama with a terminal, gates and aircraft. The cost is that the figure is a selection. The report chooses the first accepted output of up to five attempts, the acceptance test is "a recognisable scene and functioning controls", and the eight prompts "were used during development rather than as an independent blind test". It is a fair picture of what a good draw looks like and a poor estimate of how often you get one.

## Why a 4B model can do this at all (reasoned)

The scenes in this data are code, and code for a closed world: one library, one import, a fixed idiom for lights, fog, orbit controls and procedural textures. That is the kind of task where a small model that has seen 3,933 examples of the idiom can compose the parts it knows into a new building. It is also the kind where small models fail in a particular way. A 1,350-line file declares dozens of names, and a model that does not track them redeclares one. The card's "known issues" and the report's "repeated variable declarations" are that failure, and the report's own rescue (change `const` to `let`) is its mild form. In JavaScript, `const x` twice in one scope is a SyntaxError that stops the whole script, so one repeated name is a blank page, not a slightly worse scene.

The other cost is the trace. A model that thinks for 16,000 tokens before 12,800 tokens of code is a long decode. The evaluation allowed a 32,768-token reasoning budget and a 64,512-token generation limit. Eight of 18 calls hit a reasoning loop before delivering any HTML, and at that length a loop is expensive.

If you wanted to run it on a CPU, the arithmetic is unkind. The median training row is 26,363 tokens. At a hypothetical 3 tokens a second, which is the sort of speed a 4B model reaches on a few CPU threads (I did not measure this model), a median answer takes 2.4 hours, and a call that loops costs the same hours for nothing. This is the reason I did not run the model here.

## What this changes

For someone who wants to use this: take the Q8_0 GGUF, use the system prompt verbatim, set temperature 0.6 to 0.8 and top-k 20 to 40 as the card says, give it at least a 64K context (the cache is about 2 GiB), and plan on retries. The card's own advice is that some scenes need another attempt, and the report's numbers say to budget about two calls a usable scene (18 for 8), with eight of the 18 lost to reasoning loops. Keep a browser open next to it. The cheapest check of a result is whether the page loads and the console is empty, and the harness above does exactly that in a few seconds a scene.

For someone who wants to train on it: the dataset is the real contribution. It is clean (all 213 validation scenes drew, none threw), stratified, MIT, and long. Filter by `level` or token length before you do anything else, because the 16,384-token tail is 6% of the rows. The report's own list of open questions, LoRA from the base, FP16 against Q8_0, SFT against SFT plus RL, an evaluation on prompts that were not used in development, is the right one.

What I would not take from this is the headline. "A 4B model can generate interactive 3D scenes" is true in the sense that it does, sometimes, and the report says how often it did on eight prompts it had been developed against. That is a good start and an honest preview, which is what the card calls it.

## What I did not run

- **The model.** No weights were downloaded beyond headers, no generation was done, and none of the report's eight scenes was reproduced. Every statement about student output is the report's.
- **The base model on these prompts.** The comparison in Figure 4 is the author's.
- **A training run, or the loss.** The 49 TFLOP/s figure is arithmetic on reported totals.
- **The training split.** I measured the 213-row validation split, not the 3,933 training rows, so the 28,861 comparison is across two different sets of rows.
- **Visual quality.** My pass records errors and blank frames, not whether a scene looks like a canal lock.

Related reading on this site: [3DCodeVerse](/articles/3dcodeverse) counts a much larger corpus of executable 3D programs, [3D Model Anything](/articles/3d-model-anything) shows the same target (three.js code) written by an agent from a LiDAR scan, and [Neuron steering](/articles/neuron-steering) opens up the Qwen3.5-4B this model started from.