SpatialLM: a room scan in, a floor plan out as Python

mdjsonmcpcopy↗ai

2026-10-15 · 17 min · 3d · point-cloud · multimodal · qwen · small-models · object-detection · synthetic-data · benchmarks

Why read this

Notable· top 60%

How SpatialLM turns points into tokens and tokens into walls, with param counts from the weights and an editable floor plan running the repo's own parser.

• Original analysis
• Runs on a consumer GPU
• A lasting reference

·3D & spatial·Non-commercial licence·Practitioner model

How this was scored

Is it new?

1 of 3: An incremental tweak

Can I trust it?

2 of 3: Measures key facts from files, code or configs

Can I run it?

2 of 3: Open code or weights with real limits

Will I understand it?

2 of 3: Mechanism from first principles with figures

Can I act on it?

2 of 3: A concrete recipe, numbers or comparison

Will it last?

2 of 3: A reference for a year or more

Does it affect many?

1 of 3: A specialist community

Only here?

2 of 3: A teardown or measurement few others did

Score 62 of 100, ranked 281 of 577 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored

A post from @LearnOpenCV put SpatialLM back in front of me: "SpatialLM turned a bedroom video into structured text in 16s on a laptop GPU, with a half-billion-parameter LLM. Point encoder → adapter → LLM, one record per object. Bed size within 5 cm, 5 of 6 walls and the window line up. A few misses too. One room, one run." That is an honest post. It says one room and one run, and it says there were misses. I wanted to know what is inside the pipe it describes, and which of those numbers I can check.

The work is SpatialLM from Manycore Tech (the company behind the Kujiale interior-design platform) and HKUST, described in arXiv 2506.07491 and accepted at NeurIPS 2025. The code and weights have been public since March 2025; version 1.1, with a new point encoder, followed in June. The post links only the repository, and LearnOpenCV's site answers my sandbox with a Cloudflare challenge, so I could not read any longer write-up of their test. The post is all I have from it, and I treat its numbers as reported. Everything else below comes from the repository at commit 8913c44, the paper's v2 HTML, the model configs and the safetensors headers, which I range-read without downloading the weights. I did not run the model.

Why this is hard, and what SpatialLM bets on

A room scan is a few hundred thousand to a few million XYZRGB points. What you want from it is short: where the walls are, where the doors and windows cut through them, and a box for each bed, sofa and wardrobe. The classic answer is a specialist per output. RoomFormer predicts room polygons from a 2D density map. V-DETR regresses 3D boxes with a DETR head. Each is a bespoke network with a bespoke loss.

SceneScript (Meta, ECCV 2024) made the first bet: write the scene as a sequence of commands, and train a transformer decoder from scratch to emit them. SpatialLM makes the second bet: do not train a decoder at all. Take an off-the-shelf small LLM, put a point-cloud encoder in front of it the way LLaVA puts a vision encoder in front of an LLM, and ask it to write Python dataclass instances. The paper's argument for text is that it is readable, editable, extensible with new classes, and lets the model reuse whatever an LLM already knows about code.

expand

From points to tokens

The input pipeline in inference.py is short enough to read in full. The cloud must already be z-up and roughly axis-aligned with the walls, and in metres; EXAMPLE.md tells you to fix both yourself (rescale so the room is about 2.5 m tall, align with a vanishing-point or Manhattan-frame tool). Then:

1. cleanup_pcd voxel-downsamples to the model's grid and removes statistical outliers (15 neighbours, 2.0 sigma).
2. PositiveShift moves the cloud so every coordinate is non-negative.
3. GridSample snaps points to a grid of 32 / num_bins metres. The 1.1 configs set num_bins: 1280, so the grid is 2.5 cm (measured, from config.json). Grid coordinates are clipped to 1,279, so anything more than 32 m from the min corner piles up on the last cell. That is a lot of house, but not a warehouse.

The encoder is where version 1.1 changed. Version 1.0 used SceneScript's sparse 3D CNN; 1.1 uses Sonata, the self-supervised encoder-only variant of Point Transformer V3. I covered why PTv3's grid size is the unit everything inside it measures in, and what that costs when you change it, in the Invaria piece. Here the grid is fixed, and that matters for the token count.

expand

The config I read from Hugging Face matches the figure exactly: enc_depths [3, 3, 3, 12, 3], enc_channels [48, 96, 192, 384, 512], stride [2, 2, 2, 2], and four serialisation orders (z, z-trans, hilbert, hilbert-trans). Four stride-2 pools take the 2.5 cm grid to 40 cm. Sonata's code says the same thing a different way: reduced_grid_size = num_bins / 2**4, which is 80 cells across the 32 m world.

One point token is one occupied 40 cm cell. There is no Q-Former, no resampler and no fixed token budget. forward_point_cloud returns the features of every surviving coarse cell, point_proj maps each from 512 to Qwen's 896 dimensions, and the model splices the whole sequence between <|point_start|> and <|point_end|> in the prompt. A small room makes a short prompt and a big room a long one.

The paper's Table 3 measures exactly this trade. At a 5 cm finest grid the average is 123 tokens; at 2.5 cm, 510; at 1.25 cm, 2,000 (reported). Layout F1 rises from 84.6 to 87.4 between 5 cm and 2.5 cm, then falls again at finer grids, which the authors put down to sequences getting too long. The shipped models sit at the peak.

The right half of the widget is my arithmetic, not theirs. A room is mostly surfaces, so the number of occupied 40 cm cells is roughly the surface area divided by 0.16 m². A 3.2 × 4.2 × 2.8 m bedroom has about 68 m² of floor, ceiling and wall, which comes to about 427 cells (reasoned). That lands near the paper's 510 average, and it explains why the count tracks room size and voxel size rather than how many points the scanner produced.

What the 603.5M parameters are

I read the safetensors header of each checkpoint with two HTTP range requests and summed the tensor shapes by prefix (measured):

The 1.1 total matches the paper's 603.5M, and the Sonata share matches the 108.8M in its Table 2. So "a half-billion-parameter LLM" is right about the LLM; about 18% of the model is the encoder in front of it (reasoned). Hugging Face's own metadata puts SpatialLM1.1-Llama-1B at 1,345,883,776 parameters. The two 1.1 checkpoints and the 1.0 Qwen one are stored entirely as F32 (the 1.0 Llama one keeps its LLM in BF16), so the 1.1 Qwen file is about 2.4 GB on disk (reasoned: 603.5M × 4 bytes); inference.py loads it in bfloat16 and then sets the point backbone back to float32.

What the model writes

The prompt is the point tokens, a one-line task ("Detect walls, doors, windows, boxes."), and code_template.txt pasted in verbatim: four Python dataclasses whose fields are all int.

@dataclass
class Wall:
    ax: int
    ay: int
    az: int
    bx: int
    by: int
    bz: int
    height: int
    thickness: int
 
@dataclass
class Door:
    wall_id: str
    position_x: int
    position_y: int
    position_z: int
    width: int
    height: int
 
# Window has the same fields as Door
 
@dataclass
class Bbox:
    class: str
    position_x: int
    position_y: int
    position_z: int
    angle_z: int
    scale_x: int
    scale_y: int
    scale_z: int

The int is literal. entity.py (adapted from SceneScript's language code) normalises every value against a fixed range and truncates it into 1,280 bins: positions over 0 to 32 m, so 2.5 cm a bin; heights, widths and thicknesses over 0 to 25.6 m, 2 cm a bin; box sizes over 0 to 20 m, about 1.56 cm a bin; and yaw over ±6.2832 rad, about 0.56° a bin (all reasoned from NORMALIZATION_PRESET). The model never writes a decimal point. After generation, Layout.undiscretize_and_unnormalize maps bins back to metres and translate(min_extent) undoes the positive shift. That decoded form is what lands in the output .txt.

Here is the ground-truth layout of scene0001_01, a bedroom in the SpatialLM-Testset, in the decoded form, and the same room re-encoded the way the model would have to write it (measured, by running entity.py's arithmetic over the file):

wall_0=Wall(29,13,0,145,13,0,139,0)
wall_1=Wall(29,13,0,29,182,0,139,0)
wall_4=Wall(157,40,0,157,182,0,139,0)
door_0=Door(wall_4,157,159,48,45,120)
window_0=Window(wall_0,87,13,54,144,85)
bbox_4=Bbox(bed,74,107,20,799,130,142,64)
bbox_5=Bbox(carpet,108,98,0,640,122,185,0)

Read the bed: centre at bins 74 and 107, yaw 799 (about +90°), footprint 130 × 142 scale bins, which decodes to 2.03 × 2.22 m. The door is attached to wall_4 by name, which is how an autoregressive writer keeps doors on walls. The paper's Figure 4 shows RoomFormer, which predicts elements independently, leaving doors and windows floating off any wall.

Qwen2.5's pre-tokenizer splits every digit into its own token (its regex has a bare \p{N} alternative; measured, from tokenizer.json). So the output is mostly single digits and commas. I re-implemented the byte-level BPE from the released tokenizer.json and counted (measured, approximate since it is my implementation): the full bedroom script above, 15 records, is 454 tokens; the 45-record apartment in the paper's Figure 12 is 1,391. That is the decoding cost, and it is why max_new_tokens defaults to 4,096.

Two things in the widget are worth trying. Switch to bins and back: every coordinate moves by up to 2.5 cm, which is the quantisation the model works at. "Bed size within 5 cm" is a two-bin agreement (reasoned). Then delete the last number from a wall. Layout.from_str builds each wall from params[0:8]; a seven-number wall raises inside a bare try/except and is skipped without a word, and so is every door or window that names it. The widget reports those drops; the Python does not.

Training: the dataset is the model

The paper's study of how to wire this up has two useful findings. First, the encoder must be learned on raw points: pooling DINOv2 features into a fixed voxel grid, or sampling a fixed number of points, scored about 11 layout F1 and essentially zero on objects, against 79.4 to 84.6 layout F1 for learned encoders (Table 2, reported). Second, train everything at once. The common LLaVA recipe, align the projector first and keep the encoder frozen, was worse here. A single stage with encoder, projector and LLM all trainable won (Table 4, reported), which the authors read as point encoders not being general enough yet to freeze.

The training data is what makes it work. Manycore parsed designs from its own interior-design platform into 12,328 scenes with 54,778 rooms, 412,932 annotated objects over 59 categories, rendered RGBD along a simulated walk every 0.5 m, and fused the renders into point clouds (reported). The split is 11,328 scenes for training and 500 each for validation and test. The recipe: 4 epochs, batch 64, learning rate 1e-4, on 32 NVIDIA H20s for about a day (reported). Augmentation leans hard on noise: four tiers of jitter, up to a 4 m clip on the floaters a glass window throws, plus cuboid crops, elastic distortion and 90° rotations.

Table 5 and Table 6 show how much rides on that pre-training. Trained on Structured3D alone, SpatialLM scores 32.6 layout F1; trained on ScanNet alone, 2.9 object F1. Only pre-train-then-fine-tune is competitive (reported):

On layout it leads clearly. On objects it ties V-DETR at the loose threshold and trails it by 4.2 points at the strict one (reasoned), with 603.5M parameters against V-DETR's 79.4M. The paper says where: small objects such as pictures and sinks under 0.1 m³, and shower curtains, a class its dataset does not have. F1 rather than mAP is used because an autoregressive model produces no confidence scores to sweep, which also means you cannot trade precision for recall at inference time.

expand

From a phone video: what "zero-shot" means here

Video is where the LearnOpenCV demo lives, so it is the setting that matters most for anyone holding a phone. SpatialLM does not read video. Something else turns the video into a point cloud first; the README and the test set use MASt3R-SLAM, and EXAMPLE.md walks through SLAM3R as an alternative. If you have followed the streaming-reconstruction work, this is the consumer of those point maps: SpatialLM is only as good as the cloud it is handed, and it wants that cloud metric, z-up and wall-aligned.

The paper tests this on 107 room-tour videos reconstructed with MASt3R-SLAM, the same set released as SpatialLM-Testset. For the 1.1 Qwen model the README reports layout F1 @0.25 of 68.2 on walls, 47.4 on doors and 51.4 on windows, and 36.9 averaged over the 20 most common object classes, ranging from 95.2 on beds to 11.2 on cabinets and 9.7 on side tables (reported).

Two caveats on "zero-shot". The videos are "rendered from virtual scenes created by professional designers" (paper, appendix C.3), so the rooms come from the same kind of source as the training set; zero-shot means no MASt3R-SLAM reconstructions in training, not unseen kinds of rooms. And the ground truth is aligned by matching the SLAM trajectory to the true camera path, which the paper says itself costs F1 through scale and position error. Real phone footage has neither the designer rooms nor the trajectory alignment.

expand

That figure also shows a property that is useful and dangerous in the same breath. The paper notes that the model draws beds and nightstands reaching the floor where no points were seen, and fills in plausible areas from context. A box that completes an occluded bed is a good prior. It also means a confident wall can be a guess. Nothing in the output marks which.

The 16 seconds, and the one room

Back to the post. Bed within 5 cm, five of six walls and the window. Beds are the class the model scores best on in the zero-shot set, 95.2 F1, so a bed within two bins is what I would expect, and so is a missed wall (reported, then reasoned). A wall F1 of 68.2 means misses are normal.

The 16 seconds I cannot check, and the post does not say what they cover. inference.py covers outlier cleanup, the Sonata pass and generation; the video-to-cloud reconstruction is a separate program with its own runtime, and loading 2.4 GB of F32 weights is a cost of its own. The output side is something I can estimate. A bedroom like scene0001_01 is about 450 output tokens (measured above), so 16 seconds of pure generation would be about 35 ms a token (reasoned). That is plausible for Hugging Face generate streaming a 24-layer model on a laptop GPU, so "16 s" could be generation alone, or all of inference.py, or the whole pipeline on a short clip. Without LearnOpenCV's write-up I do not know which. The GPU is also not named in the post.

"One run" matters more than usual here. inference.py samples by default: do_sample=True, temperature 0.6, top-p 0.95, top-k 10, and a seed of -1 meaning none. Run it twice on the same cloud and you can get a different number of walls. If you are building on this, pass --seed, or run several samples and reconcile them.

Where it breaks

• The input contract is strict. Z-up, wall-aligned, metric, inside a 32 m cube. The repo hands alignment and scale to you. A 10° misalignment is not something the model was trained to absorb.
• Fixed vocabulary. 59 object classes from a furniture catalogue. Version 1.1 lets you ask for a subset (--category bed nightstand), but not for a class it never saw. The paper lists open vocabulary as future work.
• No confidences. F1, not mAP, and no score per box to threshold. You get a list of records and nothing about which ones the model believed.
• Silent parser drops. A malformed line, a seven-number wall or a door on an unknown wall vanishes in the try/except in layout.py. Count records before you trust a layout.
• A different stack. The Sonata path needs flash-attn (the encoder asserts it is installed) plus spconv, torch-scatter and the CUDA 12.4 toolchain the README pins; version 1.0 needs TorchSparse instead.

Licence

This is the part to read before you build a product on it. The two Qwen checkpoints on Hugging Face carry license: cc-by-nc-4.0 on their model cards and the two Llama ones say llama3.2; the README states that the 1.1 point-encoder weights are CC-BY-NC-4.0 regardless of which LLM sits behind them, and so are the dataset and test set. The repository's own LICENSE.txt is the Llama 3.2 Community License Agreement, even though most of the code is Qwen-side or Pointcept-derived (Apache 2.0, per the README). For commercial use, the weights are off the table (measured, from the cards and LICENSE.txt). The architecture and recipe are not, if you have your own designer-grade synthetic rooms. That dataset is what the paper actually shows matters.

What I take from it

SpatialLM's real contribution is not the 0.5B model. It shows that the LLaVA recipe, encoder plus MLP plus off-the-shelf LLM, all trained in one stage, reaches state-of-the-art layout estimation once you can feed it 12,000 clean synthetic scenes. Text output brings real benefits: openings are bound to walls by name, the result is editable, and adding a class means adding a word. It also brings real costs: hundreds of digit tokens per room, no confidences, sampling noise, and a parser that drops what it cannot read.

For a scan-to-BIM pipeline, the shape that fits is a draft generator. Run it on a cleaned, aligned cloud, take the walls and openings as a starting floor plan, and verify them against the points before anything goes into a model like the ones in the KOLC+ piece. For object-level work, the Carveout approach to labelling splats and the MLLMs in the first 3D roundup answer different questions; SpatialLM answers "where are the walls" better than any of them, inside a 32 m box, for non-commercial use.