PanSt3R: one set of queries for every view, so the chair keeps its id

Panoptic segmentation of one image is a solved-enough problem. Run Mask2Former, get a mask per object and a class per mask. The trouble starts with the second image. The chair that was instance 7 in frame one is instance 3 in frame two, because each frame was segmented alone and the ids mean nothing outside it. To get "this chair, in 3D", somebody has to work out which masks across which frames are the same object.

For about three years the answer has been to fit a scene to the images and let the scene reconcile them. Panoptic Lifting, Contrastive Lift and PLGS take posed images, run a 2D panoptic model per frame, and optimise a NeRF or a Gaussian splat whose rendered labels agree with all those masks at once. It works. It also needs camera poses, and the paper's own table puts it at two to fifteen hours a scene.

PanSt3R (Žust et al., NAVER LABS Europe, ICCV 2025) does it in one forward pass, with no poses. The code is at naver/panst3r. I cloned it, read it, pulled the v2 checkpoint apart and ran it on two CPU threads. Below is how the mechanism works, then what the numbers do and do not say.

expand

The lineage, in one paragraph

DUSt3R showed that a ViT given two unposed images can regress a 3D point for every pixel of both, in the first camera's frame, with no calibration. MASt3R added matching features. MUSt3R made it multi-view: the network is symmetric, and its decoder keeps a memory of the views it has already seen, so image 40 is decoded against tokens from images 1 to 39 rather than one partner. PanSt3R takes MUSt3R, freezes it, and bolts a Mask2Former-style head on top. The same family of geometry models runs through this site's 3D reconstruction roundup and the streaming reconstruction piece; PanSt3R is what happens when you ask that family for semantics too.

The mechanism

expand

Three token maps per view. Every image is cut into 16 x 16 patches. For each patch, PanSt3R takes three vectors: DINOv2-L's (1024 channels, the "what is this" signal), MUSt3R's encoder output (1024, local geometry) and MUSt3R's decoder output (768, geometry after attending to the memory of the other views). Concatenated, that is 2,816 channels a patch, which is the input_dim: 2816 # 1024 + 1024 + 768 in configs/base.yaml. A small mixer compresses them to 768-wide frame tokens, and an upscaler turns the same features into mask features at half the input resolution. In v2 the mixer is three transformer blocks feeding a LoftUp upscaler; v1's config has no mixer and a PixelShuffle upscaler.

One set of queries for the whole scene. This is the idea. Mask2Former segments an image with a fixed set of learned query vectors; each query cross-attends to the image's tokens and comes out as "one object": a class and a mask. PanSt3R keeps 200 queries (num_queries=200, six decoder layers) but flattens the frame tokens of all views into one sequence before the cross-attention. In mask_transformer.py the source sequence is built as BxCxNHW in the code's own comment: views times patches, one sequence. So query 17 does not find a chair in image 3. It finds a chair in the scene, looking at every view at once, and then predicts its mask in each view by a dot product with that view's mask features:

where $\mathbf{q}^M_j$ is query $j$'s mask embedding and $\mathbf{F}_n$ is view $n$'s mask-feature map. Same query, same instance id, every view. There is nothing to match afterwards. MUSt3R's decoder features are what make this work: they already know that patch (12, 9) of view 3 and patch (4, 20) of view 7 are the same bit of surface, so the query attending to both is attending to one thing.

Open-vocabulary classes. The class of a query is the cosine similarity between a projection of it and SigLIP text embeddings of class names ("This is a photo of chair."). That let the authors train on five datasets with incompatible label sets (ScanNet++, Aria Synthetic Environments, Infinigen, COCO and ADE20k), and it means the demo takes a class list at run time.

Keyframes for big scenes. Two hundred images times 768 tokens is too much for one cross-attention. So at test time a subset of keyframes, 50 in the paper, is chosen by retrieval and farthest-point sampling. The queries are decoded against the keyframes only, and every other frame just gets its mask features dotted with the finished queries. The paper's Table 6 shows the plateau: PQ 49.3 with 10 keyframes, 58.2 with 50, and still 58.2 with 100.

Geometry for free. MUSt3R's own 3D head still runs, so every pixel also gets a 3D point. Colour the points by instance id and you have the teaser figure.

The merge: the paper's crucial step, and v2 turns it off

The network's raw output is 200 soft masks per view, overlapping, many of them duplicates or junk. Something has to pick a subset and assign each pixel to one of them.

Mask2Former's standard merge is local. Drop queries whose class score is under a threshold. At each pixel, the query with the highest score times mask value wins. Then keep a query only if it still owns a large enough fraction of its own mask. On one image this works. Across views it has a failure the paper names: a query that is strong in one view and absent in the others can outbid the query that tracks the object everywhere, in that one view. The global query loses part of its area, drops under the keep threshold, and either vanishes (a hole in every other view) or survives beside the duplicate (one chair, two ids).

PanSt3R replaces it with a quadratic unconstrained binary optimisation. Pick a selection $u \in \{0,1\}^m$ over the proposals to maximise

where $A_i$ is the soft area of mask $i$ summed over every view and pixel, and $O_{ij} = \sum_k \min(M_{i,k}, M_{j,k})$ is the soft overlap of two masks. Coverage pays, overlap costs, $\lambda = 2$. The paper solves it with simulated annealing, since QUBO is NP-hard in general. In the code (engine/postprocess.py) that is plain NumPy: 20 restarts of 10,000 single-bit flips, recomputing the full quadratic form after every flip. An incremental update is written out and commented away. One more thing the code makes plain: class scores do not enter the selection at all. weight_from_masks has a prob_weighted switch and it defaults to False. QUBO picks masks on shape alone; classes are read off afterwards.

The toy below runs all three merges exactly as the repo writes them, on two views of sixteen pixels and six proposals I made up to show the failure. The proposals are not model output. The rules, thresholds and $\lambda$ are the repo's.

Under standard v1 (mask threshold 0.5, keep threshold 0.8, one pass) the view-1-only duplicate q2 takes the chair in view 1. The cross-view chair q1 is left owning half its area, is dropped, and the chair in view 2 becomes six void pixels. Under standard v2 (0.25, 0.5, two passes) q1 survives, so there are no holes, but the chair is q2 in one view and q1 in the other. QUBO scores whole subsets: q1, q3 and q4 give 22.9; adding q2 brings 5.7 of area and costs 2 x 5.7 of overlap. Push $\lambda$ to 3 and the wall, which overlaps everything a little, is dropped too, and holes come back. The penalty is a real knob.

The paper's Table 5 measures what QUBO is worth with LUDVIG uplifting: PQ on Hypersim goes from 58.1 to 66.7 and on ScanNet from 60.2 to 67.3, with Replica flat (60.8 against 60.7). Its Figure 4, below, shows the holes in the "w/o QUBO" row.

expand

Now the twist. The README's v2 release (26 November 2025) says "Improved standard post-processing (v2) → QUBO not necessary, faster inference". The v2 checkpoint I downloaded carries postprocess_default: standard_v2 and qubo_enabled: False in its saved arguments. So the step the paper calls "crucial" is off in the weights the README now lists second and describes as "more accurate segmentation". The README's own PQ table gives v2 59.6, 61.8 and 63.0 on Hypersim, Replica and ScanNet, against v1's 56.5, 62.0 and 65.7. That is 3.1 points better on Hypersim and 2.7 worse on ScanNet. Neither is consistently better, and the README does not say which merge produced v2's row. QUBO is still in the code and still in the demo's dropdown.

What the numbers say

The PanLift benchmark (12 ScanNet scenes, 6 Hypersim, 7 Replica, 21 classes) evaluates on held-out views, which is awkward for a method that segments the images you give it. The paper's answer: build a vanilla 3DGS from the training views, render the test views, and run PanSt3R on the renders. On that protocol PanSt3R scores 56.5, 62.0 and 65.7 PQ, ahead of every lifting method on Replica and ScanNet and behind three of them on Hypersim. On ScanNet that is 65.7 against 58.9 for PanLift and 58.7 for PLGS. PanSt3R with LUDVIG, which lifts its labels into a 3DGS and renders them, scores 66.3, 60.6 and 67.5. All reported, from Table 2.

The speed headline needs a footnote. The abstract says "orders of magnitude faster than existing methods". Table 2 gives PanSt3R about 4.5 minutes a scene, against about 450 for PanLift and about 120 for PLGS. But the PanSt3R row carries "+35", the time to build the 3DGS that renders the test views, and the benchmark numbers cannot be had without it. With it, PanSt3R is 39.5 minutes against PLGS's 120: three times faster, not a hundred (reasoned, from the table). Against PanLift it is about eleven times. The "orders of magnitude" is true when you already have the images you want labelled, which is the normal case outside a benchmark. The paper states no GPU for any of these times.

The renders cost more PQ than you would guess. Appendix Table 8 runs the ScanNet++-only model three ways on the same test views. On the original photos: 57.3 PQ. On 3DGS renders of them: 46.7. With LUDVIG: 54.8. So the benchmark protocol that makes PanSt3R comparable to the lifting methods also takes 10.6 points off it (reasoned), and direct prediction on real images beats the LUDVIG pipeline that the paper headlines.

On ScanNet++ the gap is large. With 100 classes including small objects, Table 3 has PanSt3R at 49.1 PQ in about 2.3 minutes, against 29.5 for PanLift and 28.4 for Contrastive Lift at about 500 and 460 minutes, both run by the authors on Mask2Former fine-tuned on ScanNet++. Most of the gap is on things, not stuff: 45.8 against 15.6 thing-PQ. Reported.

Both backbones earn their place. Table 4, on a smaller 224-pixel model: DINOv2 plus MUSt3R gives 50.4 PQ, MUSt3R alone 46.4, DINOv2 alone 35.7. Geometry matters more than semantics here, which is the paper's thesis in one row.

Labelling a predicted point cloud directly is worse. Appendix B tries the obvious two-stage pipeline: reconstruct with MUSt3R, then run SGIFormer, a point-cloud instance segmenter. On ScanNet++ it gets 1.8 AP50 on the MUSt3R cloud, against 19.1 for PanSt3R and 33.2 for SGIFormer on the ground-truth scan. Reported. Feed-forward point clouds are noisy in exactly the way 3D segmenters were never trained for.

What the checkpoint actually contains

The v2 checkpoint is 2,407,011,296 bytes, and its MD5 matches the README (6a7a47a2…). Measured. Loaded, it holds 499.0M float32 weights:

plus a pickled retrieval model for keyframe selection. What it does not hold is DINOv2 or SigLIP. DinoV2Encoder calls AutoModel.from_pretrained('facebook/dinov2-large') when the model is built, and the text encoder does the same for google/siglip-base-patch16-224. Those are 304.4M and 110.3M (text tower) more parameters, both Apache-2.0. So a working PanSt3R is about 914M parameters, of which the PanSt3R-trained part is 75.5M, about 8% (reasoned). The download is 2.4 GB plus another 2 GB from Hugging Face on first run.

Running it on two CPU threads

The repo ships a Gradio demo and no example images. I wanted the real inference path without a web server, so I wrote a 60-line driver that does what get_reconstructed_scene in tools/demo_panst3r.py does: the demo's own image loader, forward_inference_multi_ar with every image as a keyframe, then panoptic_inference_v2 (the checkpoint's default) and, for comparison, panoptic_inference_qubo. The input was the two-image Chateau pair from CroCo's assets, the pair the DUSt3R family uses as its smoke test. Both are 224 x 224, so the loader centre-crops them to 4:3 and upsamples to 512 x 384: a weak input, chosen because it is documented, not because it flatters.

Setup, so the numbers mean something: Python 3.12 venv, CPU PyTorch 2.7.0, a shared four-core cloud VM with nice -n 19 and two threads, no GPU. To fit the disk budget I stored the weights as float16 and upcast them to float32 on load, so the computation is float32 but the weights carry float16 rounding. The asmk retrieval package is stubbed out because retrieval is not used with two images. Measured:

The ranges are two runs on a machine shared with five other jobs. The peak includes about 1 GB of the memory-mapped checkpoint. The mask logits come out at 200 x 192 x 256 per view: 200 queries at half the input resolution, as the paper says.

The standard_v2 merge kept 12 segments: 10 visible in view 1, 8 in view 2, and 6 present in both under the same id. QUBO kept 10 segments on one run and 8 on the other, because the annealer starts from a random bit vector with no seed. It is also the slow part on CPU: 20 x 10,000 flips, each a 200 x 200 quadratic form, costs about 40 seconds whatever the image count (reasoned from the code, consistent with the timing).

expand

What it shows is the mechanism, not the accuracy. Nothing in the pipeline matched the building in view 1 to the building in view 2; one query found it in both. "Mountain" on the distant hills and a "pot" by the car are the open vocabulary doing its best on 252 class names. An upsampled 224-pixel pair is not a fair test of a model trained at 512.

The licence

Code and checkpoints are under the "PanSt3R Non-Commercial License", NAVER's own text: non-commercial use only, no sublicensing, and it terminates if the purpose stops being non-commercial. MUSt3R uses the same template. Two files, tools/demo_panst3r.py and tools/preprocess_scannetpp.py, still carry a "CC BY-NC-SA 4.0" header, which is also non-commercial but is a different licence. The NOTICE adds that the checkpoints inherit the terms of the training data: ScanNet++, Infinigen, COCO, ADE20k and Aria Synthetic Environments, the last with written permission to distribute. The Mask2Former, panopticapi and LoftUp pieces are MIT or BSD. If you build a product, this is a reference implementation to learn from, not a dependency.

Where it sits

The lifting methods, and tools like Carveout that label an existing splat with SAM 3, treat 2D segmentation as a given and spend their effort reconciling it in 3D. PanSt3R moves the reconciliation into the attention: the query that segments the chair has seen every view of it. That buys a single pass, no poses, and ids that agree by construction. It costs a hard ceiling of 200 instances a scene, a merge step that the authors themselves have now changed twice, and a model that is 914M parameters to deploy for 75.5M of new ones. SpatialLM and TrackEverything are other answers to "what is in this scan"; PanSt3R's is the most direct I have read.

What I would check next, with a GPU: v1 and v2 on the same scenes with each merge, to see whether "QUBO not necessary" holds outside the README's three numbers, and direct prediction on real held-out photos, where Table 8 says the model is at its best and the benchmark never looks.