# PanSt3R: one set of queries for every view, so the chair keeps its id

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/panst3r
> date: 2026-10-16
> tags: 3d, computer-vision, point-cloud, gaussian-splatting, vision-transformers, open-vocabulary, benchmarks

Panoptic segmentation of one image is a solved-enough problem. Run
Mask2Former, get a mask per object and a class per mask. The trouble starts
with the second image. The chair that was instance 7 in frame one is instance
3 in frame two, because each frame was segmented alone and the ids mean
nothing outside it. To get "this chair, in 3D", somebody has to work out
which masks across which frames are the same object.

For about three years the answer has been to fit a scene to the images and
let the scene reconcile them. Panoptic Lifting, Contrastive Lift and PLGS
take posed images, run a 2D panoptic model per frame, and optimise a NeRF or a
Gaussian splat whose rendered labels agree with all those masks at once. It
works. It also needs camera poses, and the paper's own table puts it at two
to fifteen hours a scene.

[PanSt3R](https://arxiv.org/abs/2506.21348) (Žust et al., NAVER LABS Europe,
ICCV 2025) does it in one forward pass, with no poses. The code is at
[naver/panst3r](https://github.com/naver/panst3r). I cloned it, read it, pulled
the v2 checkpoint apart and ran it on two CPU threads. Below is how the
mechanism works, then what the numbers do and do not say.

<Figure
  src="https://media.thesatyajit.com/ai/articles/panst3r/fig2.jpg?v=8cca6e1152762625"
  alt="Two dense coloured point clouds of indoor rooms. Left: an office with a desk, chairs, a monitor and a whiteboard, each object a different flat colour. Right: a living room with a sofa, footstools and a TV unit, each object coloured as its own instance. Small black squares float in the air where the input cameras were."
  caption="PanSt3R's output: a point cloud from MUSt3R's 3D head, coloured by instance, with the input cameras shown as small frames. One forward pass, no camera parameters given (PanSt3R paper, teaser figure)."
/>

## The lineage, in one paragraph

[DUSt3R](https://arxiv.org/abs/2312.14132) showed that a ViT given two
unposed images can regress a 3D point for every pixel of both, in the first
camera's frame, with no calibration. [MASt3R](https://arxiv.org/abs/2406.09756)
added matching features. [MUSt3R](https://arxiv.org/abs/2503.01661) made it
multi-view: the network is symmetric, and its decoder keeps a memory of the
views it has already seen, so image 40 is decoded against tokens from images
1 to 39 rather than one partner. PanSt3R takes MUSt3R, freezes it, and bolts a
Mask2Former-style head on top. The same family of geometry models runs through
this site's [3D reconstruction roundup](/articles/3d-reconstruction-roundup-2)
and the [streaming reconstruction piece](/articles/da3-streaming-reconstruction);
PanSt3R is what happens when you ask that family for semantics too.

## The mechanism

<Figure
  src="https://media.thesatyajit.com/ai/articles/panst3r/fig1.png?v=a19bbb0727c8385a"
  alt="Architecture diagram in two halves. Left, extracting frame-wise features: RGB frames go into a frozen 2D DINOv2 encoder and a frozen 3D MUSt3R network. DINOv2 gives features E^D, MUSt3R gives encoder features E^M and decoder features D^M. These feed an MLP that makes frame tokens f, an upscaler that makes mask features F_i, and MUSt3R's 3D head. Right, panoptic prediction: learnable instance queries q0 and the frame tokens enter a mask transformer, which outputs refined queries q. Each refined query is multiplied with every frame's mask features F_0 to F_N to give instance masks, and compared against SigLIP text embeddings of class names to give class probabilities."
  caption="PanSt3R. Frozen DINOv2 and MUSt3R give per-frame tokens; a mask transformer refines one shared set of instance queries against the tokens of every frame; each query then yields a mask in every view and one class distribution (PanSt3R paper, Figure 1)."
/>

**Three token maps per view.** Every image is cut into 16 x 16 patches. For
each patch, PanSt3R takes three vectors: DINOv2-L's (1024 channels, the
"what is this" signal), MUSt3R's encoder output (1024, local geometry) and
MUSt3R's decoder output (768, geometry after attending to the memory of the
other views). Concatenated, that is 2,816 channels a patch, which is the
`input_dim: 2816 # 1024 + 1024 + 768` in `configs/base.yaml`. A small mixer
compresses them to 768-wide **frame tokens**, and an upscaler turns the same
features into **mask features** at half the input resolution. In v2 the mixer
is three transformer blocks feeding a LoftUp upscaler; v1's config has no
mixer and a PixelShuffle upscaler.

**One set of queries for the whole scene.** This is the idea. Mask2Former
segments an image with a fixed set of learned query vectors; each query
cross-attends to the image's tokens and comes out as "one object": a class
and a mask. PanSt3R keeps 200 queries (`num_queries=200`, six decoder layers)
but flattens the frame tokens of **all** views into one sequence before the
cross-attention. In `mask_transformer.py` the source sequence is built as
`BxCxNHW` in the code's own comment: views times patches, one sequence. So query 17 does not find a chair in
image 3. It finds a chair in the scene, looking at every view at once, and
then predicts its mask in each view by a dot product with that view's mask
features:

$$
\mathbf{M}_{j,n} = \mathrm{sigmoid}\big(\mathbf{F}_n \cdot \mathbf{q}^{M}_j\big)
$$

where $\mathbf{q}^M_j$ is query $j$'s mask embedding and $\mathbf{F}_n$ is
view $n$'s mask-feature map. Same query, same instance id, every view. There
is nothing to match afterwards. MUSt3R's decoder features are what make this
work: they already know that patch (12, 9) of view 3 and patch (4, 20) of
view 7 are the same bit of surface, so the query attending to both is
attending to one thing.

**Open-vocabulary classes.** The class of a query is the cosine similarity
between a projection of it and SigLIP text embeddings of class names
("This is a photo of chair."). That let the authors train on five datasets
with incompatible label sets (ScanNet++, Aria Synthetic Environments,
Infinigen, COCO and ADE20k), and it means the demo takes a class list at
run time.

**Keyframes for big scenes.** Two hundred images times 768 tokens is too much
for one cross-attention. So at test time a subset of keyframes, 50 in the
paper, is chosen by retrieval and farthest-point sampling. The queries are
decoded against the keyframes only, and every other frame just gets its mask
features dotted with the finished queries. The paper's Table 6 shows the
plateau: PQ 49.3 with 10 keyframes, 58.2 with 50, and still 58.2 with 100.

**Geometry for free.** MUSt3R's own 3D head still runs, so every pixel also
gets a 3D point. Colour the points by instance id and you have the teaser
figure.

## The merge: the paper's crucial step, and v2 turns it off

The network's raw output is 200 soft masks per view, overlapping, many of
them duplicates or junk. Something has to pick a subset and assign each pixel
to one of them.

Mask2Former's standard merge is local. Drop queries whose class score is
under a threshold. At each pixel, the query with the highest score times mask
value wins. Then keep a query only if it still owns a large enough fraction
of its own mask. On one image this works. Across views it has a failure the
paper names: a query that is strong in one view and absent in the others can
outbid the query that tracks the object everywhere, in that one view. The
global query loses part of its area, drops under the keep threshold, and
either vanishes (a hole in every other view) or survives beside the
duplicate (one chair, two ids).

PanSt3R replaces it with a quadratic unconstrained binary optimisation. Pick
a selection $u \in \{0,1\}^m$ over the proposals to maximise

$$
\sum_i u_i A_i \;-\; \lambda \sum_{i<j} u_i u_j O_{ij}
$$

where $A_i$ is the soft area of mask $i$ summed over every view and pixel,
and $O_{ij} = \sum_k \min(M_{i,k}, M_{j,k})$ is the soft overlap of two masks.
Coverage pays, overlap costs, $\lambda = 2$. The paper solves it with
simulated annealing, since QUBO is NP-hard in general. In the code
(`engine/postprocess.py`) that is plain NumPy: 20 restarts of 10,000 single-bit
flips, recomputing the full quadratic form after every flip. An incremental
update is written out and commented away. One more thing the code makes
plain: class scores do not enter the selection at all. `weight_from_masks`
has a `prob_weighted` switch and it defaults to `False`. QUBO picks masks on
shape alone; classes are read off afterwards.

The toy below runs all three merges exactly as the repo writes them, on two
views of sixteen pixels and six proposals I made up to show the failure. The
proposals are not model output. The rules, thresholds and $\lambda$ are the
repo's.

<MergeLab />

Under `standard v1` (mask threshold 0.5, keep threshold 0.8, one pass) the
view-1-only duplicate `q2` takes the chair in view 1. The cross-view chair
`q1` is left owning half its area, is dropped, and the chair in view 2
becomes six void pixels. Under `standard v2` (0.25, 0.5, two passes) `q1`
survives, so there are no holes, but the chair is `q2` in one view and `q1`
in the other. QUBO scores whole subsets: `q1`, `q3` and `q4` give 22.9;
adding `q2` brings 5.7 of area and costs 2 x 5.7 of overlap. Push $\lambda$ to
3 and the wall, which overlaps everything a little, is dropped too, and holes
come back. The penalty is a real knob.

The paper's Table 5 measures what QUBO is worth with LUDVIG uplifting: PQ on
Hypersim goes from 58.1 to 66.7 and on ScanNet from 60.2 to 67.3, with Replica
flat (60.8 against 60.7). Its Figure 4, below, shows the holes in the
"w/o QUBO" row.

<Figure
  src="https://media.thesatyajit.com/ai/articles/panst3r/fig3.png?v=bb57e39db05f2448"
  alt="A grid of five indoor ScanNet++ scenes in columns: a bathroom, a bedroom with chairs, an office with bookshelves and a door, a kitchen corner with a microwave, a living room with a plant. Rows: the rendered image, PanSt3R without QUBO with many black void blocks, PanSt3R with QUBO with far fewer, and PanSt3R plus LUDVIG with smooth, complete segments."
  caption="Novel-view panoptic segmentation on ScanNet++. The standard merge (second row) leaves black void blocks that QUBO (third row) fills; LUDVIG uplifting (bottom) smooths them across views (PanSt3R paper, Figure 4)."
/>

Now the twist. The README's v2 release (26 November 2025) says "Improved
standard post-processing (v2) → QUBO not necessary, faster inference". The
v2 checkpoint I downloaded carries `postprocess_default: standard_v2` and
`qubo_enabled: False` in its saved arguments. So the step the paper calls
"crucial" is off in the weights the README now lists second and describes as
"more accurate segmentation". The README's own PQ table gives v2 59.6, 61.8
and 63.0 on Hypersim, Replica and ScanNet, against v1's 56.5, 62.0 and 65.7.
That is 3.1 points better on Hypersim and 2.7 worse on ScanNet. Neither is
consistently better, and the README does not say which merge produced v2's
row. QUBO is still in the code and still in the demo's dropdown.

## What the numbers say

<LiftCost />

The PanLift benchmark (12 ScanNet scenes, 6 Hypersim, 7 Replica, 21 classes)
evaluates on held-out views, which is awkward for a method that segments the
images you give it. The paper's answer: build a vanilla 3DGS from the training
views, render the test views, and run PanSt3R on the renders. On that
protocol PanSt3R scores 56.5, 62.0 and 65.7 PQ, ahead of every lifting
method on Replica and ScanNet and behind three of them on Hypersim. On
ScanNet that is 65.7 against 58.9 for PanLift and 58.7 for PLGS. PanSt3R with LUDVIG, which lifts its labels into a 3DGS
and renders them, scores 66.3, 60.6 and 67.5. All reported, from Table 2.

**The speed headline needs a footnote.** The abstract says "orders of
magnitude faster than existing methods". Table 2 gives PanSt3R about 4.5
minutes a scene, against about 450 for PanLift and about 120 for PLGS. But
the PanSt3R row carries "+35", the time to build the 3DGS that renders the
test views, and the benchmark numbers cannot be had without it. With it,
PanSt3R is 39.5 minutes against PLGS's 120: three times faster, not a hundred
(reasoned, from the table). Against PanLift it is about eleven times. The
"orders of magnitude" is true when you already have the images you want
labelled, which is the normal case outside a benchmark. The paper states no
GPU for any of these times.

**The renders cost more PQ than you would guess.** Appendix Table 8 runs the
ScanNet++-only model three ways on the same test views. On the original
photos: 57.3 PQ. On 3DGS renders of them: 46.7. With LUDVIG: 54.8. So the
benchmark protocol that makes PanSt3R comparable to the lifting methods also
takes 10.6 points off it (reasoned), and direct prediction on real images
beats the LUDVIG pipeline that the paper headlines.

**On ScanNet++ the gap is large.** With 100 classes including small objects,
Table 3 has PanSt3R at 49.1 PQ in about 2.3 minutes, against 29.5 for PanLift
and 28.4 for Contrastive Lift at about 500 and 460 minutes, both run by the
authors on Mask2Former fine-tuned on ScanNet++. Most of the gap is on things,
not stuff: 45.8 against 15.6 thing-PQ. Reported.

**Both backbones earn their place.** Table 4, on a smaller 224-pixel model:
DINOv2 plus MUSt3R gives 50.4 PQ, MUSt3R alone 46.4, DINOv2 alone 35.7.
Geometry matters more than semantics here, which is the paper's thesis in one
row.

**Labelling a predicted point cloud directly is worse.** Appendix B tries the
obvious two-stage pipeline: reconstruct with MUSt3R, then run SGIFormer, a
point-cloud instance segmenter. On ScanNet++ it gets 1.8 AP50 on the MUSt3R
cloud, against 19.1 for PanSt3R and 33.2 for SGIFormer on the ground-truth
scan. Reported. Feed-forward point clouds are noisy in exactly the way 3D
segmenters were never trained for.

## What the checkpoint actually contains

The v2 checkpoint is 2,407,011,296 bytes, and its MD5 matches the README
(`6a7a47a2…`). Measured. Loaded, it holds 499.0M float32 weights:

| Part | Parameters | Trained by PanSt3R? |
|---|---|---|
| MUSt3R encoder (ViT-L) | 303.1M | no, frozen |
| MUSt3R decoder | 120.3M | no, frozen |
| Input mixer | 23.4M | yes |
| LoftUp upscaler | 2.6M | yes |
| Mask transformer | 49.5M | yes |

plus a pickled retrieval model for keyframe selection. What it does **not**
hold is DINOv2 or SigLIP. `DinoV2Encoder` calls
`AutoModel.from_pretrained('facebook/dinov2-large')` when the model is built,
and the text encoder does the same for `google/siglip-base-patch16-224`. Those
are 304.4M and 110.3M (text tower) more parameters, both Apache-2.0. So a
working PanSt3R is about 914M parameters, of which the PanSt3R-trained part
is 75.5M, about 8% (reasoned). The download is 2.4 GB plus another 2 GB from
Hugging Face on first run.

## Running it on two CPU threads

The repo ships a Gradio demo and no example images. I wanted the real
inference path without a web server, so I wrote a 60-line driver that does
what `get_reconstructed_scene` in `tools/demo_panst3r.py` does: the demo's own
image loader, `forward_inference_multi_ar` with every image as a keyframe,
then `panoptic_inference_v2` (the checkpoint's default) and, for comparison,
`panoptic_inference_qubo`. The input was the two-image Chateau pair from
CroCo's assets, the pair the DUSt3R family uses as its smoke test. Both are
224 x 224, so the loader centre-crops them to 4:3 and upsamples to 512 x 384:
a weak input, chosen because it is documented, not because it flatters.

Setup, so the numbers mean something: Python 3.12 venv, CPU PyTorch 2.7.0, a
shared four-core cloud VM with `nice -n 19` and two threads, no GPU. To fit
the disk budget I stored the weights as float16 and upcast them to float32 on
load, so the computation is float32 but the weights carry float16 rounding.
The `asmk` retrieval package is stubbed out because retrieval is not used with
two images. Measured:

| Step | Time |
|---|---|
| Build model, load weights | 8.8-11.5 s |
| Forward, 2 views at 512 x 384 | 47.6-73.6 s |
| `standard_v2` merge | 1.4 s |
| `qubo` merge | 41.4-52.1 s |
| Peak resident memory | 5.5 GB |

The ranges are two runs on a machine shared with five other jobs. The peak
includes about 1 GB of the memory-mapped checkpoint. The mask logits come out
at 200 x 192 x 256 per view: 200 queries at half the input resolution, as the
paper says.

The `standard_v2` merge kept 12 segments: 10 visible in view 1, 8 in view 2,
and 6 present in both under the same id. QUBO kept 10 segments on one run and
8 on the other, because the annealer starts from a random bit vector with no
seed. It is also the slow part on CPU: 20 x 10,000 flips, each a 200 x 200
quadratic form, costs about 40 seconds whatever the image count (reasoned
from the code, consistent with the timing).

<Figure
  src="https://media.thesatyajit.com/ai/articles/panst3r/cpu-run.jpg?v=1c935f633749902b"
  alt="Two photos of a château with trees on either side, from two viewpoints, above the same two photos overlaid with coloured segments. The building is green and labelled building #8 in both views, the sky is pink and labelled sky #9 in both, the trees are yellow-green and labelled tree #11 in both. View 1 also has grass #10, a car #3, a sidewalk #5, a pot #2, a plant #6 and a distant mountain #12; view 2 has a fence #7 and the plant #6."
  caption="Our CPU run of the v2 checkpoint on CroCo's two-image Chateau pair, standard_v2 merge, COCO plus ADE20k class names: building #8, sky #9 and tree #11 keep the same id in both views without any matching step. Two threads on a shared cloud VM; one scene, not a benchmark."
/>

What it shows is the mechanism, not the accuracy. Nothing in the pipeline
matched the building in view 1 to the building in view 2; one query found it
in both. "Mountain" on the distant hills and a "pot" by the car are the open
vocabulary doing its best on 252 class names. An upsampled 224-pixel pair is
not a fair test of a model trained at 512.

## The licence

Code and checkpoints are under the "PanSt3R Non-Commercial License", NAVER's
own text: non-commercial use only, no sublicensing, and it terminates if the
purpose stops being non-commercial. MUSt3R uses the same template. Two files,
`tools/demo_panst3r.py` and `tools/preprocess_scannetpp.py`, still carry a
"CC BY-NC-SA 4.0" header, which is also non-commercial but is a different
licence. The `NOTICE` adds that the checkpoints inherit the terms of the
training data: ScanNet++, Infinigen, COCO, ADE20k and Aria Synthetic
Environments, the last with written permission to distribute. The Mask2Former,
panopticapi and LoftUp pieces are MIT or BSD. If you build a product, this is
a reference implementation to learn from, not a dependency.

<RepoCard repo="naver/panst3r" />

## Where it sits

The lifting methods, and tools like [Carveout](/articles/carveout-3dgs-object-labels)
that label an existing splat with SAM 3, treat 2D segmentation as a given and
spend their effort reconciling it in 3D. PanSt3R moves the reconciliation
into the attention: the query that segments the chair has seen every view of
it. That buys a single pass, no poses, and ids that agree by construction.
It costs a hard ceiling of 200 instances a scene, a merge step that the
authors themselves have now changed twice, and a model that is 914M
parameters to deploy for 75.5M of new ones. [SpatialLM](/articles/spatiallm)
and [TrackEverything](/articles/trackeverything) are other answers to "what
is in this scan"; PanSt3R's is the most direct I have read.

What I would check next, with a GPU: v1 and v2 on the same scenes with each
merge, to see whether "QUBO not necessary" holds outside the README's three
numbers, and direct prediction on real held-out photos, where Table 8 says
the model is at its best and the benchmark never looks.