InfiniSplat v2: the points stay where Depth Anything 3 put them, and opacity does the cleaning

The post that announced it, from Sida Peng on 16 October 2026, says the interesting part in one line: InfiniSplat v2 "can produce high-quality 3D Gaussian Splatting, even when the base geometry model yields noisy point clouds."

The obvious reading is that v2 repairs the point cloud: pulls stray points back onto surfaces, learns offsets, denoises geometry. It does none of that. The release code puts every Gaussian centre exactly where a frozen Depth Anything 3 depth pixel back-projects, and the paper's own ablation shows that letting the network move points makes the renders worse. What v2 learns is everything except position, and the attribute doing the cleaning is opacity.

I read the paper (arXiv:2610.20341, 14 October 2026), the v1 paper (arXiv:2608.02437), the release at zju3dv/InfiniSplat (commit a4d3f8b, 15 October), the Hugging Face Space's source, and the header of the 6.60 GB checkpoint at Edisoneh/InfiniSplat-v2 through an HTTP range request. I did not run the model: it needs an NVIDIA GPU and this machine has none. Numbers below are measured (I read them from a file), reported (the authors' figure, not re-run) or reasoned (my arithmetic on the other two).

expand

The base geometry model, and why its points are noisy

A geometry foundation model (DUSt3R, VGGT, π³, Depth Anything 3) takes a set of images and predicts, per view, a depth map and a camera. Back-project every pixel through its camera and you get a point cloud in one frame. The site has covered this family from several sides: DA3 in a ROS 2 node, streaming reconstruction on DA3-style backbones, and two roundups of feed-forward reconstruction papers and the classical geometry being bolted back on.

In v2 the "base geometry model" is concretely DA3NESTED-GIANT-LARGE-1.1 (reported, Appendix A.2), and the release's config.json instantiates the DA3 network with a vitg backbone whose features are tapped at layers 19, 27, 33 and 39 (measured). It runs at a long side of 518 pixels by default, with 2 to 32 views on the command line and up to 50 with --max-views (measured, cli.py and the README).

Those point clouds are good on average and bad at the edges. Per-view depth maps disagree slightly, so the same wall arrives as several sheets. Depth networks smooth across silhouettes, so pixels at an object boundary land in mid-air between the object and what is behind it. A lidar does not have this failure in the same form (its range is a time of flight, not an inference, as the Hesai Kosmo piece covers), but a camera-only pipeline lives with it. Render the raw cloud from a new angle and those flying points become the haze you see in the left half of each pair above.

What v2 actually predicts

expand

The whole inference path is about 1,460 lines of Python in InfiniSplat_v2/src/infinisplat/ (measured: 1,462 with wc -l). Reading it top to bottom:

Two copies of DA3. model.py builds the DA3 network twice from the same config. The geometry branch is frozen and supplies depth and cameras. The appearance branch keeps only its backbone tokens (its head is replaced with nn.Identity() in the released format) and feeds four layers of 1,536-wide features through four Linear(1536, 2048) projections into a DPT-style decoder (measured). The paper's reason for two branches is its first finding: jointly fine-tuning geometry and attributes degrades geometry, because the rendering loss is happy to move points to cover for bad colours.

One Gaussian per pixel, with no position output. The Gaussian head is small: a 3×3 convolution from 128 to 256 channels, a ReLU, and a 1×1 convolution to 11 channels (measured, gaussians.py, confirmed by the gaussian_head.layers.2.weight [11, 256, 1, 1] tensor in the checkpoint header). The 11 channels split as 4 for a rotation quaternion, 3 for log-scale, 1 for opacity and 3 for a colour residual:

# InfiniSplat_v2/src/infinisplat/gaussians.py (release a4d3f8b)
rotation, scale, opacity, sh = torch.split(raw, [4, 3, 1, 3], -1)
rgb = images[0].permute(0, 2, 3, 1).reshape(-1, 3).float()
return dict(quats=rotation / (rotation.norm(dim=-1, keepdim=True) + 1e-8),
            scales=scale.exp().clamp_max(.3), opacities=opacity[:, 0].sigmoid(),
            sh=(((rgb - .5) / SH_C0).to(sh.dtype) + sh)[:, None], base_rgb=rgb)

Three things in those lines. Colour starts at the input pixel's own RGB and the network predicts only a correction to the zeroth-order spherical harmonic, so v2 has no view-dependent colour at all (the site's piece on splat colour covers what higher-order SH costs). Scale is capped at 0.3 in the normalised scene unit. And there is no means output: back in model.py, the centres are assigned from depth_to_world_coords_points(depth, poses, K), the frozen depth unprojected through the cameras, after the head has run.

Normalise, merge, prune. Depth and camera translations are divided by the mean distance of all back-projected points from the reference camera, which the paper calls Mean Radial Distance Normalization:

where $\mathbf{x}_j$ is a back-projected point and $\hat{D}_i$ the predicted depth of view $i$. So the scene's unit is "average distance to the stuff", and the 0.3 scale cap means no Gaussian is wider than about a third of that.

Then every view's Gaussians, 518 × 392 of them for a 4:3 image at the default size (203,056 per view, so about 6.5 million for 32 views; reasoned), go through merge(). It snaps centres to a voxel grid whose side is 1.4 times the median Gaussian scale, averages every attribute inside each occupied voxel, and drops anything whose averaged opacity is under 0.01 (measured). The export keeps every survivor; there is no count cap.

One mismatch I could not resolve: Appendix A.2 says voxel merging "uses a side length of 0.004 in the scale-normalized reconstruction frame", while the release computes it per scene as median * 1.4, and pipeline.py refuses to load a config whose voxel_multiplier is not 1.4 (measured). The two can agree for a typical scene, but they are different rules.

Why opacity is the denoiser

Positions are frozen, so the only way the network can make a flying point stop hurting the render is to make it transparent. That is the paper's third finding, and the one the tweet is about: trained with image losses only, the attribute branch learns to give low opacity to points that are inconsistent across views.

The authors test it on held-out Hypersim scenes with labels that never look at opacity: a point is inconsistent if it sits in free space that ground-truth depth says at least two views looked straight through, and matches no observed surface depth. That labelled 36,125 points. At the 0.01 threshold, pruning removes 51.8% of the inconsistent points but only 7.5% of all points, and keeps 93.3% of the surface-consistent ones (reported, Section 4.3 and Appendix B.4).

expand

The toy below is that mechanism in a 2D side view. Three cameras look at a wall and a box. Their depth is wrong the way depth networks are wrong: the two rays just past each edge of the box bleed into mid-air, and the slider adds noise and occasional spikes. Nothing moves a point. Opacity comes from a cross-view test (does another camera's predicted depth say it saw through this spot?), which is my hand-written stand-in for the learned head; the merge rule (1.4 times the median scale) and the 0.01 prune follow gaussians.py.

Two things fall out of playing with it. At zero noise the edge bleed alone makes 4 of the 120 points inconsistent, and the test removes all 4: the flying-pixel problem exists before any random noise does. And as the noise grows, most damaged points become uncertain rather than inconsistent: they are behind or slightly off the true surface, where no camera has seen through them, and opacity has no reason to remove them. That is the toy's version of the paper's own limitation, "fixed centers cannot correct severe errors in predicted geometry." Opacity removes what is in front of a surface. A surface in the wrong place stays in the wrong place.

The ablation backs the decision to freeze positions (reported, Table 2, ETH3D and DTU, before refinement):

Freezing geometry cuts LPIPS by 16.6% on ETH3D and 23.2% on DTU against fine-tuning it, and ground-truth cameras at training time cut DTU LPIPS by 36.1% (reported, and reasoned: all three reproduce from the table to the stated digit). Learned offsets, the thing you would add first if you wanted to "clean" a point cloud, are worse than no offsets on both datasets.

The other two training changes are about what the loss can see. Using predicted cameras to build points but ground-truth cameras to render targets leaves a pose residual the network tries to paint over; v2 back-projects with ground-truth cameras, rescaled by an Umeyama fit to the normalised predicted trajectory. And target views near the inputs leave a Gaussian's thickness unconstrained, so 70% of samples draw targets from the half of candidate views farthest in optical-axis angle from any input, which moves the mean minimum angle from 12.34° to 17.97° (reported, Appendix A.1).

What changed from v1

The first InfiniSplat (arXiv:2608.02437, SIGGRAPH Asia 2026) is a different model for a different input: one RGB image, optionally with a lidar depth map. The image-only variant takes its depth from Depth Pro, the lidar variant from ZJU3DV's InfiniDepth (measured, the two encoders in InfiniSplat_v1/src/model/encoder/).

expand

The direction is a little surprising. v1's paper argues against pixel-aligned Gaussians, which it says "remain weakly coupled to underlying scene surfaces", and v2 goes back to them because a multi-view geometry model already gives coherent surfaces and the job left is attributes. v2's limitations section concedes the cost: "Pixel-aligned prediction also produces redundancy not fully removed by voxel merging." The idea that survives from v1 is the training signal, supervision from views far from the input.

The checkpoint, read from its header

The safetensors header is 145,360 bytes and lists 1,160 stored tensors, all float32, totalling 1,649,014,290 parameters in a 6,596,202,528-byte file (measured). config.json adds 565 tensor_aliases: names the loader fills by pointing at an already-stored tensor (measured).

The aliases are the interesting part. Every tensor of geometry blocks 0 to 29 points at the matching appearance block; only blocks 30 to 39 are stored twice (measured). That is the paper's "only the last K Transformer blocks of the attribute branch are fine-tuned", with K = 10, written in the weights: the first 30 blocks are byte-identical between the frozen and trainable copies, so configure_inference() checks them with torch.equal and runs them once (shared_prefix_blocks). The aliases cover 967.7M parameters, so the network as constructed has about 2.62 billion (reasoned: 1,649.0M + 967.7M).

There are no refiner tensors. The paper's best rows, marked Ours∗, add three steps of a learned refiner (a KNN Transformer over the Gaussians, fed rendering gradients from the input views). The checkpoint's top-level modules are the two DA3 branches, the projections, the feature decoder and the Gaussian head, and model.forward ends at merge() (measured). The Space imports the same model.py (measured, identical file). What you can run is the feed-forward model, the row labelled Ours.

"Unbounded", and what it rests on

The tweet says "unbounded free-viewpoint rendering"; the paper does not use the word. In the code it means the absence of a box: there is no near or far plane, no scene AABB and no background model. The voxel grid is sized from the points' own minimum and maximum, with a guard that raises an error rather than overflow 64-bit cell indices (measured, gaussians.py). Scale comes from the mean point radius, not from a preset. So a street and a tabletop go through identical code.

What it does not mean is unlimited input. The CLI caps a single pass at 50 views. The paper's KITTI street scene, a sequence of over 1,000 frames, was built "using sliding windows" (reported, Appendix B.6), and no windowing or stitching code is in the release (measured). For long sequences, the bounded-memory streaming methods in this roundup are the closer fit today. The output is an SH0 .ply in normalised units, which PlayCanvas's splat-transform turns into a standalone HTML viewer (the same tool behind the SOG format).

Checking the headline numbers

The evaluation is six datasets (DL3DV, ETH3D, NRGBD, Mip-NeRF 360, RE10K, DTU), 4, 8 and 16 input views with four held-out targets each, and LPIPS and DISTS as primary metrics "following SHARP" (reported). The same trained model is scored with and without poses.

expand

Six-dataset means with poses supplied (reported, Table 1 and Appendix Table 4; lower LPIPS and DISTS are better, higher PSNR is better):

From Table 1, the six-dataset mean LPIPS with poses is 0.2472 against ReSplat's 0.3333, and mean DISTS 0.1163 against WorldMirror 2.0's 0.1734 (reported). Those give the abstract's 25.8% and 32.9% exactly (reasoned). Without poses, 0.3026 against WorldMirror 2.0's 0.3757 and 0.1341 against 0.1935, the 19.5% and 30.7% the text claims (reasoned). I also recomputed each "Mean" column as the plain average of its six datasets, and it matches for every pose-conditioned method to four decimals, with one digit of rounding on DepthSplat (reasoned).

Per cell, the feed-forward model beats the best baseline in 22 of the 24 dataset-by-metric cells. The two it loses: pose-free DTU LPIPS (0.3898 against WorldMirror 2.0's 0.3812) and pose-conditioned RE10K LPIPS (0.1202 against ReSplat's 0.1191) (reasoned, from Table 1).

The fine print is the metric choice. In Appendix Table 4, ReSplat's pose-conditioned mean PSNR is 20.91 dB and SSIM 0.7117; v2's feed-forward output is 20.28 dB and 0.6988 (reported). v2 only passes ReSplat on PSNR with the refiner, at 20.93 dB, a margin of 0.02 dB, and the refiner is the part not released. Pose-free, DA3-GS has a higher mean SSIM (0.6463) than v2's feed-forward 0.6423 (reported). None of this undoes the result: the LPIPS and DISTS gaps are large, and the qualitative figures show the sharper texture those metrics reward. But "state of the art" here is a perceptual-metric statement, and the PSNR column says ReSplat is still the more faithful reconstruction per pixel when poses are given. All baselines were run by the authors on their splits (reported); none of this is independent.

Speed: the feed-forward network takes 5.619 s for eight views at 1022 × 756 and 0.564 s at 378 × 280 on one NVIDIA H20 (reported, Table 9, neural modules only). That is 7.3 times the pixels for 9.96 times the time (reasoned), the super-linear growth you expect from global attention over every view's tokens. Training was 25,000 steps on 16 H20s, about 48 hours, plus 13 hours for the refiner stage, on a mix of Hypersim (60%), ScanNet++ (30%) and Raw3D (10%) (reported). The paper reports no memory figure. The weights alone are 6.60 GB in float32 and the loader keeps them in float32, autocasting activations to bfloat16 on CUDA (measured, pipeline.py and model.py). The Space declares ZeroGPU hardware and a 60-second GPU budget per call (measured, the Space API and quota.py). I have not run either.

The licence is three licences

• The code is ZJU3DV's Project Registration License v1.0: free for research, education, evaluation and personal use; free for any organisation's project use too, but only after registering the project through a Google Form (measured, LICENSE).
• Two retained source files (decoder.py, geometry.py) are adapted from Tencent's HunyuanWorld-Mirror, whose community licence ships in License.txt. That licence defines its territory as worldwide excluding the EU, the UK and South Korea (measured). da3_order.py is Apache-2.0 from DA3.
• The weights have no licence field on their Hugging Face card (measured). They are fine-tuned from DA3NESTED-GIANT-LARGE-1.1, which is CC BY-NC 4.0 on its own card (measured), so I would treat them as non-commercial until the authors say otherwise (reasoned, not legal advice).

One loose end: THIRD_PARTY.md says "Public redistribution of this local candidate and its weights has not been marked approved", while release_manifest.json in the same directory has "public_approved": true (measured). The second is presumably the newer statement.

Where it fits

If you have a DA3-quality point cloud and want something that renders from angles the cameras never saw, v2's design is the lesson more than the checkpoint: keep the geometry model frozen, keep centres where it put them, let the rendering loss shape and fade the Gaussians, and supervise from far views so it has to. It is one forward pass, it accepts unposed photos, and its cleaning step is a threshold anyone can audit.

It is bounded by its base model in exactly the way its title suggests. A surface DA3 places wrong stays wrong, because no Gaussian can move. Colour is one RGB per Gaussian, so reflections bake in. The best numbers need a refiner that is not released. And for commercial work, the weights' DA3 ancestry matters more than the registration form.