SAM 3.1: the 7× is at 128 objects

Meta released SAM 3.1 on 27 March 2026 with three different headline numbers, on three different surfaces, for the same piece of work.

The Hugging Face card says Object Multiplex "delivers ~7x faster inference at 128 objects on a single H100 GPU without sacrificing accuracy." The blog post says it "doubles the processing speed for videos with a medium number of objects, increasing throughput from 16 to 32 frames per second on a single H100 GPU." Appendix H of the paper, posted the day after the release, says it achieves "a 5.2× speedup at 128 objects."

None of these contradicts the others. They are three readings of one curve, and the reason they differ by a factor of three and a half is the most interesting thing about the release.

First, the workload itself, because the whole argument below is about what it costs and a still frame cannot show it:

Meta published the curve

The brief version of a scaling claim is a ratio. The honest version is a graph, and Meta shipped one — three lines, eight measured object counts, every point labelled.

expand

Read the ends. At 128 objects SAM 3.1 runs at 11.5 fps against the November release's 1.6 — 7.19×, which is the model card's number and the 7.2× callout on Meta's own figure. Against the middle line, SAM 3 carrying the same batched postprocessing and torch.compile fusion, it is 11.5 against 2.2 — 5.23×, which is the paper's. The gap between those two is not accuracy or measurement noise. It is a release that bundles an algorithm with a pile of inference engineering, headlined on the combined figure and ablated in the appendix.

The blog's 2× is the same gap read at the other end. Nothing on the published curve is exactly 16 → 32 fps; the nearest measured point is eight objects, where SAM 3 does 14.6 and SAM 3.1 does 31.6, a 2.16× that rounds to the sentence Meta wrote. So "medium number of objects" means about seven, and the blog chose the part of the curve where most readers actually live. That reading is corroborated by Meta's own November sentence about SAM 3 — that it sustains "near real-time performance for approximately five concurrent objects," which on this curve is somewhere around 18 fps. Both statements are describing the same handful of objects, four months apart.

The 128 in the headline is not arbitrary either. It is the shipped default: max_num_objects=128 in sam3/model/sam3_multiplex_base.py, with the comment "128 objects (total across all GPUs) should be able to cover nearly all cases." Meta quoted the speedup at its own declared worst case. That is a defensible place to quote from — as long as you say so, which the model card does and the blog does not.

What is actually shared

SAM 3 already shared the expensive part. Its detector and tracker sit on one Perception Encoder backbone, and the frame is encoded once no matter how many objects are in it. What was replicated was everything downstream: a memory bank per tracked object, a memory-encode and memory-attention pass per object, a mask decode per object. The November blog named this as a known limitation, in the section on what the community should work on next:

Four months later they shipped exactly that. Object Multiplex groups objects into fixed-capacity buckets of M slots — M = 16 by default — and runs the memory path once per bucket instead of once per object, taking the pass count from O(N) to O(⌈N/M⌉).

expand

The factorisation is the mechanism, and the paper states it plainly: Multiplex splits object information into (i) a shared spatial memory and (ii) object-specific embeddings. The shared memory is produced by encoding the frame features together with all sixteen slots' mask signals at once, yielding one memory representation for the whole bucket. The embeddings are what keeps the objects apart — stored in the memory bank alongside the spatial features, and handed to the mask decoder so shared features get attributed to the right object.

Both halves are in the tracker, at 2345a4a. The shared half first:

# sam3/model/video_tracking_multiplex.py — _prepare_memory_conditioned_features()
B = multiplex_state.num_buckets
# B = current_vision_feats[-1].size(1)  # batch size on this frame
vision_feat = current_vision_feats[-1].expand(-1, B, -1)
vision_mask = (
    current_vision_masks[-1].expand(-1, B, -1)
    if current_vision_masks[-1] is not None
    else None
)
vision_pos_embed = current_vision_pos_embeds[-1].expand(-1, B, -1)

The commented-out line is the diff. B used to be the batch dimension, which for a video tracker is the object dimension; now it is the bucket count. And .expand on a size-1 axis is a stride-0 view, not a copy — twenty objects see the same 5,184 × 256 frame encoding, addressed twenty times, with nothing duplicated in memory.

The object-specific half is a matmul against a partial permutation matrix, and it is where sixteen objects become one memory encode:

# the same file — _encode_new_memory(): sixteen masks in, one memory out
mux_mask_for_mem = multiplex_state.mux(mask_for_mem).squeeze(2)  # [N,1,H,W] -> [b,16,H,W]
...
embedded_conditions = multiplex_state.mux(embedded_conditions).squeeze(2)
mux_mask_for_mem = torch.cat([mux_mask_for_mem, embedded_conditions], dim=1)
...
maskmem_out = self.maskmem_backbone(pix_feat, mux_mask_for_mem, skip_mask_sigmoid=True)

One call to maskmem_backbone where SAM 3 makes sixteen. The cat is also where the encoder's 32 input channels come from, and they are not what you would guess: sixteen slot masks, plus sixteen spatial flags marking which slots were conditioned on this frame.

The decoder is the other half of the split, and there identity costs one add:

# sam3/model/multiplex_mask_decoder.py — predict_masks(): 80 queries, one pass
mask_tokens = self.mask_tokens.weight.view(
    1, self.multiplex_count, self.num_mask_output_per_object, -1
).expand(B, -1, -1, -1)
mask_tokens = mask_tokens + extra_per_object_embeddings.unsqueeze(2)
mask_tokens = mask_tokens.flatten(1, 2)
tokens = torch.cat([tokens, mask_tokens], dim=1)  # obj_score 16 + iou 16 + mask 48
...
hs, src = self.transformer(src, pos_src, tokens)

Sixteen objects, one call to self.transformer. The only thing that distinguishes them is a per-object vector summed onto their mask queries.

All of that is checkable in the released code, and it checks out:

• build_sam3_multiplex_video_model(multiplex_count: int = 16) — the bucket capacity, exactly as the paper says.
• The memory encoder's mask downsampler is built with multiplex_count=16, input_channel_multiplier=2, and inside SimpleMaskDownSampler the first convolution's input width is multiplex_count * input_channel_multiplier — 32 channels, which the cat above accounts for exactly, convolved down to one 256-dim memory. SAM 3's builder passes none of those arguments, so the same module takes one channel and runs once per object. It is not quite the same stack, though: the multiplex builder also passes starting_out_chan=4, shifting the whole channel schedule one step up — 32 → 16 → 64 → 256 → 1024 → 256 against SAM 3's 1 → 4 → 16 → 64 → 256 → 256. Every layer's output is four times wider. That widening is what the 7% further down is paying for.
• The mask decoder allocates multiplex_count × 3 mask tokens plus a 16-wide iou_token and a 16-wide obj_score_token — 80 tokens per bucket, which is the paper's 5 × M (three mask, one IoU, one object-score, per slot), and which is why sixteen objects' worth of mask decoding costs barely more than one object's.
• The object embeddings arrive as extra_per_object_embeddings, documented in the decoder as "a tensor with shape b * multiplex_count * C to be added to the mask tokens" — added, not concatenated, as the fence above shows.

The justification for why this does not destroy information is one sentence and worth pausing on: "per-object memory features are largely spatially disjoint in most frames; overlaps are sparse, so joint encoding preserves object-specific information while avoiding redundant computation." Sixteen objects in a frame mostly occupy sixteen different places. Sharing the canvas costs you almost nothing because they were never fighting over it.

The lineage is cited and it is a good one: DataMUX (Murahari et al., NeurIPS 2022), which multiplexes up to 20–40 unrelated inputs through one network for 11–18× throughput at a 2–4% accuracy cost. Object Multiplex is the same trick with a much friendlier premise. DataMUX mixes inputs that have nothing to do with each other, so interference is pure loss. Here the sixteen things being mixed are sixteen objects in the same frame, which is why the accuracy doesn't go down — and, on the crowded benchmarks, why it goes up.

Why the number is a slope

Here is the part that the single ratio hides. Fit a straight line to Meta's own eight points and SAM 3's video cost is almost perfectly linear in object count:

SAM 3 (Nov 2025)   ms/frame = 30.9 + 4.645 · N     R² = 0.99967
SAM 3.1            ms/frame = 29.0 + 0.477 · N     R² = 0.978

Two lines with the same intercept and a slope 9.7× shallower. The intercept — 30.9 ms against 29.0, the same number within the noise of an eight-point fit — is the per-frame backbone, which both versions always shared. The slope is the memory path, and that is the only thing Object Multiplex touched.

Inside a single bucket the slope is flatter still: across 1 → 16 objects SAM 3.1 goes from 29.6 to 33.1 ms, a marginal 0.23 ms per object, against SAM 3's 4.29 ms over the same range. Eighteen times cheaper per additional object, up to the sixteenth. Then a second bucket opens and the line kinks.

Which means "7× faster" is not a property of the model. It is one evaluation of a ratio between two lines, and you can make it come out at almost any value by choosing where to stand:

If you track a handful of objects — which, for most video work, you do — Object Multiplex gives you somewhere between nothing and 65%, and the rest of your gain is the torch.compile work.

The 7% tax, published

At one object SAM 3.1 runs at 33.8 fps against an optimized SAM 3's 36.4. It is slower, by about 7%, and the paper says so in its own words: "At single-object tracking, the bucketization strategy in Multiplex introduces a modest overhead of approximately 7% in latency. However, beyond the crossover point (≥2 objects), SAM 3 Multiplex outperforms SAM 3."

The tax is charged in one line. A bucket is sixteen slots wide before it knows how many objects it will hold:

# sam3/model/multiplex_utils.py — MultiplexState.add_objects()
while len(object_indices) > 0:
    new_bucket = [_PADDING_NUM] * self.multiplex_count  # 16 slots, always
    for i in range(self.allowed_bucket_capacity):
        if len(object_indices) == 0:
            break
        new_bucket[i] = _pop_next()
    self.assignments.append(new_bucket)

Every tensor downstream is then shaped by that width rather than by the object count:

# the same file — MultiplexState.mux(): "padding entries filled with 0"
output_shape = (self.num_buckets, self.multiplex_count) + x.shape[1:]
...
result_flat = self.mux_matrix @ x_flat  # [buckets*16, N] @ [N, features]

mux_matrix carries a 1 for every filled slot and an all-zero row for every empty one, so a single-object bucket is a [1, 16, …] tensor that is fifteen-sixteenths zeros — and it is fed to a convolution stack four times wider than SAM 3's. Counting multiply-accumulates through both constructors at the shipped interpol_size=[1152, 1152]: SAM 3's mask downsampler is 1.355 GMAC per object, SAM 3.1's is 18.94 GMAC per bucket. Fourteen times the arithmetic, for one call that is supposed to cover sixteen objects.

Which means the memory encoder is not where multiplexing wins. At sixteen objects it is close to a wash — 21.7 GMAC for sixteen SAM 3 calls against 18.94 for one bucket. The collapse from O(N) to O(⌈N/M⌉) is in memory attention and the mask decoder, which genuinely go from N passes to one. At a single object you pay the wider encoder and get none of the collapse. Seven percent.

Meta measured that and printed it on the chart, next to the number they were selling. Credit where it is due: a release that publishes its own crossover point is telling you how to decide against it.

"Without sacrificing accuracy"

That phrase is doing more work than the speedup is, and the release notes are more careful than the model card.

On video object segmentation — mask-prompted tracking, no detector involved, so this is a clean read on the tracker change alone — SAM 3.1 improves on six of seven benchmarks. Meta says "six out of seven" and prints the table, so you can find the seventh without asking.

The seventh is YouTube-VOS 2019 val, which goes 89.7 → 89.3, down 0.4 G. It is the oldest and most saturated benchmark in the set, everything above 89 on it is noise-adjacent, and trading 0.4 there for +2.0 on MOSEv2 — the occlusion-heavy one where SAM 3 only manages 60.3 — is a trade any practitioner would take. The point is not that the regression matters. It is that a release which reports the benchmark it lost on has earned the benefit of the doubt on the ones it won.

The video-PCS table is where "without sacrificing accuracy" strains. Same release, different column:

Three of the four public benchmarks go down — LVVIS by 2.0, BURST by 1.2, YTVIS21 by 0.8 in the shipped checkpoint. The release notes' bullet reads "Mixed results on SA-Co/VEval video benchmarks, with notable improvement on YT-Temporal-1B (+2.1 cgF1)" — but SA-Co/VEval is the column where five of that table's six numbers improved. The mixed column is the public one, and the bullet does not mention it. The table does, which is the part that counts. So does the paper's prose: "results are mixed with improvements on OVIS (+1.8 mAP) but slight regressions on LVVIS, BURST, and YTVIS21."

Which way they move is not random, and it points at the mechanism. The two biggest gains anywhere in the release are OVIS on this table (+1.8) and MOSEv2 on the VOS table (+2.0). OVIS is Occluded Video Instance Segmentation; MOSE is the crowded, heavily-occluded successor to DAVIS. That is exactly where sixteen objects sharing a memory would be expected to help — a tracker that can see its neighbours is a tracker that can tell them apart when they overlap. The losses are less tidy: LVVIS and BURST are the large-vocabulary benchmarks, 1.2K and 482 noun phrases against OVIS's 25, which fits a story about concept coverage — but YTVIS21 has only 40 noun phrases and drops too, so that story is incomplete. Six deltas is not enough to name a cause.

The shift that mattered more happened in November

Object Multiplex is a performance result. The thing SAM 3 did to the field was a task change, and it is worth restating because SAM 3.1 inherits all of it unchanged.

expand

SAM 1 and SAM 2 took geometric prompts: click a point, draw a box, hand over a mask, get that one object back. SAM 3 added concept prompts — an open-vocabulary short noun phrase, or an image exemplar — and changed the return type. You do not get an object; you get every instance matching the concept, each with a stable identity across the video. Meta call the task promptable concept segmentation, and the benchmark they built for it, SA-Co, carries 270K unique concepts — "over 50 times more than existing benchmarks," by their count.

The detector is DETR, and not loosely: the architecture is "broadly based on the SAM and (M)DETR series," the object queries cross-attend to prompt-conditioned image features exactly as in Carion et al. 2020, and the first author of the DETR paper is a core contributor on SAM 3 — the BibTeX key is carion2025sam3segmentconcepts. What text prompting adds to that skeleton is a fusion encoder that conditions the image embeddings on the prompt tokens before the queries ever see them, so "a penguin" is not a label the model picks from a list; it is a vector the image features have already been bent around.

The piece with no ancestor in DETR or SAM 2 is the presence token. Asking each object query to decide both what this is and where it is puts two objectives in conflict — recognition wants global context, localisation wants local, and the two pull against each other. SAM 3 factors them: a single learned global token predicts p(the noun phrase is present at all), each query predicts p(I am a match | it is present), and the final score is the product. That is the whole trick for keeping "a player in white" from firing on a player in red, and it is what makes exhaustive open-vocabulary detection work at all.

The prompts stay short, deliberately. SAM 3 does not do "the second to last book from the right on the top shelf" — for that you bolt on an MLLM that proposes noun phrases and inspects the returned masks, which Meta ship as SAM 3 Agent. Concept prompts are a vocabulary problem, not a reasoning one, and the model is honest about which it solves.

What a minute costs

SAM 3.1 is on the Meta Model API at $2.50 per 1,000 images and $0.20 per 1,000 frames. The first thing that falls out is the ratio between them: an image costs 12.5 frames. A single image runs the full detector on one frame with no memory to amortise; a video frame gets the tracker's shared work spread across it. The price sheet is describing the architecture.

A minute of 30 fps video is 1,800 frames, so it is $0.36. An hour is $21.60. Those are flat — the price does not know how many objects are in the frame.

The compute underneath is not flat.

At the median on-demand H100 price — $3.38/GPU-hr across 39 providers on the day this was written, with the cheapest in-stock listing at $1.25 — a minute of sixteen-object video costs 5.6¢ of rented GPU to run yourself against the API's 36¢. Even at 128 objects, where SAM 3.1 drops to 11.5 fps, it is 14.7¢ against 36¢. Self-hosting is 2.4× to 7.2× cheaper on compute across the entire range, and the break-even utilisation is low: keep one H100 busy more than 14% of the time at low object counts, or 41% at 128, and renting it beats the API. That comparison is compute only, though — it prices none of the engineering, the gated download, the cold starts, or the fact that when the hosted endpoint falls over, it is not you being paged.

The more interesting arithmetic runs the other way. Under the November model, 128 objects took 1,125 seconds of one H100 per minute of video — $1.06 of rented compute for something the API sells at $0.36. Flat per-frame pricing on a cost curve with a 4.6 ms/object slope is a bet that nobody sends you crowded video. Object Multiplex takes that same minute to $0.15.

I cannot prove those two facts are connected — Meta owns its hardware, its marginal cost is not a rental rate, and nothing in the docs caps objects per request. But the only segmentation model on the Meta Model API is SAM 3.1: developer.meta.com/ai/models/sam-3-1/ resolves and .../sam-3/ returns 404, and the price sheet's only segmentation row is headed "SAM 3.1." Whatever the reason, the flat per-frame price arrived with the model that made flat per-frame pricing survivable, and not before.

Open weights, gated door

What "open" buys you here, precisely:

• The checkpoint, sam3.1_multiplex.pt, 3.50 GB. One file, PyTorch pickle. Not safetensors.
• Under the SAM License — not Apache, not MIT. Commercial use is granted, redistribution must carry the licence forward, attribution is required in publications, and there is a use policy that excludes ITAR-controlled applications, weapons, nuclear and espionage.
• Gated manual. You agree to share contact information with Meta and wait for approval. curl on the README returns 401.
• The training and fine-tuning code, the SA-Co benchmark, and the evaluation harness — which is the part that actually lets you check any of the numbers above, and is why this article could.

And the thing the card says in its own words: "This repository hosts only the SAM 3.1 model checkpoints — there is no Hugging Face Transformers integration." SAM 3 ships model.safetensors, library_name: transformers, an AutoModel mapping, and is deployable to Inference Endpoints and Azure. SAM 3.1 ships a .pt and the instruction to go install facebookresearch/sam3 from source.

You can see what that costs in the download counts. Over the same trailing thirty days, facebook/sam3 pulled 2,086,256 downloads and carries 100 Spaces; facebook/sam3.1 pulled 48,249 and carries 9. The faster model has 2.3% of the slower one's traffic. Meta calls SAM 3.1 "a drop-in replacement for SAM 3," and for anyone already on the research repo it is. For the two million downloads a month coming through transformers, it is a rewrite.

The API, meanwhile, is transformers-free by construction: text prompt in, boxes and pixel-precise masks out, identities preserved across frames, no weights, no fine-tuning, no approval form. Which is the actual trade. The weights are open and inconvenient; the API is closed and immediate; and the release that made the open path dramatically cheaper to run is the one that made it harder to start.

What I'd want measured

Meta published more of its own curve than most releases do — three lines, eight points, the crossover, the benchmark it lost, and a separate line isolating engineering from algorithm. Sorting this article's claims by how much they rest on that: reported is everything in quotation marks and every benchmark number, all of it from Meta's own release notes, paper appendix and price sheet. Measured is the file sizes, the download counts, the multiplex_count=16 and max_num_objects=128 read out of the shipped code, the two mask-downsampler MAC counts computed from their constructors, and every ratio computed from the printed labels on Meta's figure. Reasoned is the two-parameter cost model, the claim that the gains cluster on occlusion benchmarks, the reading of where multiplexing's saving actually lives, the parameter delta, and the suggestion that the pricing and the speedup are related. Those five are below, each with the experiment that would kill it.