# SAM 3 as four ONNX files: what the export pins, and what I ran
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/sam3-onnx
> date: 2026-10-31
> tags: computer-vision, segmentation, onnx, on-device, inference, benchmarks
A post from the author said SAM 3 now runs offline on your own GPU through an ONNX export, on NVIDIA CUDA and Apple CoreML, with "text, example boxes, negative boxes and clicks as prompts". The repository is [wkentaro/sam3-onnx](https://github.com/wkentaro/sam3-onnx), the runtime that downloads it is [wkentaro/osam](https://github.com/wkentaro/osam), and the weights are on Hugging Face. The post named the v0.5.0 files. By the time I read the repository, v0.6.0 had replaced them, so everything below is about v0.6.0 (commit `f921702`).
I did not take the claim on trust. I read the export script, opened all four graphs, then downloaded the files and ran them. This machine has no GPU, so the CUDA and CoreML numbers are the benchmark document's. Every figure is labelled. **Measured** is my run: Python 3.13, onnxruntime 1.31.1, CPU provider, two threads of a shared four-vCPU box that other jobs were also using. **Reported** is the repository's or osam's own benchmark. **Reasoned** is arithmetic or inference from code I read but did not execute.
## What a segmentation model needs to be split into
SAM 3 is a promptable model: you give it an image and a prompt, it gives masks. For deployment that sentence hides an asymmetry. The image is expensive and changes once. The prompt is cheap and changes constantly, because a person is clicking. A good export keeps the expensive part separate so its output can be reused.
This one does. There are four graphs, and which ones run depends on the prompt.
**Measured**, from the Hugging Face file listing and from `onnx.load` on each file:
| File | Bytes | Nodes | fp32 weights |
| --- | ---: | ---: | ---: |
| `sam3_image_encoder.onnx` | 1,858,609,604 | 2,366 | 464,055,783 |
| `sam3_language_encoder.onnx` | 1,616,390,301 | not inspected | about 404 M by size |
| `sam3_decoder.onnx` | 129,943,466 | 4,861 | 32,022,069 |
| `sam3_point_decoder.onnx` | 16,635,774 | 1,427 | 4,065,072 |
The four add up to 3,621,579,145 bytes, 3.37 GiB, and osam's README lists SAM 3 as "3.4GB" and 893 M parameters. That agrees with my sums (**reasoned**: 464 M + about 404 M + 32 M + 4 M, with the language count taken from its file size). The encoder's SHA-256 is `c6b984d9…`, the one osam pins, so I ran the file the runtime would fetch.
Everything is float32: every initializer in the three graphs I opened is `FLOAT` or `INT64`. There is no quantized variant among the v0.6.0 files, so the speed numbers below are fp32 numbers and 3.6 GB is the floor for a download.
### The encoder is the product
The image encoder is a vision transformer. The op histogram says so: 128 `Gemm`, 64 `MatMul`, 32 `Softmax`, 65 `LayerNormalization`, 34 `Gelu`. Thirty-two `Softmax` nodes means thirty-two attention blocks, with four linear layers each and two `MatMul`s (query-key, attention-value); that is **reasoned** from the counts, not from the model definition. Six `Sin` and six `Cos` are position encodings, and fifteen `Conv` plus six `ConvTranspose` build the feature pyramid.
It takes a `uint8` tensor of shape `3x1008x1008`; normalisation is inside the graph. It returns nine tensors. The decoder uses four (`vision_pos_enc_2`, `backbone_fpn_0` to `_2`), the point decoder three others (`image_embeddings`, `high_res_features1`, `high_res_features2`), and two are read by nobody: `vision_pos_enc_0` and `_1`, 85 MB and 21 MB. The decoder's input list confirms it.
## How the export is written
Reading `export_onnx.py` (575 lines) explains most of what the files look like.
**The graphs are exported from the PyTorch model, not rewritten.** The script calls `torch.onnx.export` at opset 21 on PyTorch 2.9.1 (that is the producer string in the files) with `verify=True`, so each export is compared against onnxruntime before it is kept. The two encoders use the default exporter; the decoder and point decoder use the legacy tracer (`dynamo=False`). Tracing records whatever shapes it saw, and the script's comments say so: "A fixed number of box slots, as tracing bakes the box count into the model."
**So the decoder takes exactly 16 boxes.** `MAX_BOXES = 16`. The inputs are `box_coords` of shape `[16, 1, 4]`, `box_labels` `[16, 1]` and `box_masks` `[1, 16]`. Unused slots are filled with zeros and marked as padding in `box_masks`, "which SAM3 batches the same way". A prompt with seventeen example boxes is an error in `infer_onnx.py`. A prompt with none is the same graph with sixteen slots masked.
**Everything else is fixed too.** The image input is `[3, 1008, 1008]`, the tokens `[1, 32]`, the decoder's features `[1, 256, 288, 288]`, `[1, 256, 144, 144]` and `[1, 256, 72, 72]`. The one dynamic axis in the whole set is the point decoder's `num_points`. osam's comment explains the price: "CoreML can't load the point decoder: its point_coords dimension is unbounded", so on Apple hardware the point decoder runs on the CPU.
**The output count is dynamic.** This is the one place the graph is more flexible than its inputs. The score threshold (`confidence_threshold`, applied as `out_probs > ...` inside `_Decoder.forward`) is traced into the graph, with `NonZero` and `GatherND` nodes selecting the survivors, and the declared output shapes are symbolic (`Concatboxes_dim_0`). So the decoder returns however many objects clear the threshold. I will show below what that looks like when nothing does.
**Some details I would not have guessed.**
- The tracker "enters bfloat16 autocast for the whole process on construction, which puts bfloat16 Convs into the exported model", so the script calls `bf16_context.__exit__` first.
- The point decoder is exported on the CPU because "tracing torch.repeat_interleave in the mask decoder puts its index on the CPU".
- Resizing is not in the graph. `preprocess_image` resizes to 1008 by 1008 with PIL's bilinear filter in the caller, because Resize "has difference between pytorch and onnx".
- `postprocess_onnx.py` (219 lines) rewrites the finished graph: `MatMul` plus `Add` becomes `Reshape`, `Gemm`, `Reshape`, and in v0.6.0 every `Gemm` weight is stored `{K, N}` with `transB=0`. Its docstring says every rewrite is exact. The reason is that onnxruntime's WebGPU provider runs `transB=1` at half the speed (1,477 ms to 727 ms on an RTX 3080 Ti, **reported**); the cost is that CoreML takes minutes to load `transB=0`, so `load_for_coreml()` transposes the weights back in memory on every CoreML run.
The point decoder shows the other side of tracing: of its 1,427 nodes, 397 are `Constant`, 134 `Shape`, 117 `Gather`, 117 `Squeeze`, 116 `Unsqueeze`, 61 `Cast` and 52 `Concat`. That is 994 nodes of shape bookkeeping, 70 percent of the graph, wrapped around 49 `MatMul` and 15 `Gemm`. It costs little at run time and a lot of reading.
## What I ran
I downloaded the point decoder (16.6 MB), the decoder (130 MB) and the image encoder (1.86 GB), checked the encoder's hash, and drew two synthetic scenes with PIL, 800 by 600 and 1280 by 720: a noisy gradient with four flat shapes (red circle, blue rectangle, green triangle, yellow ellipse). I drew them, so I know every mask exactly. I ran the encoder once per scene, saved its outputs (138 MB per image, **measured**), deleted it, then streamed the 1.6 GB language encoder into memory over HTTP, never storing it, and ran it with the real decoder.
### The encoder
Loading the encoder took 12.3 s and 3.2 GB of resident memory. The first run on the 800 by 600 scene took 82.9 s and peaked at 5.3 GB, arena allocation included. Warm runs: 46.4 s asking for only the three outputs the point path needs, 47.8 s for all nine. The 1280 by 720 scene took 64.3 s, but I could not keep the shared box quiet during it, so I do not read it as a resolution effect; the graph input is a fixed 1008 square either way.
The reported CPU figure is 6,563 ms on an i9-12900K. My 47 s on two threads is 7.2 times that (**reasoned**), about what a 24-thread desktop against two shared vCPUs should give. The point is that a 464 M parameter ViT on a CPU is a seconds-to-minutes operation, and the design bets you will do it once.
Asking for three outputs instead of nine saved nothing (46.4 s against 47.8 s, inside the noise). The unread outputs are position encodings, cheap to compute (**reasoned**). They cost memory and copies: the nine outputs are 244 MB of float32, and 106 MB of that is the two tensors nothing uses. `infer_onnx.py` passes `output_names`, which saves the copies and not the compute.
### The point decoder, against shapes with known masks
This is the cleanest test, because the graph takes the encoder's features and one click and nothing else. For each shape I clicked its centre and compared the result with the PIL mask. I used the repository's own post-processing: two passes (the second feeds the best mask back in as `mask_input`), a pick of the candidate with the highest predicted IoU, hole filling up to 256 px, and a bilinear resize to the source size.
**Measured**, mask IoU against ground truth, eight shapes across the two scenes:
- One pass: 0.9925 to 0.9975, mean 0.9964.
- After the refine pass: 0.9949 to 0.9996, mean 0.9970.
The second pass helped the two rectangles (0.9949 to 0.9996 and 0.9925 to 0.9993) and moved the other six by less than 0.005 in either direction. On flat shapes with hard edges there is not much to refine, so this says nothing about photographs.
Three candidates come back for every click, and they are not the same thing. For the circle at its centre the true IoUs of the three were 0.054, 0.985 and 0.997, while the model's own predicted IoUs were 0.06, 0.513 and 0.987. The graph's confidence ranking picked the right one in all four scene-0 shapes, though not always the same index: the triangle's best was candidate 0, the others' candidate 2. Predicted IoU ran about 0.01 low against the truth (0.985 to 0.989 against 0.995 to 0.997 for the kept masks). Index order is not a quality order, which is why the code takes `argmax` of the predicted scores and not the last slot.
Other click behaviour I checked on scene 0:
- A positive click on the circle plus a negative click on the rectangle: circle IoU 0.9968. The negative point did no harm, though it had nothing to fight.
- Two positive clicks, circle and rectangle: one mask, IoU 0.9936 against the union. A click prompt is single-object, but it merges what you tell it to.
- A click on empty background at (40, 570): a mask covering 71 percent of the image, predicted IoU 0.945. The background came back as an object; if your tool should reject that, it has to check the area itself.
Latency of one decoder pass, median of seven runs on two threads: 216 to 225 ms for 2, 4, 8 and 16 points, 268 ms at 32 and 280 ms at 64 (**measured**; one point read 310 ms, an outlier I did not repeat). A real click runs two passes, so about 0.44 s here against 83 ms reported on the i9 for a click.
### Text and box prompts on the real decoder
Here I needed the language encoder. It is a 1.6 GB graph for a 32-token input, and it took 593 ms per prompt on two threads (median of five, **measured**). It returns `text_attention_mask` `[1, 32]`, `text_memory` `[32, 1, 256]` and `text_embeds` `[32, 1, 1024]`; the decoder takes the first two, and `text_embeds` is exported and never read. Streaming the file into memory and creating the session peaked at 6.3 GB resident, so a laptop with 8 GB will feel this model.
Results on both scenes, matching each returned mask to the best ground-truth shape (**measured**):
| Prompt | Scene 0 (800x600) | Scene 1 (1280x720) |
| --- | --- | --- |
| "red circle" | circle, IoU 0.993, score 0.947 | circle, 0.993, 0.956 |
| "blue rectangle" | rectangle, 0.996, 0.947 | rectangle, 0.991, 0.955 |
| "green triangle" | triangle, 0.994, 0.972 | triangle, 0.994, 0.973 |
| "yellow ellipse" | ellipse, 0.994, 0.927 | ellipse, 0.993, 0.935 |
| "visual" (the default) | all four shapes, 0.988 to 0.995, scores 0.50 to 0.72 | all four, 0.991 to 0.995, scores 0.52 to 0.78 |
| "shape" | no masks | no masks |
| "cat" | no masks | no masks |
| box around the circle (+ "visual") | circle only, 0.991, score 0.991 | circle only, 0.991, 0.991 |
| the same + a negative box on the rectangle | circle only, 0.991, 0.991 | circle only, 0.991, 0.992 |
Each colour-and-shape phrase found exactly one object. The bare word "shape" found none although there are four: SAM 3 wants a concrete noun phrase, and an abstract category did not clear the 0.5 threshold. "cat" found nothing, correctly. The lowest score I saw was 0.503, the rectangle under "visual", just above that threshold, consistent with it being applied in the graph.
The box rows are the part I would test harder. A positive box around the circle narrowed the default "visual" result, all four shapes, to the circle alone, so the box acted as an exemplar and not just a region. The negative box on the rectangle changed nothing, but the rectangle was already excluded, so my test does not show a negative box doing real work. I did not draw two similar objects, and deleted the model files before I thought of it. I would not cite this table as evidence that negative boxes work.
### The sixteen slots cost the same whether you use them or not
I ran the decoder with 0, 1, 2, 4, 8 and 16 live boxes. Median of three runs, in milliseconds: 4,605; 5,428; 5,824; 4,603; 6,174; 5,845 (**measured**). There is no trend; the 4.6 to 6.2 s spread is shared-machine noise. That is what fixed slots predict (**reasoned**): masked slots still flow through attention, so one box pays for sixteen. The decoder is about 5.6 s on my box and a click 0.44 s, about 13 times; on the i9 it is about 14 (1,152 ms against 83 ms, **reported**). The language encoder added 593 ms, around a tenth of a text prompt.
One more consequence, from reading `infer_onnx.py` and osam's `_models.py` and not from running them: a box-only prompt still sets the text to "visual" and runs the language encoder on it, every time, with no cache I could find. A caller that only ever uses clicks and boxes still has to carry the 1.6 GB graph for the first and a constant embedding it recomputes for the second.
## Latency, with and without keeping the embedding
The benchmark document ([docs/benchmark.md](https://github.com/wkentaro/osam/blob/main/docs/benchmark.md), **reported**, measured on 2026-10-31 with v0.6.0, median of five runs after a warm-up, image `dogs.jpg` at 2560 by 1600) gives per-device figures:
| Device | Encode | Click | Box | Negative box | Text |
| --- | ---: | ---: | ---: | ---: | ---: |
| CPU, Intel i9-12900K | 6,563 | 83 | 1,152 | 1,162 | 1,162 |
| CUDA, NVIDIA RTX 3080 Ti | 347 | 27 | 72 | 72 | 74 |
| WebGPU, NVIDIA RTX 3080 Ti | 741 | 57 | 199 | 196 | 203 |
| CPU, Apple M5 | 15,341 | 64 | 1,475 | 1,489 | 1,519 |
| CoreML, Apple M5 | 2,466 | 61 | 375 | 376 | 376 |
| WebGPU, Apple M5 | 3,691 | 46 | 610 | 621 | 604 |
Milliseconds. Each prompt reuses the image embedding. The doc says CoreML encodes 6.2 times faster than the CPU on the M5 and runs a text or box prompt 4.0 times faster, with the click decoder on the CPU in both. (The README rounds the encoder gain to "about 5x"; 2,466 against 15,341 is 6.2.) My own row would read 47,100 / 440 / 5,600 / 5,600 / 6,200.
The widget does the amortisation arithmetic. Keeping the embedding is the point of the split: on the i9 with ten clicks, 7.4 s against 66.5 s, a factor of nine (**reasoned**, from the table); with ten boxes 4.3; on CUDA with ten boxes 3.9. The smaller the encoder's share, the less the cache buys.
### Resolution is a quality question, not a latency question
The encoder's input is `3x1008x1008` whatever your photo, so its cost is flat in source size (**reasoned** from the graph). What scales is pre- and post-processing, timed on the same noisy box with random data: resizing to 1008 square took 17 to 70 ms from 640x480 to 4000x3000, and resizing ten 288x288 masks back and thresholding took 9 ms at 640x480, 31 ms at 1920x1080, 93 ms at 2560x1600 and 243 ms at 4000x3000 (**measured**, not monotonic at the small end). Ten full-size boolean masks at 4000x3000 are 120 MB. None of it competes with the encoder.
What the square does change is what the model sees. The resize is not aspect-preserving: `preprocess_image` passes `(1008, 1008)` straight to PIL. A 2560 by 1600 photo shrinks by 0.394 horizontally and 0.630 vertically, so circles become ellipses at 1.6 to 1, and an 80 px object is 31 by 50 px, about two 14 px patches wide (1008 over a 72 by 72 patch grid, **reasoned**). Masks come out at 288 by 288, four times coarser than the input, then get bilinearly resized. Small objects in large photos are where this runs out, which is why a tile-based wrapper like [the sam3-segmentation skill](/articles/sam3-segmentation-skill) exists: its finding that the win is the tiling is the same fact from the other end.
## What it says about the license
There are two licenses. The export code is MIT (Copyright 2025 Kentaro Wada). The weights, which the model card says are "under Meta's SAM License", are not: `models/LICENSE` and the Hugging Face file are the SAM License, "Last Updated: November 19, 2025", and the card's YAML says `license: other`. The ONNX files derive from Meta's weights, and the license grants a right "to use, reproduce, distribute, copy, create derivative works of, and make modifications to the SAM Materials", on a "non-exclusive, worldwide, non-transferable and royalty-free" basis.
The clauses I would check before shipping:
- **Redistribution.** You may distribute only "under the terms of this Agreement" and must "provide a copy of this Agreement". If these files go into a product, the license travels with them.
- **Publication.** Research published "on, using, or otherwise in connection with SAM Materials" must acknowledge the use.
- **Reverse engineering.** Your use "will not involve or encourage others to reverse engineer, decompile or discover the underlying components of the SAM Materials". I opened every graph with the `onnx` package and counted its ops, which this sentence could be read to cover. I am not a lawyer, and I used the files as published.
- **Trade controls.** No use for ITAR activities or "end uses prohibited by Trade Controls, including those related to military or warfare purposes, nuclear industries or applications, espionage, or the development or use of guns or illegal weapons".
- **Litigation.** Suing Meta over the materials ends your licenses.
I did not find a commercial-use prohibition in the text. That is a reading of the license as shipped, not legal advice, and Meta "may modify this Agreement from time to time". The README separately warns that the optional SAM3-LiteText encoders use MobileCLIP architectures and may fall under Apple's model license, which "limits use to non-commercial research". That is a different, stricter file for a different download.
## What I did not run
- **No GPU.** CUDA, WebGPU and CoreML numbers are all reported; I reproduced none.
- **No parity with PyTorch.** `test_text_parity.py` needs the original SAM 3 package and Meta's gated checkpoint. I checked the graphs on shapes I drew, which tests the wiring and not fidelity to the original model.
- **Synthetic scenes only.** Flat colour on a gradient is the easy case. IoU 0.99 says the encoder's outputs feed the decoders correctly and the coordinate scaling is right. It says nothing about hair, glass or crowds.
- **The language encoder's ops.** I ran and timed it but did not count its nodes, so that row of the first table is blank.
- **Negative boxes under load**, as above.
- **Noisy timings.** Other jobs shared the CPU, so treat 47 s as an order of magnitude.
## What to take away
The export is careful engineering: four static graphs, the per-image cost in one, the per-prompt cost in the others, and a thin Python layer that pads a prompt into sixteen slots. End to end it works. Every prompt type found the right shape on my scenes at IoU 0.99, the fixed slots cost what the code says, and the structure of the speed table (one expensive encode, cheap prompts) holds on a CPU far slower than the reported ones.
If you build on it:
1. **Cache the embedding per image.** It is 138 MB in float32 for clicks, text and boxes together, 21 MB for clicks alone. Ten prompts at the reported CPU numbers is a 4 to 9 times difference.
2. **Plan memory, not just time.** The encoder wanted 3.2 GB to load and 5.3 GB while running, and the language encoder pushed my process to 6.3 GB. The 3.4 GB in the README is the download, not the footprint.
3. **Expect a square view of your photo.** The model never sees your aspect ratio or resolution; for small objects, tile (and see [SAM 3.1's multiplexing](/articles/sam-3-1-object-multiplex) for the video side).
4. **Read the license before you ship the weights, not the code.**