sam3-segmentation: the win is the tiling, not the agent

On 24 October Aurélie Guisnet posted a picture of corn with three words over it: "Skill > Model". Six frontier agents had been asked to outline every visible kernel in one photo. Without tools, they returned between 0 and 1,084 shapes, most of them grids of boxes laid over the ears. With her skill, all six returned between 1,244 and 1,350 kernel outlines that follow the kernels. Her follow-up in the thread: "the model basically makes no difference, the skill does all the heavy lifting."

The picture supports a version of that sentence. Which version is the interesting part. The skill is lillyguisnet/sam3-segmentation, MIT, version 0.1.0, released the same day. I read all of it at commit d2d5129, then ran every part that runs without the model: the tests, the demo, and the planner on a public-domain corn photo. SAM 3's weights are gated and the skill runs inference only on an NVIDIA GPU, so I did not segment a photo myself. Numbers below are measured (my run), reported (the repository's) or reasoned (my arithmetic).

What is in the folder

An agent skill is a folder with a SKILL.md at the top. The agent reads its description to decide when to load it, then follows the body like a runbook. This one holds three things:

• SKILL.md, 335 lines (measured). It tells the agent which mode to pick, what to ask the user before anything expensive, the exact commands, and what to say in its final answer.
• Seven references: setup, method, configuration, review protocol, direct SAM calls, the five non-corn trials, and a page of lessons from earlier mistakes. Together they are 1,120 lines of prose (measured).
• Scripts: segment.py (1,462 lines, the full workflow), sam.py (561 lines, direct calls), sam3_backend.py (259 lines, the only file that imports PyTorch), core.py (456 lines of geometry, RLE and tiling), and a 696-line HTML viewer. A shell launcher, scripts/run, runs each script in one of two locked environments that live outside the skill folder.

The two environments are the first design decision worth noticing. base has numpy, scipy, Pillow, pycocotools and filelock, and no PyTorch; it plans, selects, writes review sheets, validates and exports. model pins PyTorch 2.10 with CUDA 12.8 and Meta's sam3 package at commit 0570b3a, and only inference and refinement need it.

Why SAM 3 alone does not see the kernels

SAM 3 does concept segmentation. You give it a short noun phrase, "corn kernel", and it returns every instance it finds. I covered the architecture in SAM 3.1: the 7× is at 128 objects: a DETR-style detector with 200 object queries, conditioned on the text, and a presence token that decides whether the phrase is in the image at all.

The catch is resolution. The model resizes whatever you give it to one fixed input size. A 20-pixel kernel in a 960-pixel photo becomes a speck in that view. The skill's own direct-call tests, run by the author on one RTX 3090, show what happens (reported):

The rule of thumb the skill draws from this: crop until a typical object is about a sixth of the region's width or more. The lessons page records the same failure from the run the skill was distilled from. The first whole-image pass with SAM 3.1 "returned whole ears but zero kernel masks". Cropping that path recovered 91.

That explains the picture. A language model cannot place 1,300 polygons by eye. SAM 3 on the whole photo does not see kernels. SAM 3 on 120-pixel tiles does. The skill makes the agent cut the photo up correctly, keep everything the model says, and put it back together without double counting.

The crop plan

The default corn config runs three passes and one parent query:

• whole: the full image, two text prompts ("corn kernel", "kernel").
• coarse: 200 px tiles overlapping by 60 px, both text prompts, plus a grid of single-point prompts every 6 px.
• fine: 120 px tiles overlapping by 30 px, text prompts only.
• ear of corn: one text request on the whole image, for the parent objects.

Tiles start at 0 and step by size minus overlap. If the last tile does not reach the far edge, one more tile is added flush with it, so the image is always covered and never padded with invented pixels. Before any weights load, segment.py plan counts the requests and checks them against a budget in the config. It also bounds the raw candidate pool: 200 detector slots per text request, plus three hypotheses per grid point. If any count is over budget, the skill refuses and tells the agent to ask the user.

I ported that planner to TypeScript line for line and checked it against the real tool. On the 960×640 CC0 corn photo with the corn config, segment.py plan printed 262 requests, 38,115 point prompts and a 159,745-candidate upper bound (measured). The port gives the same three numbers. On a 480×500 image, the size of the original corn photo, the real tool plans 99 requests (measured), which matches the "99 requests" the lessons page reports for the author's GPU rerun.

Three things the widget makes obvious that the prose hides:

1. The grid dominates. On the 960×640 photo, the coarse grid alone is 35 tiles of 33×33 points: 38,115 prompts, 114,345 of the 159,745 candidate slots (reasoned). Text requests are 227 of the 262 requests, but only 45,400 candidate slots.
2. The default budget refuses the bigger photo. With configs/corn-starting-point.json, the same photo fails all three caps: 262 requests against 256, 38,115 grid points against 25,000, 159,745 candidates against 100,000 (measured: the tool printed all three errors). The README's hero image used a copy of the config with the budget raised to 300 / 40,000 / 170,000. That is the design working as intended. The agent cannot quietly spend more GPU than the user agreed to.
3. The fine pass is where small objects become visible. A 20 px kernel is 1/48 of the 960 px photo and 1/6 of a 120 px tile (reasoned). Push the fine tile to 200 px in the widget and the column turns red.

Here is the same plan drawn on the actual photo:

expand

From raw candidates to proposals

Inference saves everything the model returns before any filter. Text requests run at detector threshold 0.0; point prompts keep all three hypotheses, empty ones included. Each saved file records the image hash, the exact request, hashes of the skill's code, the checkpoint checksum and the runtime, so a rerun resumes and a code change invalidates the cache.

select then turns the pool into proposals. The gates are explicit and counted: empty masks, an area range (5 to 650 px for corn), a text score of at least 0.2, a point stability of at least 0.83, the clicked point actually inside its own mask, and masks cut by an internal tile edge. That last gate matters. A kernel sliced by a tile boundary is a truncated observation, and the overlap guarantees another tile sees it whole. A mask touching the real image edge is kept.

Ordering is a judgement call, and the code says so. Text and point scores come from different heads and are not comparable probabilities. So candidates are sorted text before points, then smaller crop before larger, and only then by score. Duplicates are removed within a label by IoU of at least 0.52. A second rule, taken from the original one-off scripts, merges a mask that has 87% or more of its pixels inside another same-label mask at most 2.2 times its size: a kernel and the same kernel with a leaky tip. The size cap keeps real parent and part pairs apart. Every suppressed candidate is linked to the one that absorbed it.

Then come flags, which mark where to look and never delete anything: point_semantics_unverified (an anonymous grid point, not a text prompt), inside_larger_same_label_proposal (probably a part), outside_whole_object_proposals (more than 15% outside every ear), and seven more.

Where the agent actually works

So far this is a pipeline. The agent's part starts at review, and the skill spends most of its words there.

review-sheets writes side-by-side crops: original pixels on the left, the same crop with outlines on the right, at 3× zoom, with a 42-pixel header that states the native crop origin. I ran the demo, which draws 77 synthetic ellipses and pushes them through review sheets, validation and export. It finished in 2.19 s wall time on a shared 4-core VM, CPU only (measured). One of its sheets:

expand

The review protocol asks for two separate passes. First, a source inventory: the reviewer sees the photo with no masks and lists what is there. Then a crop audit: it sees the sheets and returns at most ten observations as strict JSON (missing, merged, fragmented, leakage, uncertain), each with native points. validate-review rejects a wrong image checksum or an out-of-bounds point instead of clipping it. That rule exists because a reviewer once returned an x coordinate outside the 480 px image, and another misplaced an ear fragment while reading a whole-photo montage.

Validated points become new point prompts, not accepted labels. Wrong proposals are removed with segment.py decide and a written reason, bound to the exact pixels the agent saw; if refinement later changes the mask, the decision lapses and the mask is flagged review_decision_stale. Editing selected.json by hand is forbidden.

Refinement re-prompts each kernel in a 120 px view with one positive point at its deepest interior pixel and up to three negative points on the centres of neighbouring kernels outside it. Why negatives? A click on a kernel is also a click on the ear. The lessons page measured this on twelve targets (reported): with one positive click, SAM 3's top mask was kernel-sized for 4 of 12. With three neighbour negatives, 10 of 12. An update replaces the draft only if its score, stability, agreement with the original and area change all pass gates (agreement IoU at least 0.50, area ratio 0.5 to 1.65). A kernel with no trustworthy neighbour is skipped and flagged, not given invented negatives.

What I could not run, and why

The model environment points at facebook/sam3 at a pinned revision, 3c879f3. That is the repository's current head on Hugging Face (measured), and it is gated manual: you accept Meta's terms and wait for approval. A plain request for its config.json returns 401 (measured). The skill's setup asks for about 7 GB of Python packages and a 3.5 GB checkpoint (reported). The machine I used had under 3 GB of free disk and no GPU.

Even with weights, it would not have run. sam3_backend.py raises "SAM 3 inference needs an NVIDIA GPU (--device cuda:N). CPU and macOS/MPS inference are not implemented." It also refuses any precision but BF16, because the pinned sam3 code casts its fused MLP layers to BF16 itself. The author's thread says some SAM versions run on CPU and an agent could adapt the skill. That is an exercise the skill leaves to you. I did not look for a way around the gate, and the skill itself tells the agent never to.

What I did run (measured, shared 4-core cloud VM, CPU only, nice -n 19):

The base environment took 188 MB of disk (measured). The tests cover geometry, RLE round trips, review validation, decisions and the direct-call wrapper with a fake backend. They say nothing about mask quality, and the repository does not claim they do.

Does "Skill > Model" hold?

Here is the evidence the post rests on.

The README shows the six runs as a panel of one ImageNet validation photo (ILSVRC2012_val_00000408), which is copyrighted and outside the repo's licence, so it isn't reproduced here. The README's table, all reported:

Four things are true at once.

The comparison is SAM against eyes, not skill against model. The no-tools arm "also excluded any tools or code". So the left halves are a language model drawing polygons from a look at the photo. The right halves are SAM 3 on 120-pixel tiles. That shows a segmentation model beats a language model at segmentation, which nobody doubted. The interesting control, an agent given SAM 3 and no skill, is not in the table. The closest thing is the direct-call results above, where the whole-photo prompt found nothing.

The agreement is not accuracy, and the README says so. "First run's kernels also found" means at least half of an outline overlaps one from the original 1,310-kernel run. That run was also SAM 3. No person has checked any of these masks. High agreement between six runs of one model with one procedure is what you would expect even if a few hundred of the masks were wrong in the same way.

The 98% is a range. The post says "All the models that used this skill matched 98% of the kernels that were found by Astra originally!" The table says 92.4% to 98.6%. Five of six are 97.2% or above; Haiku is the exception.

Where the agent differs, it is in review, and there it can hurt. The spread in final counts is 106 kernels, 1,244 to 1,350 (reasoned). Almost all of it is the "Removed after review" column. Luna skipped review entirely, took 8 minutes, and has the highest agreement. Haiku removed 111 proposals, "including real kernels" by the README's own account, and has the lowest. The careful middle four removed 37 to 45 each. The README also reports the same steps with no agent review at all on the CC0 photo: 1,397 kernel proposals and 11 ear proposals in about 3 minutes on one RTX 3090.

expand

So I would restate the claim. The procedure beats the model: tiling, keeping every candidate, deduplicating by mask containment, and refining with neighbour negatives turn SAM 3 from "nothing" into about 1,300 kernel outlines. The agent's contribution is running that procedure correctly and judging edge cases, and on this photo the judging moved the count by at most 8% (reasoned: 106 of 1,350). That is still "skill over model" in the sense the post means: which LLM drives it barely matters. It is not evidence that the agent loop corrects SAM's errors. Nothing here measures errors.

Beyond corn

The skill ships five non-corn trials, each one photo, each with an image-specific config (all reported). They read like an honest lab notebook:

• Dog: 2 proposals. One is the dog. The other is a patch of a background chair, and no flag catches it because the profile has no parent object for the outside-parent flag to check against.
• Insects: 117 proposals, then 100 after the containment rule merged single wings back into whole butterflies. A light reflection and a wall switch remain, flagged only as grid-point origin.
• Mushrooms: 87 caps, 85 after the rerun, with 56 flagged as outside the cluster, "almost all ground-litter leaves".
• Jars and cows: useful whole objects, plus stand supports, lids, deadwood and separate legs counted as objects.

The skill's conclusion is the right one: a point mask does not prove its category, mask IoU is not physical identity, and a stable refinement can make a wrong leaf mask look cleaner.

The repository also ships agent evals (reported, small samples). Sonnet 5.5 and Opus 5.5 loaded the skill 30 of 30 times when they should and never when they should not; Haiku 5.5, 56 of 60. With the skill, all three passed 4 of 4 runs on telling a Mac user the truth about GPU support, refusing a pasted token and stating the 11 GB download before installing; without it, 0 of 4. These test rule-following, not masks.

What it costs and where it breaks

• An NVIDIA GPU with BF16, Linux or WSL2, about 11 GB of disk and Meta's approval. No CPU path.
• Time. 8 to 42 minutes per agent on a 480×500 photo (reported), most of it review. The no-review path is about 3 minutes on an RTX 3090 for a 960×640 photo.
• Small, dense, similar objects only. The thread says it plainly: "better for high number of tightly packed objects. Not a big gain (and even a bit worse) from SAM alone for big single objects." The dog trial agrees.
• Review has no stopping rule. The changelog lists it as a known gap, along with no guidance on which reviewer to use and nothing that stops an agent from bulk-excluding unseen proposals. Haiku's 111 removals are what that gap looks like.
• It was largely written by a model. The README opens with "This was mostly written by Opus 5.5 and not thoroughly checked by a human." The parts I checked are careful: the planner reproduces its own documented numbers exactly, and the tests pass.

The design choice I would copy elsewhere is not SAM-specific. Every stage writes its raw output before filtering, every filter is counted, every human or model decision is a file with a reason bound to the pixels it saw, and the final answer has to say "proposals", not "kernels". That is how harnesses should treat any model whose output nobody has checked. It is also why the skill is useful even if you never trust its counts: you can see exactly where each one came from.

What I'd want measured

One photo with a few hundred kernels outlined by a person, held out, and four arms on it: SAM 3 whole-image, SAM 3 through the skill's pipeline with no agent, the full skill with a strong reviewer, and the full skill with a weak one. That would separate the three things the post folds into one: the model, the procedure, and the agent. My guess, labelled as one: most of the gain is the procedure, and review adds a little precision at some cost in recall. The repository has every tool needed to run that test except the labels.

For more on why the loop matters more than the model inside it, see The harness effect. For another skill read line by line and then used, see Anatomy.