# RelateAnything: no object labels, and one gate that was told the answer

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/relate-anything
> date: 2026-09-18
> tags: scene-graphs, open-vocabulary, vision, benchmarks, explainer
Open-vocabulary detection accepts any class list at inference. Promptable segmentation returns regions with no class names at all. In both, the taxonomy stopped being a property of the model and became an argument you pass in. Relation prediction never made that move: scene-graph models are still trained and scored on the 50 predicates of VG150 or the 56 of PSG, with the relation head conditioned on predicted object classes, which welds it to one detector and one label space.

[RelateAnything](https://arxiv.org/abs/2609.12552) (Maëlic Neau, independent, arXiv 2609.12552v1, 11 Sep 2026) finishes the move. It is a 53M-parameter head that takes an image and a set of regions — from any detector, any segmenter, or ground truth — and returns scored relations over a predicate vocabulary handed to it at inference as a list of strings. Object class labels are never an input. There is no predicate classifier, so changing the vocabulary is a matrix substitution rather than a retrain.

That is a good idea and it is mostly delivered. What I went looking for is the gap between the three released artifacts and the prose describing them, and there is one worth naming: the predicate-routing gate that the paper presents as an emergent, unsupervised finding is initialised by a supervised probe fitted on a hand-derived spatial flag — and the routing table shipped inside the checkpoint is still that initialisation, separating the flag at an AUC of exactly 1.0000.

| | |
|---|---|
| Who | Maëlic Neau, independent · [paper](https://arxiv.org/abs/2609.12552) · [code](https://github.com/Maelic/RelateAnything) · [project page](https://maelic.github.io/RelateAnythingProject/) |
| What | relation prediction with the predicate vocabulary as a runtime input and no object labels anywhere |
| Size | 53.2M learned parameters (verified below), plus a 19,103 × 512 vocabulary bank that is a buffer, not a parameter |
| Backbone | DINOv3 ViT-S/16+, fully fine-tuned at 448 px, three depths fused |
| Corpus | RA-4M — 4,282,531 machine-generated relations over 10,102 free-text predicates, geometrically verified |
| Headline margin | 2.3–3.5× the mean recall of OvSGTR; 1.34–1.86× against ROBIN-3B, the paper's own stronger baseline |
| Headline latency | "20 ms/frame" is the `torch.compile` end-to-end path on an A40; the eager path the accuracy tables use is 30.5 ms |
| Out today | [`relsgg-vits16plus`](https://huggingface.co/maelic/relsgg-vits16plus) and two siblings, [RA-4M](https://huggingface.co/datasets/maelic/RA-4M), a browser demo |

<ModelCard repo="maelic/relsgg-vits16plus" />

## Scoring against a matrix you can replace

The whole architecture follows from one refusal: nothing on the text side is learned. A projection fitted to the training vocabulary would be undefined for a string supplied later, so the predicate side is a frozen bank of unit vectors and the score is a plain cosine. Swapping the vocabulary means writing a different matrix into a buffer.

<Figure
  src="/articles/relate-anything/fig1.png"
  alt="A three-band system diagram. Band 1, the visual path: one frame goes both to a region producer labelled 'any detector or segmenter' and to a fine-tuned DINOv3 backbone; a patch map fused from three depths is read by one query per box; five pooled features — subject, object, union, contact, geometry — form each ordered pair; 128 pairs of N(N−1) are kept and refined by a relation transformer and a deformable scene read. Band 2, the text path, run once offline: predicate strings go through a distilled frozen text encoder into an embedding bank, and a predicate gate reads a mixing weight alpha from each string alone, placing 'riding' and 'wearing' at the appearance-led end and 'on', 'holding' and 'above' at the geometry-led end. Band 3, training only: the InfoNCE loss on one annotated pair, and the scored relations that come out."
  caption="The architecture. The two things that make the vocabulary swappable are in band 2: a frozen text encoder with no learned projection after it, and a gate that reads its mixing weight from the string rather than from a table of known predicates. Note the label inside the gate box — 'nothing supervises it'. (Paper, Figure 2; rasterised from the project page's own architecture.svg.)"
/>

The scoring rule is Equation 2: every pair is scored twice, once from a geometry-led query and once from an appearance-led one, and the two cosines are mixed per predicate.

$$
\ell(i,j,p) \;=\; \tau\big[\alpha_p \cos(z^{\mathrm{spa}}_{ij}, e_p) + (1-\alpha_p)\cos(z^{\mathrm{sem}}_{ij}, e_p)\big] + \beta
$$

Here $e_p$ is the text embedding of predicate $p$, $z^{\mathrm{spa}}$ and $z^{\mathrm{sem}}$ are the pair's two query vectors, and $\alpha_p = g(e_p)$ is the mixing weight — read from the predicate's embedding alone, so it is defined for strings the model has never seen. The released code is that equation with nothing extra:

```python
# relsgg/model/vocab_head.py — VocabHead.score_query_dual
alpha = self.current_alpha()
cos_sem = F.normalize(q_sem, dim=-1) @ self.W.T
cos_spa = F.normalize(q_spa, dim=-1) @ self.W.T
cos = (1.0 - alpha) * cos_sem + alpha * cos_spa
scale = self.logit_scale.exp().clamp(max=100.0)
return cos * scale + self.logit_bias
```

`self.W` is the vocabulary. Installing a new one is `set_vocabulary_matrix(names, W)`, which normalises the rows and recomputes `alpha`; there is no third code path for "a predicate we didn't train on," because there is no notion of a predicate the head knows.

Two properties of the learning problem change at 19,103 predicate strings, and both fixes are real. First, the supervision has to be treated as positive-unlabeled: if a `⟨man, horse⟩` pair is annotated `riding`, then `sitting on` and `mounted on` are true but unstated, and scoring them as negatives teaches the model to suppress correct answers. The objective discounts each negative by an estimated $\hat{P}(q \mid p)$, with two exemptions the paper calls essential — the annotated predicate is never down-weighted, and directional inverses always stay full-weight negatives. Both exemptions are literally in the loader:

```python
# relsgg/training/losses.py — PredicateOntology.from_soft_supervision
np.fill_diagonal(neg_lw, 0.0)          # the annotated predicate is never discounted
# Inverse evidence ... vetoes any down-weighting: the estimator cannot tell
# opposites apart, so they must remain full negatives.
neg_lw[inv_w > 0] = 0.0
```

Second, the text space has to be repaired before it can carry direction. A contrastive encoder embeds `above` and `below` at cosine 0.95, `in front of` and `behind` at 0.94, and `to the left of` and `to the right of` at 0.99 — indistinguishable from its synonym pairs at 0.96 on average. Since the head regresses visual features onto text directions, that ceiling is the model's ceiling, and it applies to every method that scores relations against off-the-shelf text embeddings.

<Figure
  src="/articles/relate-anything/fig2.png"
  alt="Four panels. The first two are cosine-similarity matrices over twenty spatial predicates, on the same colour scale: the teacher encoder's matrix is almost uniformly dark red above 0.8, so 'above'/'below' is as close as 'above'/'on top of'; the distilled student's matrix has red synonym blocks on the diagonal and deep blue elsewhere, with the inverse blocks driven near zero. The last two are t-SNE maps under identical settings: the teacher interleaves the members of each inverse pair, while the student splits each pair into two clusters joined by a dashed line."
  caption="Why a relation model cannot use an off-the-shelf text encoder. Left pair: the teacher (dino.txt) places every spatial predicate on top of every other, so 'above' and 'below' are the same direction. Right pair: after distillation the inverse blocks fall to near zero while the synonym blocks survive — the synonym-versus-inverse AUC goes from 0.85 to 0.99, mean inverse cosine from 0.92 to 0.09, hubness from 8.1 to 1.3. (Paper, Figure 3.)"
/>

## The gate that nothing supervises

Back to $\alpha_p$. The paper's Appendix A.3 puts it plainly: *"The gate of Eq. 2 receives no supervision, and the weights it learns over the training vocabulary are bimodal and semantically ordered."* Figure 2's own diagram repeats it — *"nothing supervises it"* — and Figure 11's caption closes with *"none of this is supervised."*

I pulled the three released checkpoints apart to see what the gate actually holds. The statistics reproduce: recomputing $\alpha = \sigma(\texttt{gate\_mlp}(W))$ in float64 from the released weights gives a median of 0.0028 against the paper's 0.002, and 12.48% of the 19,103 strings above 0.5 against the paper's 12%. `behind` lands at 1.0000, `below` at 0.9998, `on` at 0.6381 — all within a rounding step of Appendix A.3. The bimodal split is real.

What the paper does not mention is where it starts. `train.sh`, the released recipe, calls `train.py` with no `--init_from`, and that path runs this:

```python
# relsgg/training/setup.py — build_model
is_spatial = union_spatial_flags(args.data_roots, pred_names)   # corpus flag, bit 0 of each relation
target = spatial_probe_alpha(model.vocab_head.W.cpu().numpy(), is_spatial)
mse = model.vocab_head.warm_start_gate(target)                  # 300 Adam steps onto that target
```

`union_spatial_flags` reads a per-predicate spatial majority straight out of the packed corpus. `spatial_probe_alpha` fits a balanced logistic regression from the text embeddings onto that binary flag. `warm_start_gate` then regresses the gate MLP onto the probe's output for 300 Adam steps. The gate has no loss term of its own during training — that much is true, and is presumably what the sentence means — but the bimodal geometry-versus-appearance split it is credited with discovering is handed to it at step zero, by a supervised fit on a label the corpus supplies.

The checkpoints make this checkable, because all three ship a stale copy of that initialisation. `vocab_head.alpha` is a buffer, recomputed from `gate_mlp` whenever a vocabulary is installed — and in every released `model.pth` the stored buffer is *not* what the stored gate computes. Correlation between the two is 0.53, the largest single disagreement is 0.998, and the two are near-identical across three independently trained towers, which a learned quantity would not be.

<GateRouting />

Toggle between the two series above. The shipped buffer separates the corpus spatial flag at an AUC of **exactly 1.0000** — all 15 flagged strings above all 228 others, no exceptions — which is the signature of a curve fitted to that flag. The gate the model actually runs scores 0.9592, because training moved 35 unflagged strings onto the geometry branch that the flag never put there: `containing`, `forming part of`, `depicting`, `accompanying`. The routing for `holding` swings from 0.17 in the buffer to 0.80 in the trained gate; `on` goes the other way, 0.93 to 0.64.

So training did real work on this gate, and the qualitative claim — projective relations to geometry, actions to appearance — holds. What does not hold is "none of this is supervised." The split was the initialisation.

<Callout type="note">
The stale buffer is a defect in the artifact, not a live bug. `set_vocabulary_matrix()` calls `_update_alpha()` on every load, so nothing in the Python API ever reads the stored value, and the 243-row deployment bank shipped beside the weights matches the recomputation to 5.9e-05. It survives only because it is never consumed — which is also why it is still readable as evidence of how the gate was started.
</Callout>

## The argument the paper is actually making

The model is the smaller half of this work. The larger half is a claim about measurement, and it is the part I would keep if I could keep only one: **scene-graph recall, as reported, is largely a prior-matching score.**

The demonstration is one table. Take the frequency baseline from Zellers et al. (2018) — for a pair of ground-truth object categories, return the most frequent training predicate. It reads no pixels. Score it and RelateAnything's zero-shot tower on the same edges, per-edge and then per-predicate.

<RecallFlip />

On micro recall — the metric leaderboards are ordered by — a lookup table that never opens the image beats a trained model by about 15% on all three benchmarks. On macro recall, over the identical predictions, the ordering inverts by 29% to 86%. The paper is careful about what this does and does not show — the edge-by-edge join finds the model right where the table is wrong on 6.9–12.8% of edges, so pixels are demonstrably doing something, and the table receives oracle object categories the model never sees. The narrow conclusion is the right one: a metric a pixel-free table can win is not measuring relation understanding.

The second prior is the one I had not seen stated this cleanly. A benchmark and a training corpus can agree on a predicate *string* and still never agree on what it is asserted of — `on` between a person and a horse and `on` between a book and a table are different acts. **Shared triplet mass** is the share of a training corpus's relation instances whose `⟨subject category, predicate, object category⟩` triple the benchmark also annotates:

| training corpus | matched on | VG150 | PSG | IndoorVG | Haystack |
|---|---|---|---|---|---|
| the released mixture (19,103 predicates) | predicate string | 53.4% | 38.7% | 49.5% | 38.7% |
| | both object categories | 44.4% | 36.7% | 26.8% | 30.7% |
| | **the whole triple** | **12.8%** | **10.7%** | **6.6%** | **4.9%** |
| VG150 train (typical baseline, 50 predicates) | predicate string | **100.0%** | 57.3% | 95.7% | 57.3% |
| | both object categories | 100.0% | 22.1% | 12.3% | 7.1% |
| | **the whole triple** | **90.9%** | 8.6% | 10.4% | 0.6% |

On VG150, the baseline's own fine-tuning corpus reproduces 90.9% of its relation mass as triples the benchmark also annotates. RelateAnything's mixture reproduces 12.8%. Micro recall tracks that statistic; the tail metrics do not. The paper's own evidence for that is the best line in it: an arm trained with a larger share of raw Visual Genome reached **54.3 R@50 on VG150, the best zero-shot figure the author is aware of, while being the worst model they trained on every tail metric.** They did not release it.

The honesty extends to the leaderboard. RelateAnything's released tower has seen all 50 VG150 predicate strings, so its rows do not satisfy the open-vocabulary protocol on the novel half; the author retrained with the 15 novel predicates and their synonym groups removed, reports 22.4 Base+Novel and 11.8 Novel R@50 — mid-table, behind every published row but one on Novel recall — and writes *"We make no claim to lead on those columns."* The 15 "novel" predicates carry 55% of the test relations, and `on`, `of` and `in` account for 93% of that novel mass by themselves, so novel recall on this split is, to within a few points, recall on three of the most frequent strings in Visual Genome.

## Numbers, with their denominators

The abstract's headline is "2.3–3.5× the mean recall of the strongest open-vocabulary method of comparable scale." That reproduces exactly against OvSGTR, from Table 4: mR@50 of 28.2 vs 10.4 on VG150 (2.71×), 30.6 vs 8.8 on PSG (3.48×), 29.5 vs 12.8 on IndoorVG (2.30×), 12.7 vs 4.5 on HICO-DET (2.82×). Rare-bucket recall is 5.4× to 21.5×, undefined on VG150 where the baseline scores exactly 0.0. All of it is cross-dataset — no benchmark contributed a training image — and the baseline receives ground-truth object labels throughout while RelateAnything receives none.

The load-bearing words are *of comparable scale*. The paper's own Table 3 names a stronger baseline on that axis: ROBIN-3B, a scene-graph model built on a 3B vision–language model, which leads OvSGTR on F1@50 on all three benchmarks both were run on. Against ROBIN the same mean-recall multiples are 28.2/15.2 = 1.86×, 30.6/22.9 = 1.34× and 29.5/21.5 = 1.37×. Still a lead on both metrics on all three, at 53.2M parameters against 3B — 1.8% of them — which is the more interesting result anyway. The abstract quotes the bigger multiple against the weaker opponent; Table 3 quotes the smaller one against the stronger. Both are in the paper, one is on the front page.

Two more places where a headline needs its footnote:

**The composite is computed against one system.** The 40.1-against-11.8 figure on the project page and in Figure 1 is a chance-corrected harmonic mean over five axes, and OvSGTR is the only system runnable on all five, so it is the only one that can carry a composite. ROBIN-3B, which beats OvSGTR where they overlap, is not in that number. The repo says outright: *"Never select a model on it."* The `OVS-F1` column in the repo's model table is a *different*, four-axis composite (37.2 for the released tower) that drops the judge-run axis — the README flags the difference, which is more than most projects do.

**A6, the spatial axis, is printed beside a number it cannot be compared to.** Table 3's row reads `SpatialSense, macro AUC (pooled AUC) — 59.1 (61.7) baseline, 69.0 (67.5) ours`, with the note *"boxes-only baseline 68.8."* Read quickly, 69.0 clears 68.8. But 68.8 is the SpatialSense authors' **accuracy** for a baseline that never sees the image, and §6.6 gives the comparable figure: *"Neither model reaches the 68.8% of the original authors' boxes-only baseline ... we reach 62.5% and the baseline 59.0%."* On the metric where the comparison is apples to apples, the model is 6.3 points behind a baseline that reads only box coordinates. The paper says so, once, in prose, and calls A6 "our weakest axis." The table does not.

The axis I trust most is A2, because it is scored against negatives an annotator actually adjudicated rather than negatives the protocol assumed: 2,870 positives against 23,174 explicit negatives on Haystack. Mean federated AP goes 52.1 → 72.6 and rare-predicate fAP 44.6 → 70.7. The denominator matters here too — the baseline scores every ordered pair while RelateAnything's sampler reaches 89.0% of the 26,044 labelled cells and the rest are scored zero, a correct rejection for a negative and a miss for a positive. That is the conservative choice, and the paper states the coverage in the table rather than in a footnote. The same section carries the sharpest single number in the paper: those identical predictions score AP 0.035 when every unlabelled pair is counted as a false positive, and fAP 0.56 against explicit negatives. A factor of sixteen, entirely in what the protocol assumes about pairs nobody looked at.

## One model behind any region source

The claim that object labels are never an input has a visible consequence, and it is the part of the paper that is most worth looking at rather than reading.

<Figure
  src="/articles/relate-anything/fig3.png"
  alt="Left: a photograph of people queueing at a food truck in a courtyard, with seven coloured instance masks numbered 0 to 6 and labelled person, food truck and courtyard. Right: the relation graph the model reads from those regions — four people 'waiting for' or 'waiting at' the food truck at scores 0.51 to 0.64, the food truck 'parked on' the courtyard at 0.60, and two people 'standing on' the courtyard at 0.46 and 0.47. A header reads '8 relations at p ≥ 0.45, 4,585-tag prompt-free regions, 19,103 predicates'."
  caption="One checkpoint, at one calibrated operating point, reading relations from an open-vocabulary segmenter prompted with 4,585 tags. 'food truck' and 'courtyard' are names no scene-graph vocabulary contains, and the names are drawn for the reader — they never reach the relation model. (Paper, Figure 6, top row.)"
/>

The paper's own caption for this figure is careful in a way I want to repeat: both panels use a proxy-scale development checkpoint, and they illustrate the interface rather than the operating quality. The companion panel, with a class-agnostic segmenter supplying twelve unnamed regions, returns `wearing clothing` and `holding` alongside `carrying person wearing` and `wearing under` — strings that are what a 19,103-way open vocabulary looks like when it degrades, not clean output.

The project page publishes a rendered reel of the same idea in motion:

<Video
  src="/articles/relate-anything/reel"
  poster="/articles/relate-anything/reel-poster.jpg"
  alt="A seated guitarist outdoors under a tree, with a speaker on a tripod to his left. In the first half, orange, blue, cyan and green boxes label man, guitar, sun hat, speaker and tripod, and white labels on dark plaques read relations between them: wearing 0.77, holding 0.74, playing 0.83, resting on 0.69, behind 0.51, beside 0.59. A tag in the bottom-left reads 'Input: boxes'. Halfway through, the boxes are replaced by per-instance segmentation masks in the same colours, the tag changes to 'Input: masks', and the relations are re-read from the new regions — playing 0.82, holding 0.74, wearing 0.77, beside 0.60."
  caption="The same model, the same frame, two region contracts: six seconds of boxes, then four of masks. Trimmed by me to seconds 9-19 of the 45-second reel on the project page and scaled to 1152 px; muted and looped here, otherwise unedited — the 'Input:' tag and the CC BY-SA credit are the source's own overlays and are not cropped. This is an offline render (deploy/render_video.py), not a live capture, so playback rate is not a frame-rate claim. Clip: David Farmer, guitarist, Copped Hall open day, by Acabashi, CC BY-SA 4.0, via the project page's reel."
/>

The measurement behind the picture is the one that matters, and it survives: with a shared YOLO-World detector supplying the regions instead of ground truth, every margin from the ground-truth table holds with the same structure — mean recall 3.3×, rare bucket 14× — because the relation model does not care where the boxes came from. What collapses is everything else. Retention relative to ground-truth boxes is 62–63% on PSG and 31–32% on IndoorVG, and when the matcher also requires the detected class to agree, 46% and 19%. A relation cannot be recovered unless both endpoints are detected, pair recall is quadratic in object recall, and no relation model can exceed it. The author's own conclusion from the backbone ladder — capacity does not raise the composite; the largest tower adds 114% of the parameters and changes it by −0.1% — is that the component to improve next is the detector.

## Twenty milliseconds, and which twenty

"It runs at 20 ms/frame" is one clause in the abstract and nine measurements in the appendix.

<TwentyMilliseconds />

The abstract's figure is the fourth row: `torch.compile`, sequential, detector and head and decode, on an A40, 20.3 ms. The eager path on the same card is 30.5 ms, and the eager path is the one every accuracy number in the paper was produced under. Table 35's own caption says so — *"compilation changes the reduction order and moves evaluation metrics by far more than the noise floor, so none of the accuracy tables uses compiled inference"* — and the repo puts it more bluntly still: *"moves evaluation metrics by 40× the noise floor, so never benchmark a compiled model."* Both facts are published. They are just published a long way from the abstract. The 20 ms configuration and the configuration the benchmark tables came from are not the same configuration.

There is one number I could not reconcile. Table 11, the comparison against OvSGTR, gives the released system end to end, batch 1, eager PyTorch, on an A40, as **25.0 ms** (40.0 FPS). Table 35 gives the same system, same batch, same card, "eager, sequential," as **30.5 ms**. The two tables differ in stated vocabulary size — Table 35 names the 243-predicate release bank, Table 11 names none — and Table 11 does not say whether the decode is included. The raw cost JSONs the tables are generated from (`runs/benchmark/cost/`) are not in the repository, so I could not settle it from the artifacts. It does not move the conclusion: at either number the system is 6.4–7.8× faster end to end than the OvSGTR baseline's 194.0 ms, and the paper is unusually forthcoming about why, namely that it scores at most 128 sampled pairs among 20 boxes at 448 px in bf16 where the baseline scores about 9,500 pairs among 98 boxes at 800/1333 in fp32. *"Under the same box budget the two would be considerably closer"* is the paper's sentence, not mine.

The best thing in the cost appendix is not a speed at all. The A100 is the *slowest* of the three GPUs at batch 1 on every configuration — 28.1 ms against 19.3 on the A40 and H100 for the relation head alone — while delivering roughly twice the A40's throughput at batch 32. It reproduces on an idle second A100 node. The head issues about 1,362 kernels per frame, so batch 1 is bound by host dispatch, and the A100 sits on a Zen3 host while the other two sit on Zen4. The conclusion generalises well past this paper: at batch 1, naming a GPU does not identify an operating point unless you also name the host CPU.

## What the artifacts say that the paper doesn't

I read the three checkpoints directly — unpickling `model.pth` without torch, since the Hub cannot introspect a `.pth` and the model cards therefore publish no parameter count.

**Receipts.** Every number RelateAnything states about its own weights, checked against the three released checkpoints by reading model.pth directly — parameter counts, the predicate-routing gate, the score scale, the training mixture. Nine of twelve reproduce. Two are rounded past what the weights say. One — the routing buffer the checkpoint ships — is not what the checkpoint's own gate computes.

| quantity | claimed | measured from the weights |  |
| :--- | ---: | ---: | :--- |
| params, ViT-S/16 tower | 46.1M | 46,139,281 | holds |
| params, ViT-S/16+ (released) | 53.2M | 53,235,601 | holds |
| params, ViT-B/16 tower | 113.8M | 113,797,393 | holds |
| tensors on disk, ViT-S/16+ | not stated | 63,035,440 | 9,780,736 of them are the 19,103x512 vocabulary bank, plus its 19,103 routing weights — buffers, not learned params |
| predicate bank width | 19,103 strings | 19,103 | holds |
| gate alpha, median over the bank | 0.002 | 0.0028 | holds |
| gate alpha, share above 0.5 | 12% | 12.48% (2,385 / 19,103) | holds |
| gate alpha, "behind" / "below" / "on" | 1.000 / 0.9998 / 0.64 | 1.0000 / 0.9998 / 0.6381 | holds |
| gate alpha, "carrying" | 0.0004 | 0.0775 | off by 190x; no released tower reproduces it (0.0536 / 0.0775 / 0.1380) |
| subject/object compose gate, trained range | 0.014-0.044 | 0.0141-0.0485 | the top of the range is 0.049, not 0.044 |
| vocab_head.alpha buffer vs. the gate that produced it | same quantity | r = 0.53, max \|diff\| = 0.998 | the shipped buffer is not what the shipped gate computes; every consumer recomputes, so it is inert |
| separation of the corpus spatial flag (243-string bank) | the gate is unsupervised | trained gate AUC 0.959, shipped buffer AUC 1.000 | an AUC of exactly 1.0 against the flag the warm-start probe was fitted on |
| vocab_head.logit_scale / logit_bias | "received no gradient ... bit-identical to init" (docs/pitfalls.md) | log-scale 2.7348 (init log 5 = 1.6094), bias -4.6971 (init 0) | both trained in all three released towers |
| training mixture, per image | 72.7 / 6.3 / 21.0 | 0.7274 / 0.063 / 0.2096 | holds — training/configs/relsgg-vits16plus.json and train.sh agree |

The three towers share one vocabulary bank byte-for-byte (19,103 x 512 float32 = 9,780,736 values in each checkpoint), which is why the same 9.8M offset separates the file size from the stated parameter count in all three. The buffer row is a defect in the artifact, not in the model: relsgg/model/vocab_head.py recomputes alpha inside set_vocabulary_matrix(), and the 243-row deployment bank shipped beside the weights matches the recomputation to 5.9e-05.

> method: model.pth for maelic/relsgg-vits16{,plus} and relsgg-vitb16 downloaded from the Hub and unpickled without torch (zipfile + a custom pickle.Unpickler that stubs _rebuild_tensor_v2, reading each storage out of archive/data/<key> as float32). alpha recomputed in float64 as sigmoid(gate_mlp(normalize(W))) with an erf GELU. AUC is the exact rank statistic over the 15 spatial-flagged and 228 unflagged strings of predicate_bank.npz.
> source: https://huggingface.co/maelic/relsgg-vits16plus
> captured: 2026-09-18
> data: https://ai.thesatyajit.com/articles/relate-anything/data/checkpoint-audit.json (14 rows)

The parameter counts are the cleanest result in the audit. The ViT-S/16+ checkpoint holds 63,035,440 float32 values; subtract the 19,103 × 512 vocabulary matrix and its 19,103 routing weights, both of which are buffers rather than learned parameters, and you get 53,235,601 — the claimed 53.2M, exactly. The same 9,780,736-value offset separates file size from claim on all three towers, which is what you would expect from three checkpoints carrying one shared bank, and it lands 46,139,281 and 113,797,393 against claims of 46.1M and 113.8M. Excluding the bank is the right call and it is the paper's own thesis restated in arithmetic: the vocabulary is an input, so it is not part of the model.

Four smaller things fell out along the way.

**The published corpus is 2,069 images smaller than the paper's corpus.** Table 20 and Figure 13 both count 474,413 training images, and the paper derives that number cleanly — 474,420 MegaSG training images, seven of which lose every candidate to the gates. The Hub's own parquet row counts for `maelic/RA-4M` are 472,344 train and 24,964 val. The relation total, 4,282,531, is identical in both, which is what makes it look carried over rather than recounted. It is 0.44% and it changes nothing, but a corpus card and its paper should agree on how many images are in the corpus.

**The benchmark is the one released artifact I could not reach.** The abstract says "Model, corpus and benchmark are public." The model repos and RA-4M resolve fine. `huggingface.co/datasets/maelic/OV-SGG-Bench`, which the README links for the evaluation packs, negatives and calibration files, returns 401 to the same unauthenticated client, today. The *protocol* is genuinely public — `benchmark/SPEC.md` and every scorer and entry point are in the repository — so this is a missing download rather than a missing method, and it may be a gate someone forgot to open.

**RA-4M is CC BY-NC 4.0, and nothing outside the dataset card says so.** The repo's licence section covers code (Apache-2.0), weights (DINOv3 licence) and annotations ("generated by Gemma, distributed with the Gemma Terms of Use notice"). The non-commercial clause on the corpus itself appears only in the dataset card's YAML. If you were planning to train on 43.8 GB of RA-4M for anything commercial, that is the line to read first.

**`docs/pitfalls.md` contradicts `docs/deployment.md` on the claim the headline latency rests on.** The README tells you to read pitfalls *"before you trust a number."* Pitfalls says `torch.compile` with `reduce-overhead` "produce[s] incorrect results in this pipeline" and that "resolution changes and plain compilation buy nothing anyway." Deployment says the `reduce-overhead` finding "was a measurement bug" and that compilation is the largest single gain measured, 1.5–1.8× on every GPU. The paper, the README and the abstract's 20 ms all side with deployment. Pitfalls is the stale file, and it is stale on exactly the mechanism the headline number depends on.

One more, on the corpus itself, because it is visible in the figure the paper chose to show the corpus at its best — and in the relation the caption singles out as the win:

<Figure
  src="/articles/relate-anything/fig4.png"
  alt="Two rows, each a photograph with numbered coloured boxes beside two lists of relations. Top row, a living room: the MegaSG source list has seven relations, six of which are 'on'; the RA-4M list has twelve, including '2 hat worn by 1 person', '5 monitor resting on 4 cabinet', '3 lamp casting light on 4 cabinet' and '8 camera capturing 5 monitor'. Bottom row, a horse-drawn carriage at a show ground: the source list has seven relations, five of them 'near'; the RA-4M list has sixteen, including '2 horse pulling 1 carriage', '5 person riding 1 carriage', '7 bench part of 1 carriage' and '2 horse behind 1 carriage'."
  caption="The same images and the same boxes, annotated by MegaSG's own pipeline and by RA-4M. The density and vocabulary gain is real: 9.03 relations per image against 5.29, 107× the vocabulary, mean object degree 1.88 to 3.20. The paper's caption names '8 camera capturing 5 monitor' as the semantics a closed schema cannot express; zoom in and object 8 is a camcorder lying flat on the cabinet top beside the television, cable trailing, lens not toward it. (Paper, Figure 16.)"
/>

That is the failure mode the paper itself names — a prior-driven label on a plausible pair, which no geometric gate can catch because geometry does not constrain `capturing`. It is a small thing to find in a showcase figure, and a large thing to find in the relation the caption picked out.

RA-4M's precision claim is structural, and the paper says so in its limitations: the geometric gate rejects 11.3% of candidates, a rejection is a true negative up to box error, and predicates geometry cannot constrain — gaze above all, where roughly 22% of `looking at` / `watching` annotations have disjoint boxes and cannot be verified — pass through unchecked and are counted rather than checked. *"We have no human-audited precision estimate, a sample of generated relations read and scored by a person. That is the measurement a reader should expect before relying on 4.3M machine-generated annotations, and the one to add first."* I agree, and I would put it above every remaining item on the roadmap.

## The result I did not expect

Section 7.3 asks whether relation supervision produces relational features — the representational premise behind every architecture that shares one backbone between object and relation prediction. The answer, measured on the author's own model, is no, and it is reported against interest.

Fine-tuning on relations raises class selectivity in the dense features from 0.242 to 0.281 and leaves relation selectivity flat, 0.133 to 0.138. The probability that a relation partner is more similar to the subject than an unrelated object *of the same class* falls from 0.469 to 0.385, against a chance level of 0.500 — relation supervision made the features **worse** at the one discrimination relations require. The interaction region does emerge where two objects meet, and retrieving the object class from it works at 0.761 against a chance level of 0.068; retrieving the verb reaches 0.269 against a majority-class chance of 0.424, which is below chance, at every depth. The region encodes which object is handled, not what is being done with it.

That result is consistent with the lesion ladder in §7.4: removing the object-identity channel from the relation logit costs 0.1% of micro accuracy on all three benchmarks, and an exact variance decomposition puts 87–93% of the semantic logit's variance in pair context against 0.1% and 0.0% for the subject and object features. Shuffle the object labels handed to OvSGTR and its output statistics do not move — its predicate is a function of the labels, not of the image.

The architectural argument follows from those two measurements rather than from taste: there is no shared representation to justify a shared backbone, and there is a measurable cost to the shared input. The author is careful about the scope — this shows that *this* recipe does not produce relational dense features, not that none could, since nothing in it supplies a patch-level relational objective. It is the kind of negative result that usually stays in a lab notebook.

## What I'd still want to know

**A human audit of RA-4M.** Everything downstream rests on 4.3M machine-generated relations whose precision has never been sampled by a person. A thousand relations, read and scored, would cost a day and would change how much weight the rest of the paper can carry.

**The gate, retrained without its warm start.** The claim that routing emerges unsupervised is falsifiable in one run: delete the probe, initialise the gate at random, train, and report the α distribution. If it still comes out bimodal with `behind` at 1.0 and `riding` at 0.01, the paper's sentence was right and only its provenance was unstated. If it does not, the finding was the initialisation.

**The 25.0-versus-30.5 ms gap.** Publishing `runs/benchmark/cost/*.json` would close it in a minute; the tables are already generated from those files.

**A general held-out-concept run.** Every benchmark predicate occurs in the training vocabulary, and the paper says so. The held-out variant exists for the VG150 leaderboard split only; the architecture permits holding concepts out across all four benchmarks, because the predicate matrix is never learned, and that is the experiment that would test the open-vocabulary claim rather than the transfer claim.

**Whether anyone else reproduces the 54.3.** The discarded arm that hit the best zero-shot VG150 recall the author is aware of, while being the worst model they trained on every tail metric, is the strongest evidence in the paper for its own thesis — and it is a sentence, not a table.

<ChangeMyMind>
  <Falsifier claim="The predicate-routing gate is warm-started by supervision, and the paper says it is not.">
    A run of `train.sh` whose logs show `build_model` taking the `--init_from` branch would mean the released recipe skips the probe. It does not: `train.sh` passes no `--init_from`, so `union_spatial_flags` → `spatial_probe_alpha` → `warm_start_gate` is the path. Separately, if the stored `vocab_head.alpha` buffer turned out to be a post-training snapshot rather than the warm start, its AUC of exactly 1.0000 against the corpus flag — and its near-identity across three independently trained towers — would need another explanation. I would take one.
  </Falsifier>
  <Falsifier claim="53.2M is the right parameter count, and the 19,103 × 512 bank is correctly excluded from it.">
    A checkpoint whose bank differed per tower, or a forward pass that updated `W` by gradient, would make the bank a parameter and the count wrong by 9.8M. Neither is the case: `W` is a `register_buffer`, the three towers carry byte-identical bank dimensions, and the arithmetic closes to the digit on all three.
  </Falsifier>
  <Falsifier claim="The abstract's 20 ms is the compiled path, not the path the accuracy tables use.">
    A row in Table 35 showing an eager configuration at or under 20 ms on any GPU would overturn this. The fastest eager row is 27.9 ms (A40, concurrent backbone); the fastest compiled row is 18.1 ms (H100). If a future revision re-measures the accuracy tables under compilation and they hold, the criticism evaporates — but the repo currently says never to do that.
  </Falsifier>
  <Falsifier claim="OV-SGG-Bench's data is not publicly downloadable as of 18 Sep 2026.">
    A 200 from `huggingface.co/datasets/maelic/OV-SGG-Bench` to an unauthenticated client. I got 401 today, from the same client that fetched RA-4M and all three model repos without trouble. This is the finding here most likely to be fixed by the time you read it, and the most trivially fixed.
  </Falsifier>
  <Falsifier claim="The headline mean-recall multiple is against OvSGTR, not against the paper's strongest baseline.">
    ROBIN-3B numbers on VG150, PSG and IndoorVG that put its mR@50 below OvSGTR's would make "strongest ... of comparable scale" and "strongest" the same claim. Table 3 reports the opposite: 15.2 / 22.9 / 21.5 for ROBIN against 10.4 / 8.8 / 12.8 for OvSGTR. The margin over ROBIN is 1.34–1.86×, and it is still a win at 1.8% of the parameters.
  </Falsifier>
</ChangeMyMind>

## The line worth keeping

> The cheapest practice we recommend is to report the shared triplet mass between corpus and benchmark beside each recall number: how much of a model's supervision asserts the same relations, of the same kinds of object, as the benchmark.

That costs a join and a percentage, and on the evidence here it would have changed how several published scene-graph results were read. The model is a good model — small, genuinely detector-agnostic, and honest about where it is behind. But the measurement argument is the part I would hand to someone else's project, and it is free.