RelateAnything: no object labels, and one gate that was told the answer

Open-vocabulary detection accepts any class list at inference. Promptable segmentation returns regions with no class names at all. In both, the taxonomy stopped being a property of the model and became an argument you pass in. Relation prediction never made that move: scene-graph models are still trained and scored on the 50 predicates of VG150 or the 56 of PSG, with the relation head conditioned on predicted object classes, which welds it to one detector and one label space.

RelateAnything (Maëlic Neau, independent, arXiv 2609.12552v1, 11 Sep 2026) finishes the move. It is a 53M-parameter head that takes an image and a set of regions — from any detector, any segmenter, or ground truth — and returns scored relations over a predicate vocabulary handed to it at inference as a list of strings. Object class labels are never an input. There is no predicate classifier, so changing the vocabulary is a matrix substitution rather than a retrain.

That is a good idea and it is mostly delivered. What I went looking for is the gap between the three released artifacts and the prose describing them, and there is one worth naming: the predicate-routing gate that the paper presents as an emergent, unsupervised finding is initialised by a supervised probe fitted on a hand-derived spatial flag — and the routing table shipped inside the checkpoint is still that initialisation, separating the flag at an AUC of exactly 1.0000.

Scoring against a matrix you can replace

The whole architecture follows from one refusal: nothing on the text side is learned. A projection fitted to the training vocabulary would be undefined for a string supplied later, so the predicate side is a frozen bank of unit vectors and the score is a plain cosine. Swapping the vocabulary means writing a different matrix into a buffer.

expand

The scoring rule is Equation 2: every pair is scored twice, once from a geometry-led query and once from an appearance-led one, and the two cosines are mixed per predicate.

Here $e_p$ is the text embedding of predicate $p$, $z^{\mathrm{spa}}$ and $z^{\mathrm{sem}}$ are the pair's two query vectors, and $\alpha_p = g(e_p)$ is the mixing weight — read from the predicate's embedding alone, so it is defined for strings the model has never seen. The released code is that equation with nothing extra:

# relsgg/model/vocab_head.py — VocabHead.score_query_dual
alpha = self.current_alpha()
cos_sem = F.normalize(q_sem, dim=-1) @ self.W.T
cos_spa = F.normalize(q_spa, dim=-1) @ self.W.T
cos = (1.0 - alpha) * cos_sem + alpha * cos_spa
scale = self.logit_scale.exp().clamp(max=100.0)
return cos * scale + self.logit_bias

self.W is the vocabulary. Installing a new one is set_vocabulary_matrix(names, W), which normalises the rows and recomputes alpha; there is no third code path for "a predicate we didn't train on," because there is no notion of a predicate the head knows.

Two properties of the learning problem change at 19,103 predicate strings, and both fixes are real. First, the supervision has to be treated as positive-unlabeled: if a ⟨man, horse⟩ pair is annotated riding, then sitting on and mounted on are true but unstated, and scoring them as negatives teaches the model to suppress correct answers. The objective discounts each negative by an estimated $\hat{P}(q \mid p)$, with two exemptions the paper calls essential — the annotated predicate is never down-weighted, and directional inverses always stay full-weight negatives. Both exemptions are literally in the loader:

# relsgg/training/losses.py — PredicateOntology.from_soft_supervision
np.fill_diagonal(neg_lw, 0.0)          # the annotated predicate is never discounted
# Inverse evidence ... vetoes any down-weighting: the estimator cannot tell
# opposites apart, so they must remain full negatives.
neg_lw[inv_w > 0] = 0.0

Second, the text space has to be repaired before it can carry direction. A contrastive encoder embeds above and below at cosine 0.95, in front of and behind at 0.94, and to the left of and to the right of at 0.99 — indistinguishable from its synonym pairs at 0.96 on average. Since the head regresses visual features onto text directions, that ceiling is the model's ceiling, and it applies to every method that scores relations against off-the-shelf text embeddings.

expand

The gate that nothing supervises

Back to $\alpha_p$. The paper's Appendix A.3 puts it plainly: "The gate of Eq. 2 receives no supervision, and the weights it learns over the training vocabulary are bimodal and semantically ordered." Figure 2's own diagram repeats it — "nothing supervises it" — and Figure 11's caption closes with "none of this is supervised."

I pulled the three released checkpoints apart to see what the gate actually holds. The statistics reproduce: recomputing $\alpha = \sigma(\texttt{gate\_mlp}(W))$ in float64 from the released weights gives a median of 0.0028 against the paper's 0.002, and 12.48% of the 19,103 strings above 0.5 against the paper's 12%. behind lands at 1.0000, below at 0.9998, on at 0.6381 — all within a rounding step of Appendix A.3. The bimodal split is real.

What the paper does not mention is where it starts. train.sh, the released recipe, calls train.py with no --init_from, and that path runs this:

# relsgg/training/setup.py — build_model
is_spatial = union_spatial_flags(args.data_roots, pred_names)   # corpus flag, bit 0 of each relation
target = spatial_probe_alpha(model.vocab_head.W.cpu().numpy(), is_spatial)
mse = model.vocab_head.warm_start_gate(target)                  # 300 Adam steps onto that target

union_spatial_flags reads a per-predicate spatial majority straight out of the packed corpus. spatial_probe_alpha fits a balanced logistic regression from the text embeddings onto that binary flag. warm_start_gate then regresses the gate MLP onto the probe's output for 300 Adam steps. The gate has no loss term of its own during training — that much is true, and is presumably what the sentence means — but the bimodal geometry-versus-appearance split it is credited with discovering is handed to it at step zero, by a supervised fit on a label the corpus supplies.

The checkpoints make this checkable, because all three ship a stale copy of that initialisation. vocab_head.alpha is a buffer, recomputed from gate_mlp whenever a vocabulary is installed — and in every released model.pth the stored buffer is not what the stored gate computes. Correlation between the two is 0.53, the largest single disagreement is 0.998, and the two are near-identical across three independently trained towers, which a learned quantity would not be.

Toggle between the two series above. The shipped buffer separates the corpus spatial flag at an AUC of exactly 1.0000 — all 15 flagged strings above all 228 others, no exceptions — which is the signature of a curve fitted to that flag. The gate the model actually runs scores 0.9592, because training moved 35 unflagged strings onto the geometry branch that the flag never put there: containing, forming part of, depicting, accompanying. The routing for holding swings from 0.17 in the buffer to 0.80 in the trained gate; on goes the other way, 0.93 to 0.64.

So training did real work on this gate, and the qualitative claim — projective relations to geometry, actions to appearance — holds. What does not hold is "none of this is supervised." The split was the initialisation.

The argument the paper is actually making

The model is the smaller half of this work. The larger half is a claim about measurement, and it is the part I would keep if I could keep only one: scene-graph recall, as reported, is largely a prior-matching score.

The demonstration is one table. Take the frequency baseline from Zellers et al. (2018) — for a pair of ground-truth object categories, return the most frequent training predicate. It reads no pixels. Score it and RelateAnything's zero-shot tower on the same edges, per-edge and then per-predicate.

On micro recall — the metric leaderboards are ordered by — a lookup table that never opens the image beats a trained model by about 15% on all three benchmarks. On macro recall, over the identical predictions, the ordering inverts by 29% to 86%. The paper is careful about what this does and does not show — the edge-by-edge join finds the model right where the table is wrong on 6.9–12.8% of edges, so pixels are demonstrably doing something, and the table receives oracle object categories the model never sees. The narrow conclusion is the right one: a metric a pixel-free table can win is not measuring relation understanding.

The second prior is the one I had not seen stated this cleanly. A benchmark and a training corpus can agree on a predicate string and still never agree on what it is asserted of — on between a person and a horse and on between a book and a table are different acts. Shared triplet mass is the share of a training corpus's relation instances whose ⟨subject category, predicate, object category⟩ triple the benchmark also annotates:

On VG150, the baseline's own fine-tuning corpus reproduces 90.9% of its relation mass as triples the benchmark also annotates. RelateAnything's mixture reproduces 12.8%. Micro recall tracks that statistic; the tail metrics do not. The paper's own evidence for that is the best line in it: an arm trained with a larger share of raw Visual Genome reached 54.3 R@50 on VG150, the best zero-shot figure the author is aware of, while being the worst model they trained on every tail metric. They did not release it.

The honesty extends to the leaderboard. RelateAnything's released tower has seen all 50 VG150 predicate strings, so its rows do not satisfy the open-vocabulary protocol on the novel half; the author retrained with the 15 novel predicates and their synonym groups removed, reports 22.4 Base+Novel and 11.8 Novel R@50 — mid-table, behind every published row but one on Novel recall — and writes "We make no claim to lead on those columns." The 15 "novel" predicates carry 55% of the test relations, and on, of and in account for 93% of that novel mass by themselves, so novel recall on this split is, to within a few points, recall on three of the most frequent strings in Visual Genome.

Numbers, with their denominators

The abstract's headline is "2.3–3.5× the mean recall of the strongest open-vocabulary method of comparable scale." That reproduces exactly against OvSGTR, from Table 4: mR@50 of 28.2 vs 10.4 on VG150 (2.71×), 30.6 vs 8.8 on PSG (3.48×), 29.5 vs 12.8 on IndoorVG (2.30×), 12.7 vs 4.5 on HICO-DET (2.82×). Rare-bucket recall is 5.4× to 21.5×, undefined on VG150 where the baseline scores exactly 0.0. All of it is cross-dataset — no benchmark contributed a training image — and the baseline receives ground-truth object labels throughout while RelateAnything receives none.

The load-bearing words are of comparable scale. The paper's own Table 3 names a stronger baseline on that axis: ROBIN-3B, a scene-graph model built on a 3B vision–language model, which leads OvSGTR on F1@50 on all three benchmarks both were run on. Against ROBIN the same mean-recall multiples are 28.2/15.2 = 1.86×, 30.6/22.9 = 1.34× and 29.5/21.5 = 1.37×. Still a lead on both metrics on all three, at 53.2M parameters against 3B — 1.8% of them — which is the more interesting result anyway. The abstract quotes the bigger multiple against the weaker opponent; Table 3 quotes the smaller one against the stronger. Both are in the paper, one is on the front page.

Two more places where a headline needs its footnote:

The composite is computed against one system. The 40.1-against-11.8 figure on the project page and in Figure 1 is a chance-corrected harmonic mean over five axes, and OvSGTR is the only system runnable on all five, so it is the only one that can carry a composite. ROBIN-3B, which beats OvSGTR where they overlap, is not in that number. The repo says outright: "Never select a model on it." The OVS-F1 column in the repo's model table is a different, four-axis composite (37.2 for the released tower) that drops the judge-run axis — the README flags the difference, which is more than most projects do.

A6, the spatial axis, is printed beside a number it cannot be compared to. Table 3's row reads SpatialSense, macro AUC (pooled AUC) — 59.1 (61.7) baseline, 69.0 (67.5) ours, with the note "boxes-only baseline 68.8." Read quickly, 69.0 clears 68.8. But 68.8 is the SpatialSense authors' accuracy for a baseline that never sees the image, and §6.6 gives the comparable figure: "Neither model reaches the 68.8% of the original authors' boxes-only baseline ... we reach 62.5% and the baseline 59.0%." On the metric where the comparison is apples to apples, the model is 6.3 points behind a baseline that reads only box coordinates. The paper says so, once, in prose, and calls A6 "our weakest axis." The table does not.

The axis I trust most is A2, because it is scored against negatives an annotator actually adjudicated rather than negatives the protocol assumed: 2,870 positives against 23,174 explicit negatives on Haystack. Mean federated AP goes 52.1 → 72.6 and rare-predicate fAP 44.6 → 70.7. The denominator matters here too — the baseline scores every ordered pair while RelateAnything's sampler reaches 89.0% of the 26,044 labelled cells and the rest are scored zero, a correct rejection for a negative and a miss for a positive. That is the conservative choice, and the paper states the coverage in the table rather than in a footnote. The same section carries the sharpest single number in the paper: those identical predictions score AP 0.035 when every unlabelled pair is counted as a false positive, and fAP 0.56 against explicit negatives. A factor of sixteen, entirely in what the protocol assumes about pairs nobody looked at.

One model behind any region source

The claim that object labels are never an input has a visible consequence, and it is the part of the paper that is most worth looking at rather than reading.

expand

The paper's own caption for this figure is careful in a way I want to repeat: both panels use a proxy-scale development checkpoint, and they illustrate the interface rather than the operating quality. The companion panel, with a class-agnostic segmenter supplying twelve unnamed regions, returns wearing clothing and holding alongside carrying person wearing and wearing under — strings that are what a 19,103-way open vocabulary looks like when it degrades, not clean output.

The project page publishes a rendered reel of the same idea in motion:

The measurement behind the picture is the one that matters, and it survives: with a shared YOLO-World detector supplying the regions instead of ground truth, every margin from the ground-truth table holds with the same structure — mean recall 3.3×, rare bucket 14× — because the relation model does not care where the boxes came from. What collapses is everything else. Retention relative to ground-truth boxes is 62–63% on PSG and 31–32% on IndoorVG, and when the matcher also requires the detected class to agree, 46% and 19%. A relation cannot be recovered unless both endpoints are detected, pair recall is quadratic in object recall, and no relation model can exceed it. The author's own conclusion from the backbone ladder — capacity does not raise the composite; the largest tower adds 114% of the parameters and changes it by −0.1% — is that the component to improve next is the detector.

Twenty milliseconds, and which twenty

"It runs at 20 ms/frame" is one clause in the abstract and nine measurements in the appendix.

The abstract's figure is the fourth row: torch.compile, sequential, detector and head and decode, on an A40, 20.3 ms. The eager path on the same card is 30.5 ms, and the eager path is the one every accuracy number in the paper was produced under. Table 35's own caption says so — "compilation changes the reduction order and moves evaluation metrics by far more than the noise floor, so none of the accuracy tables uses compiled inference" — and the repo puts it more bluntly still: "moves evaluation metrics by 40× the noise floor, so never benchmark a compiled model." Both facts are published. They are just published a long way from the abstract. The 20 ms configuration and the configuration the benchmark tables came from are not the same configuration.

There is one number I could not reconcile. Table 11, the comparison against OvSGTR, gives the released system end to end, batch 1, eager PyTorch, on an A40, as 25.0 ms (40.0 FPS). Table 35 gives the same system, same batch, same card, "eager, sequential," as 30.5 ms. The two tables differ in stated vocabulary size — Table 35 names the 243-predicate release bank, Table 11 names none — and Table 11 does not say whether the decode is included. The raw cost JSONs the tables are generated from (runs/benchmark/cost/) are not in the repository, so I could not settle it from the artifacts. It does not move the conclusion: at either number the system is 6.4–7.8× faster end to end than the OvSGTR baseline's 194.0 ms, and the paper is unusually forthcoming about why, namely that it scores at most 128 sampled pairs among 20 boxes at 448 px in bf16 where the baseline scores about 9,500 pairs among 98 boxes at 800/1333 in fp32. "Under the same box budget the two would be considerably closer" is the paper's sentence, not mine.

The best thing in the cost appendix is not a speed at all. The A100 is the slowest of the three GPUs at batch 1 on every configuration — 28.1 ms against 19.3 on the A40 and H100 for the relation head alone — while delivering roughly twice the A40's throughput at batch 32. It reproduces on an idle second A100 node. The head issues about 1,362 kernels per frame, so batch 1 is bound by host dispatch, and the A100 sits on a Zen3 host while the other two sit on Zen4. The conclusion generalises well past this paper: at batch 1, naming a GPU does not identify an operating point unless you also name the host CPU.

What the artifacts say that the paper doesn't

I read the three checkpoints directly — unpickling model.pth without torch, since the Hub cannot introspect a .pth and the model cards therefore publish no parameter count.

The parameter counts are the cleanest result in the audit. The ViT-S/16+ checkpoint holds 63,035,440 float32 values; subtract the 19,103 × 512 vocabulary matrix and its 19,103 routing weights, both of which are buffers rather than learned parameters, and you get 53,235,601 — the claimed 53.2M, exactly. The same 9,780,736-value offset separates file size from claim on all three towers, which is what you would expect from three checkpoints carrying one shared bank, and it lands 46,139,281 and 113,797,393 against claims of 46.1M and 113.8M. Excluding the bank is the right call and it is the paper's own thesis restated in arithmetic: the vocabulary is an input, so it is not part of the model.

Four smaller things fell out along the way.

The published corpus is 2,069 images smaller than the paper's corpus. Table 20 and Figure 13 both count 474,413 training images, and the paper derives that number cleanly — 474,420 MegaSG training images, seven of which lose every candidate to the gates. The Hub's own parquet row counts for maelic/RA-4M are 472,344 train and 24,964 val. The relation total, 4,282,531, is identical in both, which is what makes it look carried over rather than recounted. It is 0.44% and it changes nothing, but a corpus card and its paper should agree on how many images are in the corpus.

The benchmark is the one released artifact I could not reach. The abstract says "Model, corpus and benchmark are public." The model repos and RA-4M resolve fine. huggingface.co/datasets/maelic/OV-SGG-Bench, which the README links for the evaluation packs, negatives and calibration files, returns 401 to the same unauthenticated client, today. The protocol is genuinely public — benchmark/SPEC.md and every scorer and entry point are in the repository — so this is a missing download rather than a missing method, and it may be a gate someone forgot to open.

RA-4M is CC BY-NC 4.0, and nothing outside the dataset card says so. The repo's licence section covers code (Apache-2.0), weights (DINOv3 licence) and annotations ("generated by Gemma, distributed with the Gemma Terms of Use notice"). The non-commercial clause on the corpus itself appears only in the dataset card's YAML. If you were planning to train on 43.8 GB of RA-4M for anything commercial, that is the line to read first.

docs/pitfalls.md contradicts docs/deployment.md on the claim the headline latency rests on. The README tells you to read pitfalls "before you trust a number." Pitfalls says torch.compile with reduce-overhead "produce[s] incorrect results in this pipeline" and that "resolution changes and plain compilation buy nothing anyway." Deployment says the reduce-overhead finding "was a measurement bug" and that compilation is the largest single gain measured, 1.5–1.8× on every GPU. The paper, the README and the abstract's 20 ms all side with deployment. Pitfalls is the stale file, and it is stale on exactly the mechanism the headline number depends on.

One more, on the corpus itself, because it is visible in the figure the paper chose to show the corpus at its best — and in the relation the caption singles out as the win:

expand

That is the failure mode the paper itself names — a prior-driven label on a plausible pair, which no geometric gate can catch because geometry does not constrain capturing. It is a small thing to find in a showcase figure, and a large thing to find in the relation the caption picked out.

RA-4M's precision claim is structural, and the paper says so in its limitations: the geometric gate rejects 11.3% of candidates, a rejection is a true negative up to box error, and predicates geometry cannot constrain — gaze above all, where roughly 22% of looking at / watching annotations have disjoint boxes and cannot be verified — pass through unchecked and are counted rather than checked. "We have no human-audited precision estimate, a sample of generated relations read and scored by a person. That is the measurement a reader should expect before relying on 4.3M machine-generated annotations, and the one to add first." I agree, and I would put it above every remaining item on the roadmap.

The result I did not expect

Section 7.3 asks whether relation supervision produces relational features — the representational premise behind every architecture that shares one backbone between object and relation prediction. The answer, measured on the author's own model, is no, and it is reported against interest.

Fine-tuning on relations raises class selectivity in the dense features from 0.242 to 0.281 and leaves relation selectivity flat, 0.133 to 0.138. The probability that a relation partner is more similar to the subject than an unrelated object of the same class falls from 0.469 to 0.385, against a chance level of 0.500 — relation supervision made the features worse at the one discrimination relations require. The interaction region does emerge where two objects meet, and retrieving the object class from it works at 0.761 against a chance level of 0.068; retrieving the verb reaches 0.269 against a majority-class chance of 0.424, which is below chance, at every depth. The region encodes which object is handled, not what is being done with it.

That result is consistent with the lesion ladder in §7.4: removing the object-identity channel from the relation logit costs 0.1% of micro accuracy on all three benchmarks, and an exact variance decomposition puts 87–93% of the semantic logit's variance in pair context against 0.1% and 0.0% for the subject and object features. Shuffle the object labels handed to OvSGTR and its output statistics do not move — its predicate is a function of the labels, not of the image.

The architectural argument follows from those two measurements rather than from taste: there is no shared representation to justify a shared backbone, and there is a measurable cost to the shared input. The author is careful about the scope — this shows that this recipe does not produce relational dense features, not that none could, since nothing in it supplies a patch-level relational objective. It is the kind of negative result that usually stays in a lab notebook.

What I'd still want to know

A human audit of RA-4M. Everything downstream rests on 4.3M machine-generated relations whose precision has never been sampled by a person. A thousand relations, read and scored, would cost a day and would change how much weight the rest of the paper can carry.

The gate, retrained without its warm start. The claim that routing emerges unsupervised is falsifiable in one run: delete the probe, initialise the gate at random, train, and report the α distribution. If it still comes out bimodal with behind at 1.0 and riding at 0.01, the paper's sentence was right and only its provenance was unstated. If it does not, the finding was the initialisation.

The 25.0-versus-30.5 ms gap. Publishing runs/benchmark/cost/*.json would close it in a minute; the tables are already generated from those files.

A general held-out-concept run. Every benchmark predicate occurs in the training vocabulary, and the paper says so. The held-out variant exists for the VG150 leaderboard split only; the architecture permits holding concepts out across all four benchmarks, because the predicate matrix is never learned, and that is the experiment that would test the open-vocabulary claim rather than the transfer claim.

Whether anyone else reproduces the 54.3. The discarded arm that hit the best zero-shot VG150 recall the author is aware of, while being the worst model they trained on every tail metric, is the strongest evidence in the paper for its own thesis — and it is a sentence, not a table.

The line worth keeping

The cheapest practice we recommend is to report the shared triplet mass between corpus and benchmark beside each recall number: how much of a model's supervision asserts the same relations, of the same kinds of object, as the benchmark.

That costs a join and a percentage, and on the evidence here it would have changed how several published scene-graph results were read. The model is a good model — small, genuinely detector-agnostic, and honest about where it is behind. But the measurement argument is the part I would hand to someone else's project, and it is free.