The rubric rewrite was mostly a threshold

Last night I fixed the orphan problem on this site. Measured this morning, before this piece was written: 254 articles, 634 internal links written by hand in the prose, and 122 of those articles had nothing pointing at them — 48% of the corpus reachable only from the sitemap, which is the weakest signal a crawler has. I wrote lib/related.ts, which is 123 lines of tag arithmetic, makes zero model calls, and takes the orphan count to zero at build time.

This morning stas4000/jev-linkmap landed, attacking the same problem from the other end: crawl a 566-page site, enumerate 8,460 candidate links, and send every one of them to a decision model as a yes/no question. 5.9 seconds, $0.27, 679 links placed. Against Claude Opus 5 on the same queue and the same clock: zero pages finished when Jev was done.

It is a good project. It is MIT, it commits its run files, its README is noticeably more honest than its announcement, and the best idea in it needs no model at all. So I cloned it and recomputed every published number from out/ and runs/.

Most of them reconcile exactly, several to the sixth decimal. Four do not, and two of the four are the numbers the System 2 story rests on.

expand

The thing their method can do that mine cannot

Start with the concession, because it is the whole reason this is worth writing about.

lib/related.ts scores every pair of articles by Jaccard overlap of their tags, takes the top four, and renders a Related articles block at the foot of the page. That is all it can ever do. It cannot tell you whether a link belongs on these particular words, because it never reads the words. Every one of my 254 articles gets the same furniture in the same place, and a reader who has already decided to leave the page is the only reader who sees it.

jev-linkmap decides per candidate anchor. candidates.py finds, for each target, the two-to-six-word spans already present in the source page's copy that carry the target's topic terms, and asks the model which of them is an honest anchor for that target — or none. Text that is already a link is masked out first, so nothing gets double-linked, and nothing is ever written: every anchor is the author's own sentence.

That is a different product, and on the axis a reader actually notices it is the better one. An in-prose link at the moment the reader is thinking about the thing is worth more than a grid of four cards after the conclusion. I do not have a version of that, and tag overlap cannot get me one.

The honest failure mode on my side is concentration. Ranking is per-page and coverage is global, so my recency tie-break sends the same few recent articles to the top of many lists: on that same morning snapshot the busiest page took 41 inbound links while a long tail took none, which is exactly why a second, coverage-driven pass exists. Pass one alone left 20 orphans. Pass two places each of them on the page it matches best, with deterministic tie-breaks so the HTML does not churn between builds.

Two methods, two shapes, one shared disease. Both put most of their new edges on a small set of pages.

8,460 decisions, and 2,383 of them said yes

The announcement's arithmetic is 679 links out of 8,460 decisions, which reads as an 8% yes rate — a model being careful. I could not reproduce that reading from out/jev-results.json.

Applying rubrics/v3.json's own gate to the raw verdicts — link probability at or above 0.65, anchor not none, anchor confidence at or above 0.40 — 2,383 decisions cleared the bar. That is 28.2%. The step from 2,383 to 679 is not the model. It is judge.py:place_links enforcing max_links_per_page: 3 and one use per anchor phrase, and it discards 1,704 model yes answers. 221 of the 334 pages that got links had more than three candidates clear.

Two smaller things fall out of the same file. 8,460 is not 566 x 15, which is 8,490; two pages fall under candidates.py's min_words = 120 and are handed an empty target list, so 30 slots are never asked. And the 8,460 counts only the link questions — every target with at least one candidate phrase also carries an anchor multiple-choice, which is what the README means by "30 questions" per page. 566 requests, 8,460 link decisions, and up to 8,460 anchor decisions on top of them.

None of this makes the tool worse. A hard cap of three links per page is the right call and the README says it is there. But "679 of 8,460" is a property of a constant in a JSON file, not a measurement of the judge, and it is being quoted as the latter.

$0.27, and $17.91

The headline cost is real and it reconciles to the sixth decimal: out/report.json says cost: 0.273116 for 566 pages.

It is also the cost of the mapping pass after everything that made the mapping pass work. The 287 links that are live on the site required a System 2 loop that cost $15.51 and an editor pass that cost $2.07.

Per kept link that is $0.0624 rather than $0.000952 — 65 times the figure you get from the headline alone. And the comparison that the announcement draws, against a full Opus pass at about $67, inverts once you count the whole bill: $17.91 against $66.37 is 3.7x, not 240x.

The defence is good and I want to state it properly, because it is the right way to read this. The $15.51 is one-time per site; the $0.27 recurs. Map this site a second time and the 240x is real. The break-even is the interesting number, and it is computable: $15.51 of tuning equals the cost of mapping 32,147 pages at Jev's measured $0.000483 each. This site has 566. For a corpus this size the tuning costs 57 times more than mapping the entire thing, and you would have to re-map it 57 times before the amortisation argument starts paying.

One number is a cost, the other is a clock

The two vivid claims sit side by side and are not the same claim.

240x is cost per page. It is $0.11727 — the mean referee cost over the 120 page-judgements in runs/*/block.json — divided by Jev's $0.000483 from the full-site run. That division is 243x, so "about 240" is right, and it is if anything conservative: priced on the same 120 pages for both judges it is 278x.

There is a second figure for the same claim inside the repository, and it disagrees. out/report.json, which the README names as the file every number comes from, carries times_cheaper: 55.7 and frontier_full_pass_cost: 15.21. That one is computed from the 8 pages Opus finished during the race — and the 8 pages that finish first are the shortest pages on the site. /privacy-policy and /terms came in at 4,938 and 5,206 input tokens against a block average of 9,287. The race meter is biased low by construction, the README quotes the block meter instead, and the README is right to. But the artifact and the headline differ by 4.4x on the same claim and nothing in the repo says which to believe.

Zero pages against 566 is throughput, and it does not follow from the cost claim at all. A slower call does not cost more per page for being slower. Matched on the same 120 block pages, Opus took 11.26 s per page against Jev's 0.628 s — about 18x. Against Jev's full-site 0.319 s it is 35x. Run Opus at Jev's own concurrency of 32 instead of the 8 it was given, and a full 566-page pass lands in 3.3 minutes; at 8 workers, 13.3 minutes. Neither is 5.9 seconds, and the gap is the whole argument for a decision model. But "zero pages at second 5.9" is a restatement of "one Opus call takes longer than 5.9 seconds," and the chart makes it look like a different fact.

The referee reads the rubric it was used to write

This is the finding I would want to know about if I had shipped the repo.

The System 2 loop is the most interesting claim in the release: two rubric rewrites, for $15.51 once, take Jev from finding 45% of the referee's links to 65% and lift anchor agreement from 71% to 88%. That is a claim about the question being the product, and it is a claim this site has made repeatedly — in the negative-space piece on decision models, in the grep-by-proposition tool, and in the 245x teardown of WindTunnel's board. I wanted it to be true.

expand

Two things had to be checked first.

Is the 65% measured against Opus as ground truth? Yes. frontier_yes_found in runs/holdout-v3/block.json is both_yes / frontier_yes — 76 of the 117 decisions where the Opus referee placed a link. The ceiling on that metric is "agrees with an LLM", which is not the same as "correct", and the repository says so outright: "The referee is a strong reader, not ground truth." The announcement drops that sentence.

Does the referee see the rubric? Also yes, and this is the part that matters. judge.py builds the referee's system prompt with FRONTIER_SYSTEM.format(**rubric), interpolating the same link_instructions and anchor_instructions Jev gets, and job_state() attaches the rubric's examples to both judges' payloads. System 2's own brief says it in its own words: the lessons are "shown to both judges with every page."

So the loop rewrites the exam and the examiner together. You can see the examiner move. All three holdout runs cover the identical 24 pages and the identical 360 (source, target) pairs with the identical anchor options — I checked, the queue is frozen and hashed — and across them the Opus referee places 106, then 115, then 117 links. Between the v1 and v3 passes, 43 of the 133 decisions in their union flip; the Jaccard overlap of the two "ground truth" sets is 0.677.

With the queue frozen, the decomposition is clean. There are exactly two knobs: the rubric text, and the thresholds, which went from 0.70 / 0.50 in v1 to 0.65 / 0.40 in v2 and v3. Hold one and move the other.

The hand-written v1 rubric, with not one word changed and only the two thresholds moved to v3's values, scores 64.2% recall at 84.0% precision. The published gain is 45.3% to 65.0% at 82.6%. Almost all of the recall the $15.51 is credited with is available for nothing, at higher precision than the trained rubric achieves. Score all three against the single fixed referee pass and sum precision plus recall, which is exactly how site.py:best_rubric picks its winner: 1.281 for v1 as published, 1.482 for v1 with the thresholds moved, 1.502 for v3. The threshold move captures 91% of the total gain; the two rewrites add the last 9%.

The anchor half goes the other way, and it goes further than I expected. Against each version's own referee pass, v3's anchor agreement is 88.2% where v1's was 70.8%. Against the v1 referee pass held fixed — the only referee run made under a rubric no model wrote — v3's anchors agree 51.4%, below the rubric it replaced. The 71-to-88 improvement is not two judges converging on better anchors. It is two judges converging on each other, because somebody handed them the same six lessons.

What the editor could actually see

679 links proposed, 287 kept, 392 cut. 42.3% survival, which means 3.39% of the 8,460 decisions became a live link.

Before reading that as a verdict on the model, read verify.py. The editor is handed four fields per link: the source page's URL slug with its dashes turned into spaces and truncated at 100 characters, the one sentence the anchor sits in, the anchor itself, and the target page's title. No page copy. No description of the target. Strictly less context than Jev had when it made the call. And the last line of its system prompt is "When in doubt, cut: a missing link costs nothing, a wrong link costs trust."

That is a defensible way to build a final gate and an indefensible way to grade a model. Nobody measured the editor. Nothing in the repo checks whether the 392 cut links were actually bad.

Jev's confidence does predict the editor's verdict — monotonically, and far too weakly to act on. Mean confidence is 0.9013 across the links proposed and 0.9139 across the links kept. At the very top of the model's range the editor still cuts more than half.

The anchor is the product, and the anchors collide

Here is the one place my dumb method wins outright, and it is not an accident of implementation.

Of the 287 links the editor kept, which the README says are now live on the site, 141 use an anchor phrase that, elsewhere in the same map, points at a different page. The phrase hubspot–salesforce integration is the anchor for 34 links going to 19 distinct destinations. field mapping is the anchor for 24 links going to 10.

In-context this is often fine — a reader on a page about sync errors who clicks "field mapping" probably does want that page's field-mapping guide. But anchor text is the one thing a link says about its destination, and nineteen destinations sharing a phrase say nothing about any of them. An answer engine reading the graph learns less from 34 links than it would from 34 distinct ones.

lib/related.ts gets this right by construction and by accident: the anchor text is the target's own title, so the mapping is one-to-one, descriptive, and unique per target across all 254 articles. I did not earn that. It is what falls out of being unable to put the link in a sentence at all.

What I would take, and what I would change

The best thing in jev-linkmap is not the decision model. It is candidates.py, and it runs on the standard library.

# candidates.py — spans of 2-6 words already in page i's copy that speak page j's topic
sent = re.sub(r"⟦[^⟧]*⟧?", " | ", sent)  # already a link: not an anchor
...
inbound = Counter(k for p in pages for k in p["links_out_all"])
# a page linked from most of the site is menu chrome, a contextual link to it adds nothing
self.chrome = {k for k, c in inbound.items() if c >= 0.5 * n}

TF-IDF cosine with the page's own headings weighted 4x, a chrome filter that drops any page half the site already links to, anchor candidates restricted to text the author wrote, and already-linked spans masked so nothing is linked twice. No model touches any of it, and it is what makes the model's job small enough to be a yes/no question. I want this on my own corpus and I would build it before I bought a decision model.

Three things I would change, all of them cheap:

1. Sweep the threshold before paying for a rewrite. It is a free axis on data you already have, and here it was worth 19 points of recall. The two rewrites on top of it are worth somewhere between 1 and 6 more, depending on which referee pass you score against — and that ambiguity is itself the second problem.
2. Freeze the referee. Run the reference pass once, under a rubric nobody rewrote, and score every version against that fixed set. Between the first and last measurement a third of the referee's yes-set turned over — 43 flips in a union of 133, Jaccard 0.677 — because the loop is editing the referee's prompt too.
3. Publish the one number that answers the question. crawl.py already computes links_out — body-copy links, nav and footer stripped — which is exactly the orphan baseline. data/ is in .gitignore, so the repository cannot tell you how many of the 566 pages had no in-copy inbound link before the run or after it. Kept links reach 151 pages, 26.7% of the site. Whether that fixed the orphan problem is unknowable from the artifacts, and it is a one-line addition to report.json.

One more, which is not a criticism of the run but of the quickstart: rubrics/v3.json — the rubric the System 1 command uses by default — contains the strings HubSpot, Salesforce, reverse ETL, multi-touch attribution, and the literal fragment 'Field mapping is where most HubSpot'. It is a bles-software.com rubric. Point System 1 at your own site and Jev is reading another company's product names as its test for whether a link belongs. The README does say System 2 exists to train the rubric on your site. The one-command quickstart does not.