The lesson is not in the skill

Every use of a decision model I have written about on this site puts it inside the loop. Stanley asks Jev which workflow a request belongs to, then runs the workflow. The xArm7 harness asks it for four signs, then moves the arm. jev-linkmap asks it whether a link belongs on a phrase, then places the link. In all three the model is in the critical path, deciding the next action while the agent is still running.

Asymptote-Labs/agent-beacon puts it after the loop. Beacon captures agent sessions across Claude Code, Cursor, Codex, OpenCode and two dozen more harnesses, normalizes them into one event stream, and then — only when you type the command — sends a finished trace to a decision model and asks whether that run contained anything worth teaching to the next agent. High scores become review candidates. Approved candidates become Agent Skills on disk.

That is a genuinely different problem, and on this site's own framework it should be an easier one. Three tiers argued the decision tier's two real advantages are latency and cost, and its two real costs are that it cannot see across its options and cannot explain itself. Judging a finished trace is offline, so latency does not matter at all. There is no next step to get wrong. The option space is a fixed rubric rather than a menu that changes per step. Half the reasons to be careful with a decision model do not apply here.

The other half is worth stating up front too, because it is where the interesting question is. Beacon prices a trace at DefaultCostPerTrace = 0.00035 and spends exactly one call on it, so a thousand traces is 35 cents. At that scale cost is not the binding constraint either, and neither is latency. Which leaves precisely one reason to reach for this tier instead of an LLM: the bounded answer with a calibrated number on it. Hold onto that. It does not survive the next function.

So I cloned it at commit 63d43f4 and read the loop.

The capture layer is excellent and I will get to it. The learning layer is 1,509 lines of Go written in a single day, and three of its choices do not survive reading.

The rubric, in full

jev-linkmap established that with these models the rubric is the product — the yes/no question and the numbers you compare its answer to are where all the behaviour lives. So the first thing to find is the literal question text. It is not in the README, and the docs paraphrase it. It is in cli/beacon/internal/learning/evaluator.go, as a package-level var.

Three questions, one request, noul type — Jev's yes/no with a calibrated probability. No examples, no confidence floor, and no per-question criteria: the two criteria strings are one sentence and its negation, sent identically with all three questions, which means every bit of instruction the model receives is the one-line prompt.

Two things about the third question. It asks whether the reusable lesson is supported by evidence — definite article, referring to the lesson the second question is being asked to decide exists. That is a chain. And the one rule the Jev Engineering guide states about batching, quoted in the three-tiers piece, is that questions cannot read one another's answers. The rubric is written as a conjunction and answered as three independent judgements, which is exactly the relational limit the middle tier has. It is not fatal here — "is this trace well evidenced" is a sensible question on its own — but the rubric is not asking what its wording implies.

The third thing is the aggregation, and it is the one that matters.

An average has no veto

evaluationScore returns the unweighted mean of whatever came back. CandidateFromEvaluation compares that mean to CandidateScoreThreshold = 0.60 and either mints a review candidate or does nothing. That is the entire application policy: one constant, one comparison, used in exactly one place in the repository.

There is no third band. A trace either becomes a candidate awaiting review or it becomes nothing — the promote/review/discard triple is really score, queue, and silence. The one thing the discard path does right is that the evaluation itself is persisted either way, so beacon memory evaluations list shows you the sub-threshold scores and the discard is at least auditable. From there a person moves each candidate to approved, rejected or superseded; those four states are the whole lifecycle. (The docs tell you to filter the queue with --state pending, which matches no state the code defines. The one you want is --state candidate.)

Mean ≥ 0.60 over three answers is the same test as sum ≥ 1.80, which makes the trades between the three visible.

reusable_correction is the only question in the rubric about reuse. It is the whole point of the product. And it has no floor of its own, so a successful, well-evidenced run carries 1.81 on its own and clears the gate with the reuse answer at zero — the model putting the probability that anything here is worth reusing at nothing at all, and the trace becoming a review candidate anyway.

I want to be fair about what this costs in practice. Everything past the gate is a human review queue, so a bad candidate wastes a person's attention, not a skill file. But the gate exists to make that queue worth reading, and a gate that promotes a certain "no" on its central question is not filtering on the thing it is named after. The fix is one line: require reusable_correction to clear its own floor before the mean is consulted.

Where the constant came from

Nothing says. 0.60 appears once, with no comment, in the commit that added the review lifecycle. It is a Go const, so it is not settable by flag, by environment variable or by config file — beacon memory evaluations run has ten flags of its own, including one for the estimated cost per trace, and none of them is this one.

That matters more than it looks, because of where the comparison sits. CandidateFromEvaluation is called inside the run loop, immediately after each model call. The evaluation is persisted with its score, but candidate creation is not a separate pass over stored evaluations. So moving the threshold is: edit the source, rebuild the binary, and re-run the scoring, paying for every trace again. In jev-linkmap a threshold sweep was free — the verdicts were on disk — and it was worth 19 of the 20 recall points a $15.51 rubric rewrite was credited with. Here the cheapest knob in the system is the one you cannot turn.

The number the model returns, and the number nobody reads

Jev's answer carries a probability and a confidence. Beacon parses the confidence, stores it, prints it in beacon memory evaluations show, and reads it in no policy anywhere in the repository. I grepped: two occurrences, one write and one Printf.

Compare Stanley's router, which this site read line by line: three tests on the shape of the returned distribution — minConfidence: 0.6, minProbability: 0.55, and a minMargin: 0.15 between the top two options — because the distribution is the only thing a decision model gives you. Beacon takes three calibrated numbers, averages them, throws the confidence away and compares the result to a constant. Three tiers argued you reach for this tier because you want a number attached. One function later, the number is a boolean.

What the model is actually shown

Now the part I did not expect.

Beacon's event model is good. Every adapter normalizes into one TraceEventV1 carrying a typed action, an actor, a command with its exit code, a file with its operation and diff, an approval with its decision, token usage, and a fidelity marker saying whether the action was observed from a vendor hook or inferred by a pattern match over prose. That last field is unusually careful engineering; most telemetry products do not distinguish a report from a reading.

BuildProjection keeps six fields.

Read that against the rubric. The first question is did the trace complete the task successfully. command.exit_code is in the struct. It is not in the projection. The second question is does the trace contain a correction — the announcement's phrase for this is "human correction signals" — and actor is in the struct, and not in the projection, and neither is approval.decision, the only decision in the whole trace that a person definitely made.

So the model is asked to infer, from prose summaries, two facts the record already holds exactly. A regex over event.action == "command.executed" && exit_code == 0 is not an approximation of the first question. It is the first question.

The figure below is the project's own dashboard, and it is why I trust the capture layer: 1,025 events off one machine, with approval.allowed and command.executed counted as first-class actions.

expand

The truncation bug proves the point

Until the day I read it, BuildProjection did this:

// cli/beacon/internal/learning/evaluator.go, before 2791dcd
events := show.Events
if len(events) > maxProjectionEvents {   // 80
    events = events[:maxProjectionEvents]
}

The first 80 events. In a long session the fix, the passing test and the correction are all at the end, so task_success was being scored on a trace that stopped before the outcome. Issue #615 reported it on 2026-09-22; commit 2791dcd fixed it the same day by keeping 40 from the head and 40 from the tail with an explicit omitted marker in between, and added a regression test. The fix is right and the turnaround is good.

What is worth noticing is how it was found: by someone reading the function. Not by a score moving. A judge scoring the wrong half of every long trace produces probabilities that look exactly like probabilities, and there is nothing in this repository that would have noticed.

What gets promoted

Say a trace clears the gate. What lands on disk?

beacon memory candidates approve mints a memory record; beacon memory skills install renders it to .agents/skills/<slug>/SKILL.md. Here is that file, for the trace visible in the project's own log-search screenshot — a Factory Droid session whose first user message was tell me about htis repo, typo included.

There is no lesson in it. Not a badly-written lesson — none. candidateBody is a five-line template with the trace id, the harness name, the session title and the three probabilities printed to two decimals. The rubric asks whether a reusable lesson exists and nothing afterwards ever asks what it is. The promoted artifact is a pointer to a trace, plus the score that made it a pointer.

Two derivations under that are worth stating on their own:

• The memory kind is a keyword switch over the first thing the user typed. candidateKind runs four strings.Contains cases over the trace title — convention|standard, gotcha|pitfall, workflow|process, fix|debug|fail — and falls through to correction. The trace title is the first user_message truncated to 80 characters. So a session that opens "fix the flaky retry test" is filed as a debugging pattern and one that opens "the retry test is flaky" is filed as a correction, and the model had no say in either.
• The skill's description is that same 80-character prompt fragment, and description is the field a future agent reads when deciding whether to load the skill. Retrieval through MCP is a case-insensitive substring match over title, body and kind, capped at five results, ordered by recency. There is no embedding anywhere.

expand

Two things this gets right that the last one got wrong

I have been hard on the learning layer, so let me be precise about what it does not do, because both are failure modes this site has documented in other projects and Beacon avoids them.

The reviewer is a person. When I wrote up jev-linkmap I first described its editor pass as a human gate. It was claude-opus-5. So I checked this one the same way. Every write to the memory store in the entire repository happens in cli/beacon/cmd/memory.go, and every one of them is reached from a cobra command a person types. The MCP server asserts its own tool list at startup and all three memory tools are reads. The dashboard's /api/memory returns 405 to anything but GET. No workflow file, no hook and no CI job calls the evaluator or the approver. The docs state the boundary and the code keeps it.

The honest qualifier: this is a gate made of absence, not of enforcement. beacon memory candidates approve is a shell command, and a coding agent with a shell can type it. Of the ten commits that built this package, five are co-authored by Cursor and one by Claude Opus 5 — so in this repository, "a human runs the command" is a convention rather than an invariant. Stanley's equivalent — an agent writes a workflow, a human runs --promote-candidate — has the same shape and the same hole.

The loop does not grade its own exam. This is the failure mode jev-linkmap actually exhibited: its System 2 loop rewrote the rubric, the referee's system prompt was built from that same rubric, and agreement improved by construction. Beacon cannot do that. RubricQuestions is a compile-time var; nothing writes to it; there is no rubric file, no rubric endpoint, no rubric rewriter. Better, every stored evaluation carries RubricHash() — a SHA-256 over the question set — and the evaluations table has a unique index on (project_id, trace_id, rubric_hash). Change one word of one prompt and every prior evaluation is automatically a different row rather than silently comparable. That is the discipline jev-linkmap needed and did not have, shipped here on day one.

What promoted skills feed is the agents' context, not the rubric. The loop is: skill lands on disk, a future agent loads it, that session becomes a trace, the same fixed rubric scores it. task_success should go up if the skill helps. That is the intended mechanism and it is not circular.

It is also unmeasured. Nothing compares scores before and after a skill is installed, and nothing stops the same lesson being re-promoted from every session that used it. A smaller version of the same gap: CandidateID hashes the candidate's body, and the body contains the three probabilities at two decimals — so re-scoring the same trace and getting 0.82 where you got 0.81 mints a second candidate for the same lesson, next to the first one you already rejected.

The capture layer is the product

"20+ harnesses" is a strong claim and I expected to find it thin. It is not. Every row of the README's runtime table has real code behind it, and the mechanisms are wildly different from each other, which is the interesting part.

A hook installer writes Beacon into ~/.claude/settings.json. The Cursor poller opens state.vscdb, the SQLite database Cursor keeps in its VS Code global storage, and walks ~/.cursor/projects/*/agent-transcripts. The browser extension monkeypatches window.fetch in the page's own JS context and tees SSE chunks to a local collector. Those are three different pieces of engineering wearing the same word. Normalizing them into one typed event is hard, and Beacon does it well enough to mark each event observed or inferred so a downstream rule can insist on the former.

Then the learning layer flattens all of it to six strings.

The baseline is in the next directory

Here is the question I came to answer: is the decision model doing work a cheap heuristic could not?

The repository contains the cheap heuristic. Not as a suggestion — as a shipped, tested subsystem. rules/ holds 75 detection rules, each a declarative expression over the same normalized event:

# rules/credential-access/password-manager-db-read.rule.yaml — abridged
match: >
  e.event.action == "file.read" && (
    e.file.path.matches("(?i)(^|/)[^/]+\.kdbx$") ||
    e.file.path.matches("(?i)(^|/)\.password-store/") || ...
  )
tests:
  - name: read_kdbx
    verdict: match
    events:
      - event: { action: file.read }
        file: { path: "/home/u/Documents/vault.kdbx" }
  - name: read_unrelated_sqlite      # eight cases in this file,
    verdict: no_match                # three of them negatives
    events:
      - event: { action: file.read }
        file: { path: "/repo/app/data/cache.sqlite" }

Every rule carries its own labelled cases, including negatives, in the same file. Across the 75 rules there are 509 of them, and the conformance suite runs in CI on macOS, Linux and Windows.

Against that, the learning rubric has zero. The learning package has 19 tests and they are all plumbing — dry run does not need an endpoint, the projection redacts secrets, the store scopes by project, the skill renders its provenance. The one test that touches the score assigns it by hand:

// cli/beacon/internal/learning/candidate_test.go
eval.Score = 0.3
if _, ok := CandidateFromEvaluation(eval); ok { t.Fatal("...") }
eval.Score = 0.9

No fixture trace. No committed evaluation. No runs/ directory, no out/. jev-linkmap, whatever else I took issue with, committed its run files, which is the only reason I was able to recompute its numbers and disagree with three of them. Beacon publishes the loop and none of its output, so nothing here is recomputable, by me or by its authors.

Which is the finding, and it is not a gotcha. The only defensible reason to pay a model for this is the part a regex cannot do: telling a reusable lesson from a task-specific fix. That distinction is real, it is genuinely hard, and it is plausibly what Jev is good at. Nothing in the repository measures whether it does it. And the two questions either side of it — did the task succeed, was there a correction — have exact answers in fields the projection deletes on the way out. So the model is being asked three questions: two it should never have been asked, and one nobody has checked it can answer.

What I would change

All four of these are cheap, and none of them needs a rubric rewrite.

1. Put the structured fields back in the projection. command.exit_code, actor, approval.decision and fidelity are four more fields per event. Better: compute task_success deterministically from the last command's exit code and stop asking. Two of three questions go away and the third gets the whole budget.
2. Give reusable_correction its own floor. An unweighted mean over a conjunction lets any one answer be zero. One if.
3. Make the threshold a flag, and re-derive candidates from stored evaluations. The scores are already in SQLite. A beacon memory candidates rebuild --min-score that re-runs the gate over persisted rows makes the cheapest experiment in the system free, which is exactly the lesson jev-linkmap paid $15.51 to learn.
4. Extract the lesson. The rubric asks whether one exists; the pipeline never asks what it is. This is the one place a reasoning model belongs — after the gate, on the small set of traces that got through, where the cost argument for the decision tier has already done its work.

And one for the reader rather than the maintainers: the capture layer is worth installing on its own. It is 44,000 lines of careful normalization across 29 runtimes, it is MIT, it runs local-first, and it does not need the memory feature to be useful. The memory feature landed on 2026-09-21. Read it before you believe its loop.

Read at commit 63d43f4 (2026-09-22) of Asymptote-Labs/agent-beacon, MIT. The two dashboard screenshots are the project's own, reproduced for commentary with the licence and a notice committed beside them in public/articles/agent-beacon/; every diagram is mine. The probabilities in the gate and skill diagrams are worked illustrations — the repository publishes no scored trace — while the thresholds, the arithmetic and the generated file's structure are read from the source.