# The lesson is not in the skill
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/agent-beacon
> date: 2026-09-22
> tags: explainer, agents, llm, evaluation, calibration, open-source
Every use of a decision model I have written about on this site puts it **inside** the
loop. Stanley asks Jev which workflow a request belongs to, then runs the workflow.
The xArm7 harness asks it for four signs, then moves the arm. jev-linkmap asks it
whether a link belongs on a phrase, then places the link. In all three the model is in
the critical path, deciding the next action while the agent is still running.
[Asymptote-Labs/agent-beacon](https://github.com/Asymptote-Labs/agent-beacon) puts it
**after** the loop. Beacon captures agent sessions across Claude Code, Cursor, Codex,
OpenCode and two dozen more harnesses, normalizes them into one event stream, and then
— only when you type the command — sends a finished trace to a decision model and asks
whether that run contained anything worth teaching to the next agent. High scores
become review candidates. Approved candidates become Agent Skills on disk.
That is a genuinely different problem, and on this site's own framework it should be an
easier one. [Three tiers](/articles/three-tiers) argued the decision tier's two real
advantages are latency and cost, and its two real costs are that it cannot see across
its options and cannot explain itself. Judging a finished trace is offline, so latency
does not matter at all. There is no next step to get wrong. The option space is a fixed
rubric rather than a menu that changes per step. Half the reasons to be careful with a
decision model do not apply here.
The other half is worth stating up front too, because it is where the interesting
question is. Beacon prices a trace at `DefaultCostPerTrace = 0.00035` and spends exactly
one call on it, so a thousand traces is 35 cents. At that scale cost is not the binding
constraint either, and neither is latency. Which leaves precisely one reason to reach for
this tier instead of an LLM: the bounded answer with a calibrated number on it. Hold onto
that. It does not survive the next function.
So I cloned it at commit `63d43f4` and read the loop.
The capture layer is excellent and I will get to it. The learning layer is 1,509 lines
of Go written in a single day, and three of its choices do not survive reading.
## The rubric, in full
[jev-linkmap](/articles/jev-linkmap) established that with these models **the rubric is
the product** — the yes/no question and the numbers you compare its answer to are where
all the behaviour lives. So the first thing to find is the literal question text. It is
not in the README, and the docs paraphrase it. It is in
`cli/beacon/internal/learning/evaluator.go`, as a package-level `var`.
Three questions, one request, `noul` type — Jev's yes/no with a calibrated probability.
No examples, no confidence floor, and no per-question criteria: the two criteria strings
are one sentence and its negation, sent identically with all three questions, which means
every bit of instruction the model receives is the one-line prompt.
Two things about the third question. It asks whether **the reusable lesson** is
supported by evidence — definite article, referring to the lesson the *second* question
is being asked to decide exists. That is a chain. And the one rule the Jev Engineering
guide states about batching, quoted in the three-tiers piece, is that **questions cannot
read one another's answers.** The rubric is written as a conjunction and answered as
three independent judgements, which is exactly the relational limit the middle tier has.
It is not fatal here — "is this trace well evidenced" is a sensible question on its own —
but the rubric is not asking what its wording implies.
The third thing is the aggregation, and it is the one that matters.
## An average has no veto
`evaluationScore` returns the unweighted mean of whatever came back.
`CandidateFromEvaluation` compares that mean to `CandidateScoreThreshold = 0.60` and
either mints a review candidate or does nothing. That is the entire application policy:
one constant, one comparison, used in exactly one place in the repository.
There is no third band. A trace either becomes a candidate awaiting review or it becomes
nothing — the promote/review/discard triple is really *score, queue, and silence*. The
one thing the discard path does right is that the evaluation itself is persisted either
way, so `beacon memory evaluations list` shows you the sub-threshold scores and the
discard is at least auditable. From there a person moves each candidate to `approved`,
`rejected` or `superseded`; those four states are the whole lifecycle. (The docs tell you
to filter the queue with `--state pending`, which matches no state the code defines. The
one you want is `--state candidate`.)
Mean ≥ 0.60 over three answers is the same test as sum ≥ 1.80, which makes the trades
between the three visible.
`reusable_correction` is the only question in the rubric about reuse. It is the whole
point of the product. And it has no floor of its own, so a successful, well-evidenced run
carries 1.81 on its own and clears the gate with the reuse answer at **zero** — the model
putting the probability that anything here is worth reusing at nothing at all, and the
trace becoming a review candidate anyway.
I want to be fair about what this costs in practice. Everything past the gate is a human
review queue, so a bad candidate wastes a person's attention, not a skill file. But the
gate exists to make that queue worth reading, and a gate that promotes a certain "no" on
its central question is not filtering on the thing it is named after. The fix is one
line: require `reusable_correction` to clear its own floor before the mean is consulted.
### Where the constant came from
Nothing says. `0.60` appears once, with no comment, in the commit that added the review
lifecycle. It is a Go `const`, so it is not settable by flag, by environment variable or
by config file — `beacon memory evaluations run` has ten flags of its own, including one
for the *estimated cost per trace*, and none of them is this one.
That matters more than it looks, because of where the comparison sits. `CandidateFromEvaluation`
is called inside the run loop, immediately after each model call. The evaluation is
persisted with its score, but candidate creation is not a separate pass over stored
evaluations. So moving the threshold is: edit the source, rebuild the binary, and **re-run
the scoring**, paying for every trace again. In jev-linkmap a threshold sweep was free —
the verdicts were on disk — and it was worth 19 of the 20 recall points a \$15.51 rubric
rewrite was credited with. Here the cheapest knob in the system is the one you cannot
turn.
### The number the model returns, and the number nobody reads
Jev's answer carries a probability *and* a confidence. Beacon parses the confidence,
stores it, prints it in `beacon memory evaluations show`, and reads it in no policy
anywhere in the repository. I grepped: two occurrences, one write and one `Printf`.
Compare Stanley's router, which this site read line by line: three tests on the shape of
the returned distribution — `minConfidence: 0.6`, `minProbability: 0.55`, and a
`minMargin: 0.15` between the top two options — because the distribution is the only
thing a decision model gives you. Beacon takes three calibrated numbers, averages them,
throws the confidence away and compares the result to a constant. Three tiers argued you
reach for this tier *because* you want a number attached. One function later, the number
is a boolean.
There is a small honesty bug in the same area. `questionsFromJevAnswers` fills the
result's `Reason` field with `answer.Type` — the string `"noul"` — and `evaluations show`
prints that column under a header a reader will take for an explanation. It is the
three-tiers point made literal: a decision model returns a number and no reason, and if
you want a reason in your audit trail you have to build one. Beacon has a column for it
and nothing to put in it.
## What the model is actually shown
Now the part I did not expect.
Beacon's event model is good. Every adapter normalizes into one `TraceEventV1` carrying
a typed action, an actor, a command with its **exit code**, a file with its operation and
diff, an approval with its **decision**, token usage, and a `fidelity` marker saying
whether the action was observed from a vendor hook or inferred by a pattern match over
prose. That last field is unusually careful engineering; most telemetry products do not
distinguish a report from a reading.
`BuildProjection` keeps six fields.
Read that against the rubric. The first question is *did the trace complete the task
successfully*. `command.exit_code` is in the struct. It is not in the projection. The
second question is *does the trace contain a correction* — the announcement's phrase for
this is "human correction signals" — and `actor` is in the struct, and not in the
projection, and neither is `approval.decision`, the only decision in the whole trace that
a person definitely made.
So the model is asked to infer, from prose summaries, two facts the record already holds
exactly. A regex over `event.action == "command.executed" && exit_code == 0` is not an
approximation of the first question. It **is** the first question.
The figure below is the project's own dashboard, and it is why I trust the capture layer:
1,025 events off one machine, with `approval.allowed` and `command.executed` counted as
first-class actions.
### The truncation bug proves the point
Until the day I read it, `BuildProjection` did this:
```go
// cli/beacon/internal/learning/evaluator.go, before 2791dcd
events := show.Events
if len(events) > maxProjectionEvents { // 80
events = events[:maxProjectionEvents]
}
```
The first 80 events. In a long session the fix, the passing test and the correction are
all at the *end*, so `task_success` was being scored on a trace that stopped before the
outcome. Issue #615 reported it on 2026-09-22; commit `2791dcd` fixed it the same day by
keeping 40 from the head and 40 from the tail with an explicit `omitted` marker in
between, and added a regression test. The fix is right and the turnaround is good.
What is worth noticing is how it was found: by someone reading the function. Not by a
score moving. A judge scoring the wrong half of every long trace produces probabilities
that look exactly like probabilities, and there is nothing in this repository that would
have noticed.
## What gets promoted
Say a trace clears the gate. What lands on disk?
`beacon memory candidates approve` mints a memory record; `beacon memory skills install`
renders it to `.agents/skills//SKILL.md`. Here is that file, for the trace visible
in the project's own log-search screenshot — a Factory Droid session whose first user
message was `tell me about htis repo`, typo included.
There is no lesson in it. Not a badly-written lesson — none. `candidateBody` is a
five-line template with the trace id, the harness name, the session title and the three
probabilities printed to two decimals. The rubric asks whether a reusable lesson exists
and nothing afterwards ever asks **what it is**. The promoted artifact is a pointer to a
trace, plus the score that made it a pointer.
Two derivations under that are worth stating on their own:
- **The memory kind is a keyword switch over the first thing the user typed.**
`candidateKind` runs four `strings.Contains` cases over the trace title —
`convention|standard`, `gotcha|pitfall`, `workflow|process`, `fix|debug|fail` — and
falls through to `correction`. The trace title is the first `user_message` truncated
to 80 characters. So a session that opens "fix the flaky retry test" is filed as a
debugging pattern and one that opens "the retry test is flaky" is filed as a
correction, and the model had no say in either.
- **The skill's `description` is that same 80-character prompt fragment**, and
`description` is the field a future agent reads when deciding whether to load the
skill. Retrieval through MCP is a case-insensitive substring match over title, body and
kind, capped at five results, ordered by recency. There is no embedding anywhere.
## Two things this gets right that the last one got wrong
I have been hard on the learning layer, so let me be precise about what it does not do,
because both are failure modes this site has documented in other projects and Beacon
avoids them.
**The reviewer is a person.** When I wrote up jev-linkmap I first described its editor
pass as a human gate. It was `claude-opus-5`. So I checked this one the same way. Every
write to the memory store in the entire repository happens in `cli/beacon/cmd/memory.go`,
and every one of them is reached from a cobra command a person types. The MCP
server asserts its own tool list at startup and all three memory tools are reads. The
dashboard's `/api/memory` returns `405` to anything but `GET`. No workflow file, no hook
and no CI job calls the evaluator or the approver. The docs state the boundary and the
code keeps it.
The honest qualifier: this is a gate made of *absence*, not of enforcement.
`beacon memory candidates approve` is a shell command, and a coding agent with a shell
can type it. Of the ten commits that built this package, five are co-authored by Cursor
and one by Claude Opus 5 — so in this repository, "a human runs the command" is a
convention rather than an invariant. Stanley's equivalent — an agent writes a workflow, a
human runs `--promote-candidate` — has the same shape and the same hole.
**The loop does not grade its own exam.** This is the failure mode jev-linkmap actually
exhibited: its System 2 loop rewrote the rubric, the referee's system prompt was built
from that same rubric, and agreement improved by construction. Beacon cannot do that.
`RubricQuestions` is a compile-time `var`; nothing writes to it; there is no rubric file,
no rubric endpoint, no rubric rewriter. Better, every stored evaluation carries
`RubricHash()` — a SHA-256 over the question set — and the evaluations table has a unique
index on `(project_id, trace_id, rubric_hash)`. Change one word of one prompt and every
prior evaluation is automatically a different row rather than silently comparable. That
is the discipline jev-linkmap needed and did not have, shipped here on day one.
What promoted skills feed is the *agents' context*, not the rubric. The loop is: skill
lands on disk, a future agent loads it, that session becomes a trace, the same fixed
rubric scores it. `task_success` should go up if the skill helps. That is the intended
mechanism and it is not circular.
It is also unmeasured. Nothing compares scores before and after a skill is installed,
and nothing stops the same lesson being re-promoted from every session that used it. A
smaller version of the same gap: `CandidateID` hashes the candidate's body, and the body
contains the three probabilities at two decimals — so re-scoring the same trace and
getting `0.82` where you got `0.81` mints a **second** candidate for the same lesson,
next to the first one you already rejected.
## The capture layer is the product
"20+ harnesses" is a strong claim and I expected to find it thin. It is not. Every row of
the README's runtime table has real code behind it, and the mechanisms are wildly
different from each other, which is the interesting part.
**Receipts.** Beacon's headline is "20+ other harnesses", and the count holds: every row of the README's runtime table has real code behind it. What the count hides is how unlike each other the adapters are — a settings-file hook, a SQLite reader and a monkeypatched window.fetch are three different pieces of engineering, and Beacon normalises all of them into one typed event. Then the learning layer, which is the part the announcement is about, flattens that typed event down to six string fields and sends at most 80 of them to a decision model. Line counts are non-test Go (non-test TypeScript for the extension) at commit 63d43f4.
| capture mechanism | harnesses | what it actually reads | lines |
| :--- | ---: | :--- | ---: |
| Hook installers | 22 | the harness's own settings file, to register beacon-hooks with it | 7,439 |
| Hook event translators | 33 cmds | the typed payload each vendor's hook hands it on stdin | 13,382 |
| Session-log pollers | 14 + 1 cloud | files the harness wrote for itself: ~/.claude/projects/**/*.jsonl, Cursor's state.vscdb SQLite and agent-transcripts/ | 21,201 |
| OTLP configuration | 7 | the harness's own OpenTelemetry exporter, pointed at a local collector | 2,127 |
| Browser extension | 2 sites | window.fetch, monkeypatched in the page, teeing SSE chunks | 1,573 |
| rules/ threat detection | 75 rules | the full typed event — event.action, file.path, command, approval — with 509 labelled test cases | 5,598 |
| internal/learning | all of them | 6 string fields per event, at most 80 events, no exit code, no actor, no approval decision — with 0 labelled cases | 1,509 |
The two rows worth reading against each other are the last two. The rules engine and the learning rubric ask questions of the same event stream. One is a declarative expression over typed fields, shipped with 509 labelled cases that run in CI on three operating systems. The other is a hosted model call over prose, shipped with none.
> method: Counted by walking the repository at commit 63d43f4. Hook installers: files in cli/beacon/internal/endpoint/hooks/ excluding tests and the four shared files (runtime.go, settings_hooks.go, managed_extension.go, dsh_patch.go). Hook event translators: non-test commands in cli/beacon-hooks/cmd/. Session pollers: packages matching cli/beacon/internal/*session plus devincloud. OTLP: the entries of harness.DiscoverAll() whose Capability is otel_env, otel_config, otel_hooks or admin_otel. Browser: browser-extension/src. Rules: *.rule.yaml under rules/, with labelled cases counted as occurrences of 'verdict:' in those files. Learning: cli/beacon/internal/learning/ excluding _test.go.
> source: https://github.com/Asymptote-Labs/agent-beacon
> captured: 2026-09-22
> data: https://ai.thesatyajit.com/articles/agent-beacon/data/capture-census.json (7 rows)
A hook installer writes Beacon into `~/.claude/settings.json`. The Cursor poller opens
`state.vscdb`, the SQLite database Cursor keeps in its VS Code global storage, and walks
`~/.cursor/projects/*/agent-transcripts`. The browser extension monkeypatches
`window.fetch` in the page's own JS context and tees SSE chunks to a local collector.
Those are three different pieces of engineering wearing the same word. Normalizing them
into one typed event is hard, and Beacon does it well enough to mark each event
`observed` or `inferred` so a downstream rule can insist on the former.
Then the learning layer flattens all of it to six strings.
## The baseline is in the next directory
Here is the question I came to answer: **is the decision model doing work a cheap
heuristic could not?**
The repository contains the cheap heuristic. Not as a suggestion — as a shipped,
tested subsystem. `rules/` holds 75 detection rules, each a declarative expression over
the same normalized event:
```yaml
# rules/credential-access/password-manager-db-read.rule.yaml — abridged
match: >
e.event.action == "file.read" && (
e.file.path.matches("(?i)(^|/)[^/]+\.kdbx$") ||
e.file.path.matches("(?i)(^|/)\.password-store/") || ...
)
tests:
- name: read_kdbx
verdict: match
events:
- event: { action: file.read }
file: { path: "/home/u/Documents/vault.kdbx" }
- name: read_unrelated_sqlite # eight cases in this file,
verdict: no_match # three of them negatives
events:
- event: { action: file.read }
file: { path: "/repo/app/data/cache.sqlite" }
```
Every rule carries its own labelled cases, **including negatives**, in the same file.
Across the 75 rules there are **509 of them**, and the conformance suite runs in CI on
macOS, Linux and Windows.
Against that, the learning rubric has **zero**. The `learning` package has 19 tests and
they are all plumbing — dry run does not need an endpoint, the projection redacts
secrets, the store scopes by project, the skill renders its provenance. The one test that
touches the score assigns it by hand:
```go
// cli/beacon/internal/learning/candidate_test.go
eval.Score = 0.3
if _, ok := CandidateFromEvaluation(eval); ok { t.Fatal("...") }
eval.Score = 0.9
```
No fixture trace. No committed evaluation. No `runs/` directory, no `out/`. jev-linkmap,
whatever else I took issue with, committed its run files, which is the only reason I was
able to recompute its numbers and disagree with three of them. Beacon publishes the loop
and none of its output, so **nothing here is recomputable, by me or by its authors.**
Which is the finding, and it is not a gotcha. The only defensible reason to pay a model
for this is the part a regex cannot do: telling a *reusable lesson* from a *task-specific
fix*. That distinction is real, it is genuinely hard, and it is plausibly what Jev is
good at. Nothing in the repository measures whether it does it. And the two questions
either side of it — did the task succeed, was there a correction — have exact answers in
fields the projection deletes on the way out. So the model is being asked three questions:
two it should never have been asked, and one nobody has checked it can answer.
## What I would change
All four of these are cheap, and none of them needs a rubric rewrite.
1. **Put the structured fields back in the projection.** `command.exit_code`,
`actor`, `approval.decision` and `fidelity` are four more fields per event. Better:
compute `task_success` deterministically from the last command's exit code and stop
asking. Two of three questions go away and the third gets the whole budget.
2. **Give `reusable_correction` its own floor.** An unweighted mean over a conjunction
lets any one answer be zero. One `if`.
3. **Make the threshold a flag, and re-derive candidates from stored evaluations.**
The scores are already in SQLite. A `beacon memory candidates rebuild --min-score`
that re-runs the gate over persisted rows makes the cheapest experiment in the system
free, which is exactly the lesson jev-linkmap paid \$15.51 to learn.
4. **Extract the lesson.** The rubric asks whether one exists; the pipeline never asks
what it is. This is the one place a reasoning model belongs — after the gate, on the
small set of traces that got through, where the cost argument for the decision tier
has already done its work.
And one for the reader rather than the maintainers: the capture layer is worth installing
on its own. It is 44,000 lines of careful normalization across 29 runtimes, it is MIT, it
runs local-first, and it does not need the memory feature to be useful. The memory
feature landed on 2026-09-21. Read it before you believe its loop.
This is arithmetic over `evaluationScore` and `CandidateScoreThreshold = 0.60` at commit `63d43f4`: the mean of `(1.00, 0.00, 0.81)` is 0.603. It is falsified by a gate I misread — if the hosted TypeSafe endpoint returns something other than the three `noul` answers Beacon asks for, or returns a top-level `score` field, `Evaluate` overwrites the computed mean with `*resp.Score` and the aggregation I am describing never runs. I have no API key and no captured response body to check that against. Anyone with a key can settle it in one call: score a trace, run `beacon memory evaluations show --json`, and see whether `score` equals the mean of the three probabilities.
`TraceEventV1` carries `command.exit_code` and `actor`; `learning.ProjectedEvent` carries neither. The claim fails if those fields are mostly empty in practice — a harness that reports commands without exit codes, or an adapter that never sets an actor, makes the deterministic answer unavailable and the model's guess the only option going. The test is a census, not an argument: run `beacon traces --json` over a few hundred real sessions per harness and count the fraction of `command.executed` events carrying a non-null `exit_code` and the fraction of events carrying an actor. If either is low across the harnesses people actually use, the projection is dropping fields that were not there anyway.
Read against the whole tree at `63d43f4`: 19 tests in `internal/learning`, all plumbing, the only score-touching one assigning `eval.Score` by hand; no fixture trace, no committed evaluation, no run artifacts. Falsified by an artifact I did not find — a held-out set, an agreement study, anything with a denominator. It is also the falsifier I most want someone to close, because the interesting claim is the one nobody has tested: label 200 traces by hand for "contains a lesson a future agent should have", score them, and publish precision and recall against a regex baseline that fires on a user prompt following a tool call. If the model beats the regex by a wide margin, the whole design is vindicated and my objection collapses to the projection.
Traced through `candidateBody`, `RenderSkill` and `InstallSkill`. The escape hatch is editing: a memory record is stored as JSON and a person reviewing a candidate could rewrite `body` before approving. Nothing in the CLI offers that today — there is no `--body`, no `--edit`, and `ApproveCandidate` copies the candidate's body verbatim — but a maintainer could add it in an afternoon, and if they do, the artifact stops being a pointer and this criticism becomes a criticism of a version that was current for a week.
`RubricQuestions` is a compile-time `var` with no writer, and `RubricHash()` is stored on every evaluation. The second-order path is still open, though, and I have not measured it: a promoted skill changes how future agents behave, those sessions become traces, and the same rubric scores them. If skills systematically raise `task_success` without raising the quality of what is learned, the loop starts promoting the traces of agents that already read its own output — circular one level up. Install a skill, then compare score distributions on sessions before and after across the same task mix. A rise in `task_success` with no rise in `reusable_correction` is what that failure looks like.
Counted by walking the tree: 22 hook installers, 33 hook translators, 14 session pollers plus one cloud poller, 7 OTLP configurations, two browser adapters. What a file census cannot tell you is whether any of it works against the current version of each vendor's format — a session-log parser is a standing bet on someone else's private file layout, and it rots silently. The repository's `beacon-sandbox` runs 16 end-to-end scenarios, which is a long way short of 29 runtimes. The falsifier is operational: install it, use four different harnesses for a day, and check `beacon traces` against what you remember doing.
---
*Read at commit `63d43f4` (2026-09-22) of [Asymptote-Labs/agent-beacon](https://github.com/Asymptote-Labs/agent-beacon), MIT. The two dashboard screenshots are the project's own, reproduced for commentary with the licence and a notice committed beside them in `public/articles/agent-beacon/`; every diagram is mine. The probabilities in the gate and skill diagrams are worked illustrations — the repository publishes no scored trace — while the thresholds, the arithmetic and the generated file's structure are read from the source.*