Altar-1: a security model measured on recall alone

Aikido Security shipped Altar yesterday: "our first open-weight security model. Frontier-grade defensive AI, built to deploy." The pitch is sovereignty — a bank under a data-residency mandate, a hospital group, an air-gapped OT network, none of which can send unremediated findings to somebody else's inference endpoint.

A security model is a detector, and a detector is worth exactly what its evaluation says it is worth. So this is mostly an article about the evaluation. The short version: Aikido published more of it than almost anyone does — including the per-case grid, which is rare and which I will use against them — and the half they published is the half that cannot say whether the thing is deployable.

What it is, to the byte

The model card calls it "a 504B parameter Prune of GLM-5.3". The Hub's own safetensors metadata says 500,825,296,352, and summing the 39 shard headers by hand agrees.

The derivation that makes me trust the rest of this accounting: 482.0B of those 500.8B parameters are routed-expert weights spread over 168 experts per layer. Scale that bank back to 256 and add the 18.8B that is not expert weights, and the unpruned parent is 753.3B parameters — 1,506.7 GB in bf16, which is the figure Aikido publishes to the tenth of a gigabyte. The same 753B is printed in Paradigma's evaluation table, which I look at separately — for GLM 5.2, the previous point release, so read it as the family holding its size across a version rather than as a second measurement of this parent. The lineage is:

zai-org/GLM-5.3            753.3B params · BF16 · 1,506.7 GB
  └─ cyankiwi/GLM-5.3-AWQ-INT4      W4A16 routed experts · 488.2 GB
       └─ REAP prune, keep 168/256 experts
            └─ AikidoSec/altar-1    500.8B params · 327.9 GB

Two things to say plainly about that. "W4A16" describes the expert weights, not the file. 27.0B parameters were never quantised — the embeddings, the LM head, the shared expert, the three dense layers, and, interestingly, the routed experts of layers 3, 77 and 78 (the first MoE layer, the last, and the multi-token-prediction head). Those three layers alone are 38.1 GB. Add 37 GB of group scales and zero points — group_size: 32, asymmetric, one bf16 scale plus one int8 zero point per 32 weights, which is 0.75 bits of overhead on every quantised weight — and the file's effective rate is 5.24 bits per parameter.

And pruning buys memory, not speed. The card is explicit and correct about this: "Routing is untouched: 8 experts per token out of the 168 that remain, ~40B active parameters, same as the unpruned model." Deleting 88 experts does not make a token cheaper to compute. It makes the model fit on four H200s instead of six or more, which is the entire point when the constraint is "inside the customer's rack" — but a reader who sees 78.2% less storage and expects 78.2% less latency will be disappointed.

Worth noting who did what. The prune itself was performed by 0xSero, who also released 569B W4A16 and 500B EXL3 builds of the same cut; Aikido's card credits this directly. Aikido's own contribution is the calibration choice — traces from their pentesting harness plus multilingual text — and the security evaluation. For a security vendor that is the right division of labour, and saying so in the credits rather than in a footnote is to their credit.

The compression study, which nobody plotted

Aikido links a fidelity study as support for the claim that "selection matters as much as size." It is a sweep_summary.json with 16 pruning plans, and it is the most informative artefact in the release. Here it is drawn.

The claim holds, and holds hard. At a fixed 192 experts the best domain-max plan sits at 0.361 KL against the unpruned model and the frequency-selection baseline at 0.635 — a wider gap than two entire rungs of size. Random selection is 0.685, so frequency pruning is barely better than random at that depth, which is a genuinely useful negative result and exactly the thing Aikido's blog asserts in prose.

Switch to the domain view and the second result is the one that justifies the whole project. Averaged across plans, code KL rises from 0.048 to 0.20 as experts are removed; prose KL rises from 0.126 to 0.82. Pruning a mixture-of-experts costs natural-language fidelity roughly four times what it costs code. A model whose job is reading source and tool output is the right thing to prune; a model whose job is reading a French requirements document is not. The blog says this; the sweep shows it.

Now the part to notice: the sweep bottoms out at 192 experts. Altar ships 168. Every plan comparison, every selection-strategy result, every domain curve is measured two rungs above the cut that was actually released. Altar's own card publishes a single number at 168 — KL 0.506 on a "sealed 25-prompt panel" — with no sibling plan at that depth to compare it against, and no top-1 agreement figure at all, although agreement is in every row of the sweep.

The security evaluation

Here is the benchmark, in Aikido's own words: "32 known vulnerabilities across 30 repositories, with three runs per case", run through their AI Code Analysis harness. Results:

And — this is the unusual part — they published the per-case grid:

expand

Transcribing that grid back into run counts gives a check on the transcription: the mean of Altar's 32 cells is 1934/32 = 60.44%, the AWQ column is 1968/32 = 61.50%, and the BF16 column is 2099/32 = 65.59%. The count of cases with at least one hit comes out at 23, 23 and 25, which are the published coverage figures. All six reproduce, so the grid is the complete data and "average recall per run" is just the mean of those 32 cells.

Which means this is a paired comparison, and the paired view is much sharper than three averages.

Altar and the AWQ parent it was pruned from give identical results on 31 of the 32 cases. They differ on one: Code Injection 2, where the parent hit 2 runs of 3 and Altar hit 1. That single run is the entire gap. (1/3) / 32 = 1.04 percentage points — the "approximately one percentage point less average recall" in the announcement is one run, on one case, out of ninety-six. It is not a small measured difference. It is the smallest difference this benchmark is capable of expressing.

Against the full-precision parent it is five runs on five cases — Code Injection 1, Incorrect Authorization 1, Path Traversal 4, Deserialization, Incorrect Authorization 2 — with Altar losing exactly one run on each and never winning one. Twenty-seven cases tie. An exact two-sided sign test on those five discordant pairs gives p = 0.0625: consistently in one direction, not significant at 5%, and precisely the kind of result that needs more cases rather than more prose.

Aikido's own framing — "we report average recall per run separately from coverage across three runs: finding a vulnerability once is different from finding it consistently" — is exactly right, and the scope caveat that follows it is unusually honest:

This measures targeted CVE rediscovery within a pipeline that uses other models for surrounding stages. It does not measure blind discovery across an entire codebase, execute exploits to validate findings, or evaluate the fix-proposal stage.

That paragraph does most of my work for me. One more thing belongs in it.

The cell that cannot exist

Every case in this benchmark is a repository that is known to contain a specific CVE. The task is to rediscover it. So a run can be scored true positive or false negative, and that is all — there is no case whose correct answer is "there is nothing here", and therefore no true negatives and, structurally, no false positives. A run that reports the real vulnerability alongside five imagined ones scores identically to a run that reports only the real one.

Recall is half a detector. The other half is precision, and precision is the half that decides whether anybody can use the thing. This site has been here before: the confidence-gated fraud demo reported 100% precision measured on fifty negative examples, which bounds the false-positive rate no tighter than 5.8%, and at a realistic base rate that bound puts precision at 7%. The same arithmetic applies here with the negatives removed entirely — Altar's false-positive rate is not bounded at all, in either direction.

This is not hypothetical for the product. Aikido runs Altar inside an autonomous pentesting appliance across a customer fleet. At one spurious finding per repository per pass, over 30 repositories, precision on this workload lands near 39%; at three, near 18%. Those are inputs I invented and the curve is the point, not the numbers on it. A security scanner's threshold is the product, and there is no threshold in any of the published material — no confidence score, no calibration curve, no alert budget.

The release does gesture at the other end of the evidence chain: "Shortly after deployment, it identified a valid critical-severity vulnerability during a client production pentest." That is one true positive in the field, which is a real data point and also, by itself, tells you nothing about the denominator.

expand

Is the eval public

No, and it is worth being precise about what is.

Public: the weights, the pruning toolkit, the fidelity study with all 16 plans, and the per-case recall grid as an image. That last one is more than most vendors publish and it is what made the paired analysis above possible.

Not public: the 32 CVEs, the 30 repositories, the harness, the prompts, and the raw agent transcripts — the announcement says the study is shared "without publishing raw agent conversations", which is a defensible position for a firm whose benchmark is built from client work. The consequence is that nobody outside Aikido can run this benchmark, and the numbers above are unreproducible by construction. Against that, they are a comparison across 13 models on one fixed set, which makes the relative ordering more useful than the absolute values.

"Frontier-grade defensive AI" is the phrase to weigh against all this. What is substantiated: Altar retains its quantised parent's CVE-rediscovery behaviour to within one run in ninety-six, at a third less storage, on hardware a customer can own. That is a good compression result and a well-measured one. What is not substantiated by anything published: that it is good at finding vulnerabilities in the first place, rather than good at re-finding ones a harness has already been pointed at, and that its findings can be acted on without a triage queue nobody has sized.

What I would want next

1. A negative set. Thirty repositories at the commit after the fix, run through the same harness. Every finding is a false positive by construction, and the whole missing half of the matrix appears for the cost of re-running an existing benchmark.
2. Confidence, and a reliability diagram. If the pipeline emits a severity or a confidence, bucket it and plot accuracy per bucket. Without that there is no threshold to tune and no cost model to attach to one.
3. A plan at 168 experts in the sweep, next to massmax192 and freq192. It is one more row in a file that already has sixteen, and it is the only way to know whether Altar's 0.506 is a good 168-expert cut or merely a 168-expert cut.
4. More cases. Thirty-two is small enough that the difference between Altar and a full-precision 753B model is five runs, and small enough that a sign test on it does not clear 5%.