# Altar-1: a security model measured on recall alone

> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/altar-1
> date: 2026-09-22
> tags: security, evaluation, quantization, moe, calibration, open-weights, benchmarks
Aikido Security shipped [Altar](https://www.aikido.dev/blog/aikido-altar-open-weight-ai-sovereign-security)
yesterday: *"our first open-weight security model. Frontier-grade defensive AI, built to
deploy."* The pitch is sovereignty — a bank under a data-residency mandate, a hospital group, an
air-gapped OT network, none of which can send unremediated findings to somebody else's inference
endpoint.

A security model is a detector, and a detector is worth exactly what its evaluation says it is
worth. So this is mostly an article about the evaluation. The short version: **Aikido published
more of it than almost anyone does — including the per-case grid, which is rare and which I will
use against them — and the half they published is the half that cannot say whether the thing is
deployable.**

<ModelCard repo="AikidoSec/altar-1" />

## What it is, to the byte

The model card calls it "a 504B parameter Prune of GLM-5.3". The Hub's own safetensors
metadata says 500,825,296,352, and summing the 39 shard headers by hand agrees.

<WeightLedger />

The derivation that makes me trust the rest of this accounting: 482.0B of those 500.8B
parameters are routed-expert weights spread over 168 experts per layer. Scale that bank back to
256 and add the 18.8B that is not expert weights, and the unpruned parent is **753.3B
parameters — 1,506.7 GB in bf16**, which is the figure Aikido publishes to the tenth of a
gigabyte. The same 753B is printed in Paradigma's evaluation table, which I
[look at separately](/articles/limite-1b-violetto) — for GLM **5.2**, the
previous point release, so read it as the family holding its size across a
version rather than as a second measurement of this parent. The lineage is:

```text
zai-org/GLM-5.3            753.3B params · BF16 · 1,506.7 GB
  └─ cyankiwi/GLM-5.3-AWQ-INT4      W4A16 routed experts · 488.2 GB
       └─ REAP prune, keep 168/256 experts
            └─ AikidoSec/altar-1    500.8B params · 327.9 GB
```

Two things to say plainly about that. **"W4A16" describes the expert weights, not the file.**
27.0B parameters were never quantised — the embeddings, the LM head, the shared expert, the
three dense layers, and, interestingly, the routed experts of layers 3, 77 and 78 (the first
MoE layer, the last, and the multi-token-prediction head). Those three layers alone are 38.1 GB.
Add 37 GB of group scales and zero points — `group_size: 32`, asymmetric, one bf16 scale plus
one int8 zero point per 32 weights, which is 0.75 bits of overhead on every quantised weight —
and the file's effective rate is **5.24 bits per parameter**.

**And pruning buys memory, not speed.** The card is explicit and correct about this: *"Routing
is untouched: 8 experts per token out of the 168 that remain, ~40B active parameters, same as
the unpruned model."* Deleting 88 experts does not make a token cheaper to compute. It makes the
model fit on four H200s instead of six or more, which is the entire point when the constraint is
"inside the customer's rack" — but a reader who sees 78.2% less storage and expects 78.2% less
latency will be disappointed.

Worth noting who did what. The prune itself was performed by
[0xSero](https://huggingface.co/0xSero), who also released 569B W4A16 and 500B EXL3 builds of
the same cut; Aikido's card credits this directly. Aikido's own contribution is the calibration
choice — traces from their pentesting harness plus multilingual text — and the security
evaluation. For a security vendor that is the right division of labour, and saying so in the
credits rather than in a footnote is to their credit.

## The compression study, which nobody plotted

Aikido links a [fidelity study](https://huggingface.co/datasets/0xSero/glm-5.3-reap-fidelity-study)
as support for the claim that *"selection matters as much as size."* It is a `sweep_summary.json`
with 16 pruning plans, and it is the most informative artefact in the release. Here it is drawn.

<FidelitySweep />

The claim holds, and holds hard. At a fixed 192 experts the best domain-max plan sits at 0.361
KL against the unpruned model and the frequency-selection baseline at 0.635 — a wider gap than
two entire rungs of size. Random selection is 0.685, so **frequency pruning is barely better
than random at that depth**, which is a genuinely useful negative result and exactly the thing
Aikido's blog asserts in prose.

Switch to the domain view and the second result is the one that justifies the whole project.
Averaged across plans, code KL rises from 0.048 to 0.20 as experts are removed; prose KL rises
from 0.126 to **0.82**. Pruning a mixture-of-experts costs natural-language fidelity roughly
four times what it costs code. A model whose job is reading source and tool output is the right
thing to prune; a model whose job is reading a French requirements document is not. The blog
says this; the sweep shows it.

Now the part to notice: **the sweep bottoms out at 192 experts. Altar ships 168.** Every plan
comparison, every selection-strategy result, every domain curve is measured two rungs above the
cut that was actually released. Altar's own card publishes a single number at 168 — KL 0.506 on
a "sealed 25-prompt panel" — with no sibling plan at that depth to compare it against, and no
top-1 agreement figure at all, although agreement is in every row of the sweep.

<Callout type="note">
Two releases this week publish a mean KL and neither is comparable to the other. Altar's 0.506
is on a sealed 25-prompt panel over a 154,880-token vocabulary; the
[Penjing-27B quant ladder](/articles/penjing-27b) reports 0.4217 on wikitext2 at ctx 2048. Same
statistic, different corpora, different vocabularies, different model families. Mean KL is the
right thing to report and it is currently unusable for cross-release comparison, which is a
field-wide problem rather than either release's fault.
</Callout>

## The security evaluation

Here is the benchmark, in Aikido's own words: *"32 known vulnerabilities across 30
repositories, with three runs per case"*, run through their AI Code Analysis harness. Results:

| | average recall per run | vulnerabilities covered |
|---|---:|---:|
| GLM-5.3, BF16 | 65.6% | 25 of 32 |
| GLM-5.3, AWQ INT4 | 61.5% | 23 of 32 |
| **Altar-1** | **60.4%** | **23 of 32** |

And — this is the unusual part — they published the per-case grid:

<Figure
  src="/articles/altar-1/fig1.png"
  alt="A heatmap titled 'Recall by vulnerability type', showing 32 rows of vulnerability types against 13 model columns including GLM-Flash, Luna, Altar, Flash 0731, Sol, GLM-5.3, GLM-5.3 AWQ, Grok 4.6, Qwen3.8, Opus 5, Flash 4.1, Pro 0813 and Astra. Every cell is 0, 33, 67 or 100, labelled 'percent of 3 runs'. The top nine rows are almost entirely 100 across all models; the bottom eight rows, from Config Injection down to Request Smuggling, are almost entirely 0 for every model."
  caption="Per-case recall for 13 models on Aikido's internal CVE benchmark. Every cell is one of four values because there are three runs. (Aikido Security, 'Introducing Aikido Altar', 21 Sep 2026.)"
/>

Transcribing that grid back into run counts gives a check on the transcription: the mean of
Altar's 32 cells is 1934/32 = **60.44%**, the AWQ column is 1968/32 = **61.50%**, and the BF16
column is 2099/32 = **65.59%**. The count of cases with at least one hit comes out at 23, 23 and
25, which are the published coverage figures. All six reproduce, so the grid is the complete
data and "average recall per run" is just the mean of those 32 cells.

Which means this is a **paired** comparison, and the paired view is much sharper than three
averages.

<PairedRuns />

Altar and the AWQ parent it was pruned from give **identical results on 31 of the 32 cases.**
They differ on one: Code Injection 2, where the parent hit 2 runs of 3 and Altar hit 1. That
single run is the entire gap. `(1/3) / 32 = 1.04` percentage points — the "approximately one
percentage point less average recall" in the announcement is one run, on one case, out of
ninety-six. It is not a small measured difference. It is the smallest difference this benchmark
is capable of expressing.

Against the full-precision parent it is five runs on five cases — Code Injection 1, Incorrect
Authorization 1, Path Traversal 4, Deserialization, Incorrect Authorization 2 — with Altar
losing exactly one run on each and never winning one. Twenty-seven cases tie. An exact two-sided
sign test on those five discordant pairs gives **p = 0.0625**: consistently in one direction,
not significant at 5%, and precisely the kind of result that needs more cases rather than more
prose.

Aikido's own framing — *"we report average recall per run separately from coverage across three
runs: finding a vulnerability once is different from finding it consistently"* — is exactly
right, and the scope caveat that follows it is unusually honest:

> This measures targeted CVE rediscovery within a pipeline that uses other models for
> surrounding stages. It does not measure blind discovery across an entire codebase, execute
> exploits to validate findings, or evaluate the fix-proposal stage.

That paragraph does most of my work for me. One more thing belongs in it.

## The cell that cannot exist

<MissingHalf />

Every case in this benchmark is a repository that is known to contain a specific CVE. The task
is to rediscover it. So a run can be scored true positive or false negative, and that is all —
there is no case whose correct answer is "there is nothing here", and therefore no true
negatives and, structurally, **no false positives**. A run that reports the real vulnerability
alongside five imagined ones scores identically to a run that reports only the real one.

Recall is half a detector. The other half is precision, and precision is the half that decides
whether anybody can use the thing. This site has been here before: the
[confidence-gated fraud demo](/articles/confidence-gated-fraud-detection) reported 100%
precision measured on fifty negative examples, which bounds the false-positive rate no tighter
than 5.8%, and at a realistic base rate that bound puts precision at 7%. The same arithmetic
applies here with the negatives removed entirely — Altar's false-positive rate is not bounded
at all, in either direction.

This is not hypothetical for the product. Aikido runs Altar inside an autonomous pentesting
appliance across a customer fleet. At one spurious finding per repository per pass, over 30
repositories, precision on this workload lands near 39%; at three, near 18%. Those are inputs I
invented and the curve is the point, not the numbers on it. A security scanner's threshold *is*
the product, and there is no threshold in any of the published material — no confidence score,
no calibration curve, no alert budget.

The release does gesture at the other end of the evidence chain: *"Shortly after deployment, it
identified a valid critical-severity vulnerability during a client production pentest."* That is
one true positive in the field, which is a real data point and also, by itself, tells you
nothing about the denominator.

<Figure
  src="/articles/altar-1/fig2.png"
  alt="Two bar charts side by side under the heading 'Compression Impact'. Left, 'Stored Weights (GB)': GLM-5.3 BF16 full precision at 1.51 TB, GLM-5.3 AWQ INT4 at 488 GB, Altar pruned plus INT4 at 328 GB. Right, 'Vulnerabilities Covered (of 32)': GLM-5.3 BF16 at 25, GLM-5.3 AWQ INT4 at 23, Altar at 23. A caption reads '92% of the full-precision model's coverage kept, at 78.2% less storage.'"
  caption="The release's summary chart. The right-hand panel is the whole quality argument: two vulnerabilities out of 32. (Aikido Security, 'Introducing Aikido Altar', 21 Sep 2026.)"
/>

## Is the eval public

No, and it is worth being precise about what is.

Public: the weights, the [pruning toolkit](https://github.com/0xSero/model-toolkit), the
fidelity study with all 16 plans, and the per-case recall grid as an image. That last one is
more than most vendors publish and it is what made the paired analysis above possible.

Not public: the 32 CVEs, the 30 repositories, the harness, the prompts, and the raw agent
transcripts — the announcement says the study is shared "without publishing raw agent
conversations", which is a defensible position for a firm whose benchmark is built from client
work. The consequence is that nobody outside Aikido can run this benchmark, and the numbers
above are unreproducible by construction. Against that, they *are* a comparison across 13 models
on one fixed set, which makes the relative ordering more useful than the absolute values.

"Frontier-grade defensive AI" is the phrase to weigh against all this. What is substantiated:
Altar retains its quantised parent's CVE-rediscovery behaviour to within one run in ninety-six,
at a third less storage, on hardware a customer can own. That is a good compression
result and a well-measured one. What is not substantiated by anything published: that it is good
at finding vulnerabilities in the first place, rather than good at re-finding ones a harness has
already been pointed at, and that its findings can be acted on without a triage queue nobody has
sized.

## What I would want next

1. **A negative set.** Thirty repositories at the commit *after* the fix, run through the same
   harness. Every finding is a false positive by construction, and the whole missing half of the
   matrix appears for the cost of re-running an existing benchmark.
2. **Confidence, and a reliability diagram.** If the pipeline emits a severity or a confidence,
   bucket it and plot accuracy per bucket. Without that there is no threshold to tune and no
   cost model to attach to one.
3. **A plan at 168 experts in the sweep**, next to `massmax192` and `freq192`. It is one more
   row in a file that already has sixteen, and it is the only way to know whether Altar's 0.506
   is a good 168-expert cut or merely a 168-expert cut.
4. **More cases.** Thirty-two is small enough that the difference between Altar and a
   full-precision 753B model is five runs, and small enough that a sign test on it does not
   clear 5%.

<ChangeMyMind>
  <Falsifier claim="Altar-1 is 500.8B parameters at 5.24 effective bits, not 504B at four bits.">
    A recount. Mine sums all 39 safetensors shard headers, treating each `weight_packed` int32
    as eight four-bit weights and excluding `weight_scale` and `weight_zero_point` from the
    parameter count; it totals 500,825,296,352 parameters in 327,938,345,136 tensor bytes,
    against the index's own `total_size` of 327,957,368,440 — a 19 MB gap that is shard headers
    and alignment padding. If "504B" counts something else — the MTP head differently, or the
    pre-prune shape of a tensor — that is the explanation and I would want it stated.
  </Falsifier>
  <Falsifier claim="The one-percentage-point gap to the AWQ parent is a single run on a single case.">
    A per-case export that disagrees with my transcription of the heatmap. The check that it is
    right is that the 32-cell mean reproduces 60.44%, 61.50% and 65.59% against published
    figures of 60.4%, 61.5% and 65.6%, and the count of cases with at least one hit reproduces
    23, 23 and 25 — six independent hits. If the published averages
    exclude incomplete runs in a way that happens to land on the same three numbers, I am wrong
    and the raw counts would show it.
  </Falsifier>
  <Falsifier claim="The benchmark cannot measure false positives.">
    A description of the harness showing that a run is scored down for reporting vulnerabilities
    other than the target — a precision term inside the recall metric. Nothing in the
    announcement suggests one, and "completed runs without a finding count as misses" reads like
    a pure hit/miss criterion, but the harness is not public and I am inferring from the metric's
    name and the grid's shape.
  </Falsifier>
  <Falsifier claim="Pruning buys memory rather than latency.">
    A measured throughput or time-to-first-token comparison between the 488 GB AWQ parent and
    the 328 GB prune at the same batch size and context on the same four H200s. Active parameters
    are identical at ~40B, so I expect a small win from reduced memory pressure and nothing more
    — but MoE kernels are sensitive to expert-bank size in ways that do not follow from the
    active-parameter count, and a measurement could surprise me in either direction.
  </Falsifier>
  <Falsifier claim="Frequency-based expert selection is barely better than random at 192 experts.">
    This is read straight off `sweep_summary.json` at revision `ed76f87`: `freq192` at 0.635 KL
    and 75.02% agreement, `random192` at 0.685 and 73.45%. A larger evaluation panel — 25
    prompts is not many — could separate them. It would not change the ordering against
    `massmax192` at 0.361, which is the comparison that matters.
  </Falsifier>
</ChangeMyMind>