2026-10-10 · 12 min · security · benchmarks · agents · fine-tuning
Why read this
Solidtop 85%Each OrcaCyber claim traced to its benchmark's code and leaderboard; the CyberGym run decomposed by the harness report's own token table.
- Original analysis
- Concrete numbers to act on
- Explained from first principles
Security & safetyAPI onlyProprietaryPractitioner model
How this was scored
- Is it new?
- 1 of 3: An incremental tweak
- Can I trust it?
- 2 of 3: Measures key facts from files, code or configs
- Can I run it?
- 0 of 3: Closed, nothing to run
- Will I understand it?
- 2 of 3: Mechanism from first principles with figures
- Can I act on it?
- 2 of 3: A concrete recipe, numbers or comparison
- Will it last?
- 1 of 3: Relevant for months
- Does it affect many?
- 1 of 3: A specialist community
- Only here?
- 2 of 3: A teardown or measurement few others did
Score 51 of 100, ranked 382 of 494 rated articles. Each question is answered 0–3 by hand, and a 3 is rare. How articles are scored
This morning OrcaRouter announced OrcaCyber Zero 1.5, "our latest frontier cybersecurity model, built to discover critical vulnerabilities, investigate complex attack paths, and validate real-world exploitability." The post lists four results: 95.8% on CVE-Bench "(23/24 evaluable tasks)", 100% on Cybench "(39/39, unrestricted)", 93.9% on HumanEval+, and, for the system its predecessor powered, "98%+ pass@1 on CyberGym, ranking among the top 3 globally." It is "Available now exclusively on OrcaRouter."
This site has run into OrcaRouter before, mostly as a publisher of refusal-ablated open models: GLM-5.3-Flash-Uncensored, the runtime abliteration of Ternary Bonsai, and quantizations like OrcaSAQ-2. This is the first time it has released something it calls a frontier model of its own. A 100% and a 98% next to each other is the kind of line I want to check before anyone repeats it, so I read the three security benchmarks' papers and code, pulled their public leaderboards, and found OrcaRouter's own technical report for the CyberGym run.
Two things surprised me. The CyberGym number is real, third in the leaderboard's own data, and not mainly a measurement of OrcaCyber Zero: another model ran every one of the 1,507 tasks, and Zero 1.0 was called in on 656. And 100% on Cybench is less extraordinary than it sounds, because a frontier lab already reported it. It is the conditions around these numbers that are missing, and they are what makes a security score comparable.
What OrcaRouter has published
There is no paper, no model card in the usual sense, no system card and no weights. What exists is a model page on OrcaRouter's site, its machine-readable twin at /api/public/models/orca/orcacyber-zero-1.5, the thread under the launch post, and, for the predecessor, a CyberGym report on GitHub.
The page's description says Zero 1.5 is "post-trained for security research and authorized security engineering: vulnerability research and reproduction, exploit development, penetration testing, security auditing, and advanced cyber reasoning." The JSON behind it adds a few things the page does not show prominently. The model is locked, with required_tier: "sec-uncensored" and required_band: "offensive". Context is 1,000,000 tokens and maximum output 128,000. Pricing is $3 per million input tokens and $7.50 per million output. The license field is empty. And the benchmark block carries a fourth number the X post leaves out, 76.5% on "SWE-bench Pro V2", every row with "source": "orca".
The predecessor's JSON is more specific. OrcaCyber Zero 1.0, released 17 September, is described as "a cybersecurity post-trained coding model", and its one benchmark is "CyberGym Level 1 pass@1", 98.07%, "(1,478/1,507)". Keep that 1,478 in mind.
CyberGym: what a pass means
CyberGym is a large benchmark built from real bugs that OSS-Fuzz found in open-source projects. At level 1, the setting everyone reports, an agent gets the vulnerable source tree and a text description of the bug, and it has to produce one input that crashes the vulnerable build. The grader then runs the same input against the patched build. A pass means the input crashes the first and not the second, so the agent has to hit the specific bug rather than any crash. There are 1,507 such tasks. The paper's opening figure shows the setup:
The number OrcaRouter quotes for the harness, 98.01%, is 1,477 of 1,507 in its own technical report. Zero 1.0's model card says 98.07%, which is 1,478. One task apart, so I take both as the same run reported twice, and I could not find which one was re-graded. Either way the score is real in the narrow sense that it is in CyberGym's leaderboard data, third behind 99.2 and 98.47.
What the report also shows is who did the work. The harness is a router over two models. DeepSeek V4 Flash runs every task. OrcaCyber Zero 1.0 is called in only when the router decides the worker has stalled, which happened on 656 of the 1,507 tasks. The report's cost table gives per-task means for both, so the split can be put on one footing:
The DeepSeek worker ran every task. The router handed a session to OrcaCyber Zero 1.0 only after stall signals crossed a threshold, at most once per task: 656 times, 43.5% of tasks.
Averaged over all 1,507 tasks, Zero 1.0 accounts for about 18% of the input tokens, 26% of the output tokens and 65% of the estimated cost. It is the expensive specialist that a cheaper model hands the hard cases to. That is a reasonable design, but it means "98% on CyberGym" is a measurement of the system, not of the model now being launched, and it is not a measurement of Zero 1.5 at all.
Two other facts frame the number. The same DeepSeek V4 Flash, inside another team's agent with no second model, scores 94.16. In DeepSeek's own agent it scores 76.7. So the agent around a model moves a CyberGym score by more than twenty points, which makes the harness the main thing being compared. And CyberGym itself has stopped ranking the top of its board: the leaderboard page now shows the twenty systems above 90% in random order, with the score "only for reference", and its submission guide from August says top systems have "nearly saturated" the benchmark.
The report says the agent containers had open internet access during the run and that OrcaRouter checked the agent logs for lookups of the answers and found none. I have no way to check that, and it is the kind of statement that a benchmark with public fixes needs an independent audit for.
Cybench: 39 of 40, and what "unrestricted" leaves out
Cybench has 40 capture-the-flag tasks taken from four professional competitions, and it defines two ways to run them: unguided, where the agent gets one objective and has to find the flag, and subtask-guided, where the task is broken into steps with hints. "Unrestricted", OrcaRouter's word, is not one of Cybench's modes, and the page does not say which of the two it means, how many attempts each task got, or what the iteration cap was.
The claim is 39 of 39. Cybench has 40 tasks, so one was left out and the post does not say which. Counted as a miss, the same run is 97.5%. That alone is not damning: frontier labs routinely report Cybench on subsets of 35, 37 or 39 tasks, usually because a task's container is broken. And 100% is not new. Claude Mythos Preview's entry on the Cybench board is also 100%, on a 35-task subset; Opus 4.7 is 96 on the same 35.
The bigger problem with any high Cybench score in 2026 is contamination. All four competitions released their tasks between October 2022 and March 2024, before the training cutoff of every current model, and 31 of the 40 tasks ship a solution script in the public repository. The Cybench paper drew this exact comparison when it was published:
A model post-trained "for cyber" on public write-ups would be expected to do well here whatever its general ability, which is why a Cybench number without the run settings and the excluded task says very little.
CVE-Bench: a denominator nobody else uses
CVE-Bench is 40 critical vulnerabilities in real web applications, each run in two settings: zero-day, where the agent gets no hint, and one-day, where it gets the public CVE description. A run passes when the grader observes one of eight defined attack outcomes against the sandboxed application. The paper's best agents in 2025 reached about 13%.
The claim is 95.8%, which is 23 of 24 "evaluable" tasks. Nothing in the CVE-Bench repository marks any task as not evaluable, so the 24 is OrcaRouter's own denominator and it is not explained. If the other 16 were failures, the run is 23 of 40, 57.5%. That is a bound under an assumption, not a score, but it is the honest way to put the two numbers side by side. The post also does not say which setting was run, zero-day or one-day, or how many attempts each task got. Two details matter for contamination: the benchmark's default setup leaves internet access on, with only a prompt asking the agent not to look up public exploits, and 15 of the 40 vulnerabilities have a public record that links to an exploit. There is no CVE-Bench leaderboard to compare against.
HumanEval+
93.9% is 154 of the 164 problems. The EvalPlus leaderboard stopped at 2024 models, where the top entry is 89.0, so the number is plausible for a 2026 model and says little. These problems have been public since 2021, and HumanEval+ is a coding benchmark, not a security one.
Here are all four against the boards they came from, with the conditions each row was measured under:
Level 1: description + vulnerable source in, one PoC out. Pass if it crashes the vulnerable build and not the patched one. % of 1,507.
Read against the board: Real, third in the board's data, and inside a cluster of 20 systems above 90% that the leaderboard deliberately shows in random order. It is a two-model system's number: the same base worker reaches 94.16 in someone else's agent.
What the model is, and who can use it
OrcaRouter does not say what Zero 1.5 is built on. In a reply under the launch post it described the model as "not only abliterated, but post trained for cyber". The harness report says Zero 1.0 ran on a local vLLM server, which points to an open-weight base. OrcaRouter's one open security model, OrcaSAQ-2-Cyber-27B, is gated, and its Hugging Face metadata names Qwen3.8-27B as the base with a 262,144-token context, not the 1M on Zero 1.5's page. Neither the base nor what "updated weights" changed is disclosed anywhere I could find.
Access is gated: the page lists "an engagement, a passkey and the terms", and a reply describes KYC with a company email. OrcaRouter replied that the model is "for offensive red teaming". I found no system card, no model-specific use policy, no independent evaluation, and no named customer behind "has already helped protect some of the world's largest software companies". The demos in the thread run against deliberately vulnerable practice applications.
What I would take from it
The CyberGym result is real and it is mostly a harness result: a cheap worker model on every task, the specialist on 43% of them, inside a benchmark whose own maintainers say the top is saturated. The Cybench and CVE-Bench numbers are self-reported on denominators the post does not explain, on public tasks that predate the model, with the settings that would make them comparable left out. None of that means the model is weak. It means that for a model sold specifically on finding real vulnerabilities, the launch gives no evidence that can be checked, and the one result that can be checked measures something else. For a security model the bar should be higher than for a chat model, not lower: a system card, the run settings, the excluded tasks, and an evaluation someone else ran.
How I checked
I read the model page and its JSON card, the launch thread and its replies, and OrcaRouter's CyberGym harness report at commit 4847a00, whose token and cost table I used for the split above. For each benchmark I read the paper and the repository (CVE-Bench at 9fb3e06) and took the comparison rows from the public leaderboard data: CyberGym's cybergym.json, Cybench's leaderboard.csv, EvalPlus's results.json, and the CVE-Bench paper's Section 4.2. I did not call the model, run any benchmark, or run any security tooling. Not verified: which Cybench task was excluded, which CVE-Bench setting and attempts were used, the SWE-bench Pro V2 figure on the page, and OrcaRouter's statement that the agent made no answer lookups.