# Code2Skill: a million skills mined from GitHub, verified by a round trip, not a test
> Satyajit Ghana — Head of Engineering @ Inkers Technology
> canonical: https://ai.thesatyajit.com/articles/code2skill
> date: 2026-09-26
> tags: explainer, agents, llm, evaluation, benchmarks, datasets, licensing
The post that went round said: *a fully automated pipeline that transforms
19,769 open-source repositories into over 1 million verified, reusable
procedural skills for AI agents.* It had no link. The paper is
[*Grounded Skill Synthesis from Code at Scale for Agentic Intelligence*](https://arxiv.org/abs/2609.05571)
(Tong, Wang, Wang, Li, Zhang, Yang and Wu, Ant International, 4 September 2026).
The bank is on Hugging Face as
[`ant-intl/DeveloperSkills-Code2Skill`](https://huggingface.co/datasets/ant-intl/DeveloperSkills-Code2Skill),
and a reference implementation is at
[ant-intl/Code2Skill](https://github.com/ant-intl/Code2Skill).
Each of the three nouns in that sentence turns out to be narrower than it
sounds. "Skill" means a JSON description of one function, not a `SKILL.md`
package. "Verified" means two language models agreed. "Over 1 million" is
1,006,822 accepted records, which filter and deduplicate to 750,748 cards, and
by my count the release covers at least 7,399 of the 19,769 repositories.
None of that makes the work worthless. The retrieval results are real and some of them are useful. But the
sentence should be read with the counts next to it.
I read the paper, its LaTeX source (which holds a commented-out Limitations
section and one unused figure), and the reference code at commit `7f384a5`.
Then I read the release: all 945,993 edge rows and all 750,748 cards. I ran
none of their code. Numbers below are **reported** (the paper or the
release's own metadata), **measured** (I counted them from the release) or
**reasoned** (my arithmetic or inference).
## What a "skill" is here
An [Agent Skill](/articles/prime-agent) in the sense most harnesses use is a
folder rooted at a `SKILL.md`: frontmatter with a name, a description and
trigger conditions, then free-form instructions, sometimes with scripts
beside it. The agent loads it when the trigger matches. [Skill2Env](/articles/skill2env)
compiled 3.4k of those into training environments;
[figures4papers](/articles/figures4papers) is one written by hand.
A Code2Skill skill is none of that. It is a record extracted from **one source
unit** (a function, a method, a command-line entry point or a whole file) by
one LLM call that fills a fixed schema: `level` (atomic, composite or
pattern), `name`, `summary` (under 80 words), `inputs`, `outputs`,
`workflow`, `invariants`, `error_cases`, `implementation_notes`, `evidence`,
`anti_goals`, three scores and a confidence. The prompt asks for arrays of at
most six items of at most 25 words and forbids quoting source code. It is a
request, not a check: 113,754 released cards (15.2%) have more than six
workflow steps, and one has 93 (measured). The paper's own
example is a composite workflow from AdGuard's DNS proxy: retry a
DNS-over-QUIC exchange once, only on a cached connection, close before
reacquiring, and clear the token store only on `Err0RTTRejected`. It is a good
record. It reads like the comment you wish the function had.
Here is a real one from the release, verbatim apart from trimming, card
`ffb1bfda6aa15fa29d3a`:
```json
{
"transfer_title": "schedule_hide_on_inactivity",
"transfer_summary": "Schedules a hide action after a delay if autohide is enabled and no user interaction (mouse or keyboard) is detected.",
"workflow": [
"Check if autohide is enabled; if not, exit",
"Check if mouse or keyboard interaction occurred; if so, exit",
"Set a timeout to call hide after configured delay"
],
"anti_goals": ["Does not handle multiple concurrent timeouts",
"Does not cancel existing timeout before setting new one"],
"representative": { "repo_name": "BuffaloWill",
"relative_path": "oxml_xxe/public/bootstrap/js/bootstrap.js",
"symbol": "Toast._maybeScheduleHide" }
}
```
Two things to notice. The record has no trigger file, no scripts and no
harness contract. It is retrieved top-k against a task and pasted into a
reviewer's prompt, which makes it closer to a code-review checklist item than
to a skill package. The paper contrasts itself with Bi et al., who *do*
serialise mined skills as `SKILL.md`; Code2Skill keeps typed records instead. And the provenance points at
`BuffaloWill/oxml_xxe`, which vendors Bootstrap. The function is Bootstrap's.
That matters later.
## The pipeline, stage by stage
**Repositories.** Every GitHub repository available by 14 April 2026 with more
than 500 stars, expanded through fork links and deduplicated: 19,769
(reported). The paper calls them "popular, actively maintained". Its own pool
chart shows a quarter last pushed more than two years ago.
**Unit selection.** A parser pulls out functions, methods, entry points and
files, skipping tests, binaries, suspected secrets and units under three lines.
The reference parser recognises Python, JavaScript, TypeScript, Go, Java, Rust,
C and C++, and those are exactly the eight families in the release (measured).
An LLM tagger then scores each unit from 0 to 1 on six axes: reusable intent,
procedural steps, boundary conditions, transferability, interface sufficiency
and non-triviality. The paper says units must clear "the configured selection
gate, subject to a per-repository cap" and gives neither number. The reference
config uses a mean score of at least 0.5 and a cap of 400 units.
**Extraction.** One call per unit returns the schema above. A candidate is
dropped if `worth_extracting` is false or `skill_value_score` is below 0.45.
Which model? The appendix says the five construction stages "share one model
configuration" and stops there. The reference code takes any
OpenAI-compatible endpoint with the model set to `your-model-name`. The only
model the release names anywhere is `DeepSeek-V4-Flash-0731`, and it ran a
later pattern pass, not extraction.
**Verification.** This is the idea the paper is built on, and it is a good one.
A regenerator gets the record and the target's signature, *not* the source
body, repository or path, and writes the function. A judge sees the original
and the rebuild side by side and returns `equivalent`. If it says no, an
adjudicator sees everything and decides whether the record was wrong or the
rebuild was. Step through what each call can see:
The decision rules are blunt. Direct accept whenever `equivalent` is true, with
no threshold on the judge's confidence. Adjudicated accept whenever `keep` is
true. Nothing is executed. The regenerated Python must parse and other languages
must be non-empty, and even that is stored as metadata, not used to decide. The
reference repository's methodology note says it plainly: "an LLM consistency
check, not formal program equivalence and not a substitute for execution-based
verification."
The abstract says "verifies".
The paper's check on the checker is a human annotation (Table 4). Direct
accepts: 96% accurate descriptions, 84% correct reconstructions, 76% worth
keeping. Adjudicated accepts: 88% accurate, 0% correct reconstructions (the judge had
already rejected every one, and the annotators agreed) and 84% worth keeping.
Rejects: 32%, 0% and 28%. That is a real gap between accepted and rejected. Two
caveats (reasoned). No sample sizes are given, and every rate is a multiple of
4%, which fits 25 records per pool. And the headline figure, 92% of accepted
descriptions judged accurate, is the unweighted mean of 96 and 88. That is only
right if direct and adjudicated accepts are equally common, and the paper
never says how many of each there were.
**Filtering and deduplication.** Accepted records then pass a deterministic
low-value filter, three regexes plus a few rules, and 60,829 records are
dropped (reported by the release's statistics). In the reference code one
regex is `\btrivial\b`, which also matches "non-trivial". Not one of the
750,748 released cards has trivial, getter or setter in its title, workflow,
inputs, outputs, invariants, error cases or anti-goals (measured). The rest
are grouped by the key `(task_family, intent_action, intent_target)`, inferred without a
model, and one existing record is kept as each group's representative. It
yields 750,748 purpose cards. The largest is `readArguments`,
command-line parsing, with 761 records from 398 owners (measured). The most
common skill in a million mined skills is parsing `argv`, which is about what
I would have guessed.
**The third type.** The abstract leads with three record types. At extraction,
270 of 945,993 mapped records were labelled `pattern` (release statistics). The
paper's granularity chart counts atomic-only, composite-only and mixed cards to
750,536 and leaves out the 212 pattern-only cards. The release adds a pattern
pass that is not in the paper. Deterministic gates settle 736,209 cards, and
DeepSeek-V4-Flash-0731 reviews 14,539 as proposer, verifier and adjudicator.
3,600 are confirmed (measured).
## The scale arithmetic
The post's arithmetic is 1,006,822 ÷ 19,769 ≈ 50.9 skills per repository
(reasoned). That treats the pool as if every repository contributed. The
release says otherwise.
The edges table stores the owner in `repo_name` (`NVIDIA`, `alibaba`), with
the repository as the first segment of each example's path. Counting owner and
repository pairs across every card's examples gives **7,399 repositories**, a
lower bound, since the examples cover 918,083 of the 945,993 mapped records
(measured). That is at least 37% of the pool. The yield per contributing
repository is then at most about 128 records, not 50.9 (reasoned: 945,993 ÷
7,399). An unused figure in the arXiv source, `construction_outputs.pdf`,
charts 6,115 "rep groups" beside the 19,769 source repositories, close to the
6,103 owners I count.
The owners hint at why. Records per owner top out at 448, whether the owner has
one repository or many. alibaba has 31 repositories in the release and 246
records; facebook has 13 and 65; NVIDIA has 17 and 402. That is what a cap of
about 400 units applied per owner, not per repository, would look like
(reasoned; the paper states neither the cap nor its unit). Add the 41.4% of the
pool whose primary language the parser does not read, and 37% coverage stops
being mysterious.
The same unused figure has one more pair of bars: 147,465 "retrieval records"
from 778 "retrieval repos". The experiments "randomly sample 10% of the
available retrieval records" as the retrieval pool. If those 147,465 are what
was available, the agents in every table searched roughly 14,700 records, not
a million (reasoned, and flagged as a reading of an unreferenced figure). Even
at the full bank, 10% is about 100,000.
**Receipts.** The public CodeSkillBank release is internally consistent on its totals, and several of its details differ from the paper. Every count below comes from reading the released parquet, the release's own metadata, or the Hugging Face and SWE-bench listings. None of it comes from the paper's prose.
| check | result | what it means |
| :--- | ---: | :--- |
| edges rows + dropped rows (metadata) | 945,993 + 60,829 = 1,006,822 | equals the paper's accepted count exactly |
| cards by granularity | 490,457 / 233,549 / 26,530 / 212 | atomic, composite, mixed, pattern-only; the paper's Figure 8 omits the 212 and totals 750,536, not 750,748 |
| distinct owners in edges | 6,103 | the unreferenced source figure says 6,115 'rep groups' |
| distinct owner/repo in card examples | 7,399 | a lower bound; the examples cover 918,083 of 945,993 records (97.0%) |
| records per owner: median / mean / max | 124 / 155.0 / 448 | alibaba: 31 repos, 246 records; facebook: 13 repos, 65 records |
| records by language family | 8 families | Python 236,827; C++ 121,559; TypeScript 114,471; C 100,293; JavaScript 95,438; Go 90,715; Java 86,603; Rust 72,177; no Kotlin, C#, Swift, PHP or Ruby |
| cards whose text contains trivial, getter or setter | 0 of 750,748 | the low-value regex ran; it also matches 'non-trivial' |
| task_family = validation | 348,612 (46.4%) | the paper's 'validation (46.4%)'; testing appears on 10,762 cards, a family in neither the paper's vocabulary nor the reference code |
| largest purpose card | 761 records, 398 owners | readArguments: command-line argument parsing |
| example paths that look vendored | 16,701 (1.8%) | for example Bootstrap's Toast._maybeScheduleHide, credited to BuffaloWill/oxml_xxe |
| cards reviewed for patterns by a model | 14,539 | all by DeepSeek-V4-Flash-0731; 3,600 confirmed; 736,209 settled by deterministic gates |
| SWE-bench Verified source repos present | 2 of 12 | django/django 109 records, astropy/astropy 260; together 253 of the 500 Verified instances |
| AdguardTeam/dnsproxy (the paper's Figure 2) in examples | 0 | AdGuard's 280 records list AdGuardHome and AdguardForiOS only |
| dropped parquet files | 0 of 2 | RELEASE_STATUS.json reports 2 files and 60,829 rows; the repository has none |
| datasets-server on the top-level card | no data files | configs point at data/cards/*.parquet; the files live under codeskillbank_hf_release_with_patterns_20260814/ |
| licence on the dataset card | apache-2.0 | the inner README and LICENSE_REVIEW.md both say license: other until a provenance audit is done |
Rows 1 and 2 confirm the paper. Rows 3 to 6 are the scale arithmetic. The rest are things the paper does not say.
> method: I downloaded all 19 edges shards and read the 16 cards shards one at a time, extracting only provenance and text fields. edges.repo_name holds a GitHub owner; the repository is the first segment of relative_path in each card's examples, and examples are capped at 8 per card. The pattern_annotations shards were read for model and label_source. Language counts go by file extension. 'Vendored' means a path under node_modules/, vendor/, third_party/, external/, deps/ or similar, or a file named like jquery.js or bootstrap.js. It is a heuristic, not an audit. SWE-bench Verified's repo counts come from the Hugging Face datasets-server statistics for princeton-nlp/SWE-bench_Verified. I executed no third-party code.
> source: https://huggingface.co/datasets/ant-intl/DeveloperSkills-Code2Skill
> captured: 2026-09-26
> data: https://ai.thesatyajit.com/articles/code2skill/data/release-audit.json (16 rows)
## The evaluation
Five models (DS4-Flash, which is DeepSeek V4 Flash; Qwen3.5 27B; Qwen3.6 27B;
Gemini 2.5 Pro; GPT 5.2) run in nine settings, with reasoning on and off
where a model has both. Eight benchmarks: SWE-bench Verified, BigCodeBench,
AIME 2026, HMMT 2025, GPQA, TerminalBench, LongCLI-Bench and AgentBench-OS. The loop drafts a
solution, reviews it and revises. The skill condition retrieves records into
the review step; the control runs the same loop without them.
I recomputed Table 1 from the LaTeX. It is consistent. There are 57 wins, 6
ties and 9 losses across 72 pairs, and the macro-average goes from 42.90 to
47.90, a 5.00-point gain that is 11.66% relative (the paper rounds it to
11.7%). SWE-bench Verified improves 9 of 9 and AIME 9 of 9. BigCodeBench is
6 of 9. LongCLI is 4 of 9, with five pairs flat, two of them at 0.00 both
ways. The largest average gains are DS4-Flash's, +7.39 and +7.29. Table 1
reports no variance.
Of the baselines one would want, the paper has **no skills** (Table 1) and
**trajectory-derived banks** (Table 2). It has no human-written skills and no
retrieval over raw code. That second omission matters most, because it is the
obvious question: does abstracting a function into a record beat retrieving the
function?
Table 2 builds Trace2Skill, ExpeL and a SkillRL-adapted bank with
Qwen3.5-397B-A17B on a held-out split and runs all four banks through the
DS4-Flash reasoning loop, five runs each. Code2Skill wins all seven benchmarks
and averages 49.5 against 31.0, 27.9 and 32.8. Its row matches Table 1's
DS4-Flash reasoning row with skills to the printed precision, so Table 1's row
without skills is the matched control. Put it back:
The same loop with no skills averages 41.6 over those seven benchmarks
(reasoned, from Table 1). Every trajectory bank is below it. On SWE-bench
Verified, Trace2Skill scores 6.0 and ExpeL 7.5 against 34.32 with nothing
retrieved. The paper's best-of-three oracle, 40.1, is also below it. And on
BigCodeBench, Code2Skill's 42.3 sits under the no-skill 43.27, though the
paper says it "ranks first on all seven benchmarks". The Limitations section in the source, commented out of the compiled
paper, says the baselines "are not construction-compute matched". What the
table establishes is that three baseline banks as built here actively hurt
this agent. [MemHarness](/articles/memharness) found raw memory replay can hurt
more than no memory. That is a known failure, and it is not the same claim as
code beating trajectories.
The ablation I would take away is RQ4, on BigCodeBench Instruct-Hard.
Going from one record to ten grows the context from 2.1K to 17.8K characters and
adds little. Rendering three records as summaries cuts the context by 88.9%,
from 6,352 to 707 characters, keeps Qwen's score and lifts DS4-Flash from
28.40 to 31.80. With full records, DS4-Flash sits at or below its no-skill
baseline. If you build on this bank, render the summary.
Two more results are thinner than their placement suggests. In coding RL,
starting from a Qwen3-32B SWE-World checkpoint, resolve rate at step 150 goes
from 24% with no skills to 31-32% with skills in the prompt or the verifier and
38% with skills in a post-generation reviewer. That is one checkpoint and no
seeds, which the paper says itself. The AI-code result compares banks of 50
records each: 25 tested human implementations and 25 GPT-5.1 rewrites, two
records apiece. On 400 LiveCodeBench tasks they score 93.00% and 93.50%, or
372 and 374. They disagree on 16 tasks, 7 against 9. An exact sign test on that
split gives p ≈ 0.80 (reasoned). There is no no-skill number for that
subset, so neither bank is shown to help.
## Contamination
The paper does not discuss it; the word appears only in a cited title. It
evaluates a fixed 475-instance subset of SWE-bench Verified's 500 without
saying which 25 it dropped or why. The
benchmark is drawn from twelve repositories. The release contains two of them,
`django/django` with 109 records and `astropy/astropy` with 260 (measured), and
they supply 231 and 22 of the 500 Verified instances. The crawl is from April
2026, years after the fixes the benchmark asks for, so any record from those
repositories describes the patched code. Whether any such record landed in the
10% retrieval sample, and whether any of them encodes a benchmark fix, cannot
be determined from what is released (reasoned). A skill bank mined from the
same repositories a benchmark is drawn from needs a filter, and this one
reports none.
## What is released, and the licence
The dataset has cards, edges and pattern annotations. It does not have the
evidence archive the paper leans on for auditability. No released field records
`accepted_stage`, the judge's rationale, a commit or a licence (measured). The
`dropped` config that the release's own status file counts as two files and
60,829 rows is not in the repository. The top-level card's configs point one
directory above where the parquet sits, and the Hugging Face datasets-server
reports no data files. The Quick Start loads `ant-intl/CodeSkillBank`, a name
that does not exist in the organisation's listing. The paper's own Figure 2
example comes from `AdguardTeam/dnsproxy`. AdGuard's 280 released records come
from two other repositories.
The licence is the real problem. The card's metadata says `apache-2.0`. Its own
body says "The dataset card therefore uses `license: other` until the
provenance audit and dataset-level licensing decision are complete." The
`LICENSE_REVIEW.md` shipped beside the data says: "Do not replace
`license: other` with a standard license until the dataset authors decide the
dataset-level terms and complete an upstream-source audit." Someone replaced
it. The code repository has no `LICENSE` file, and its README says "No license
is implied". Its publication checklist still has "Confirm model-provider terms
permit the documented workflow" unticked.
My reading, which is not legal advice: the records are model-written
descriptions, and the prompt forbids quoting code, so they are further from
copies than a code corpus is. But they paraphrase specific functions from
repositories under every licence from MIT to GPL. Attribution is the one
condition nearly all of those licences share, and it needs the provenance the
release says it lacks: no URL, no commit, and an owner where the repository
should be. Vendored code makes it worse. By my path heuristic, 16,701 example
references (1.8%) sit in vendored directories or bundled libraries. There the
credited repository is not the author, as with Bootstrap under `oxml_xxe`.
Apache-2.0 on top of that is a promise the authors' own review file says they
cannot make yet.
## The take
The mechanism is worth stealing. Source-body-blind reconstruction is a cheap,
scalable completeness test for any description of code. If the description
cannot rebuild the function, something is missing, and the adjudicator is an
honest admission that rebuilds fail for their own reasons. Retrieved records
lifting a review step in 57 of 72 matched runs is a real result, and the
summary-rendering ablation is directly usable.
What the sentence in the post claims, the release does not show.
"Verified" is model agreement with no execution. "19,769 repositories" is the
pool, and at least 7,399 appear in the bank. "Over 1 million" deduplicates to
750,748, and the evaluation searched a 10% sample. The comparison that would
show code-derived skills beat alternatives uses baselines that lose to no
skills at all. The retrieval-over-raw-code baseline, the one that would say
whether the abstraction earns its keep, is not run. Nor, before an Apache
licence goes on the card, is the provenance audit the authors wrote down as a
precondition.
[Beacon](/articles/agent-beacon) turns agent runs into skills. Code2Skill turns
code into skills before any run exists. The second is the more interesting
bet, and it deserves a comparison it could lose.
Read off the paper's decision rules and the reference code. It falls if the
production pipeline ran tests or compiled the regenerated code and the paper
under-reports it. The reference repository is not identical to what built the
release: 10,762 cards carry a `testing` task family that exists in neither the
paper's vocabulary nor the code. So a different version is possible.
The owner and repository pairs come from card examples, capped at eight per
card, covering 97.0% of mapped records. The release carries no path for the
other 27,910, so a full join could add repositories I cannot see, though the
examples already reach all but 5 of the 6,103 owners. It also falls if the
path's first segment is not the repository name. I checked eight at random,
`dpkp/kafka-python` and `dylanaraps/pywal` among them, and all eight resolve on
GitHub.
This rests on Table 1's no-skill row being Table 2's control. The Code2Skill
rows match to the printed precision, so I think it is. If the Table 2 runs used
a different loop, retrieval pool or seed set, the comparison is weaker in both
directions, and a no-skill row inside Table 2 would settle it.
This is the weakest claim here. It leans on an unreferenced figure's
147,465 "retrieval records" being the pool the 10% was drawn from. If
"available retrieval records" means all accepted records, the sample is about
100,000, and my point that the evaluation never used a million still holds.
---
*Source: Yongqi Tong, Pan Wang, Hang Wang, Jianshe Li, Xin Zhang, Jiang-Ming
Yang and Wei Wu,
[*Grounded Skill Synthesis from Code at Scale for Agentic Intelligence*](https://arxiv.org/abs/2609.05571)
(Ant International, 4 September 2026). Dataset
[`ant-intl/DeveloperSkills-Code2Skill`](https://huggingface.co/datasets/ant-intl/DeveloperSkills-Code2Skill)
as released on 14 August 2026; code at
[ant-intl/Code2Skill](https://github.com/ant-intl/Code2Skill), commit
`7f384a5`. Figures 1, 3 and 6 are rendered from the vector originals in the
paper's arXiv source for commentary. The funnel, round-trip and baseline
interactives are my own. Their paper numbers come from Tables 1, 2 and 4 and
Appendix C, and their release numbers are my counts, listed with method in the
receipts table.*