Code2Skill: a million skills mined from GitHub, verified by a round trip, not a test

mdjsonmcpcopy↗ai

2026-09-26 · 18 min · explainer · agents · llm · evaluation · benchmarks · datasets · licensing

The post that went round said: a fully automated pipeline that transforms 19,769 open-source repositories into over 1 million verified, reusable procedural skills for AI agents. It had no link. The paper is Grounded Skill Synthesis from Code at Scale for Agentic Intelligence (Tong, Wang, Wang, Li, Zhang, Yang and Wu, Ant International, 4 September 2026). The bank is on Hugging Face as ant-intl/DeveloperSkills-Code2Skill, and a reference implementation is at ant-intl/Code2Skill.

Each of the three nouns in that sentence turns out to be narrower than it sounds. "Skill" means a JSON description of one function, not a SKILL.md package. "Verified" means two language models agreed. "Over 1 million" is 1,006,822 accepted records, which filter and deduplicate to 750,748 cards, and by my count the release covers at least 7,399 of the 19,769 repositories. None of that makes the work worthless. The retrieval results are real and some of them are useful. But the sentence should be read with the counts next to it.

I read the paper, its LaTeX source (which holds a commented-out Limitations section and one unused figure), and the reference code at commit 7f384a5. Then I read the release: all 945,993 edge rows and all 750,748 cards. I ran none of their code. Numbers below are reported (the paper or the release's own metadata), measured (I counted them from the release) or reasoned (my arithmetic or inference).

What a "skill" is here

An Agent Skill in the sense most harnesses use is a folder rooted at a SKILL.md: frontmatter with a name, a description and trigger conditions, then free-form instructions, sometimes with scripts beside it. The agent loads it when the trigger matches. Skill2Env compiled 3.4k of those into training environments; figures4papers is one written by hand.

A Code2Skill skill is none of that. It is a record extracted from one source unit (a function, a method, a command-line entry point or a whole file) by one LLM call that fills a fixed schema: level (atomic, composite or pattern), name, summary (under 80 words), inputs, outputs, workflow, invariants, error_cases, implementation_notes, evidence, anti_goals, three scores and a confidence. The prompt asks for arrays of at most six items of at most 25 words and forbids quoting source code. It is a request, not a check: 113,754 released cards (15.2%) have more than six workflow steps, and one has 93 (measured). The paper's own example is a composite workflow from AdGuard's DNS proxy: retry a DNS-over-QUIC exchange once, only on a cached connection, close before reacquiring, and clear the token store only on Err0RTTRejected. It is a good record. It reads like the comment you wish the function had.

Here is a real one from the release, verbatim apart from trimming, card ffb1bfda6aa15fa29d3a:

{
  "transfer_title": "schedule_hide_on_inactivity",
  "transfer_summary": "Schedules a hide action after a delay if autohide is enabled and no user interaction (mouse or keyboard) is detected.",
  "workflow": [
    "Check if autohide is enabled; if not, exit",
    "Check if mouse or keyboard interaction occurred; if so, exit",
    "Set a timeout to call hide after configured delay"
  ],
  "anti_goals": ["Does not handle multiple concurrent timeouts",
                 "Does not cancel existing timeout before setting new one"],
  "representative": { "repo_name": "BuffaloWill",
                      "relative_path": "oxml_xxe/public/bootstrap/js/bootstrap.js",
                      "symbol": "Toast._maybeScheduleHide" }
}

Two things to notice. The record has no trigger file, no scripts and no harness contract. It is retrieved top-k against a task and pasted into a reviewer's prompt, which makes it closer to a code-review checklist item than to a skill package. The paper contrasts itself with Bi et al., who do serialise mined skills as SKILL.md; Code2Skill keeps typed records instead. And the provenance points at BuffaloWill/oxml_xxe, which vendors Bootstrap. The function is Bootstrap's. That matters later.

The pipeline, stage by stage

expand

Repositories. Every GitHub repository available by 14 April 2026 with more than 500 stars, expanded through fork links and deduplicated: 19,769 (reported). The paper calls them "popular, actively maintained". Its own pool chart shows a quarter last pushed more than two years ago.

expand

Unit selection. A parser pulls out functions, methods, entry points and files, skipping tests, binaries, suspected secrets and units under three lines. The reference parser recognises Python, JavaScript, TypeScript, Go, Java, Rust, C and C++, and those are exactly the eight families in the release (measured). An LLM tagger then scores each unit from 0 to 1 on six axes: reusable intent, procedural steps, boundary conditions, transferability, interface sufficiency and non-triviality. The paper says units must clear "the configured selection gate, subject to a per-repository cap" and gives neither number. The reference config uses a mean score of at least 0.5 and a cap of 400 units.

Extraction. One call per unit returns the schema above. A candidate is dropped if worth_extracting is false or skill_value_score is below 0.45. Which model? The appendix says the five construction stages "share one model configuration" and stops there. The reference code takes any OpenAI-compatible endpoint with the model set to your-model-name. The only model the release names anywhere is DeepSeek-V4-Flash-0731, and it ran a later pattern pass, not extraction.

Verification. This is the idea the paper is built on, and it is a good one. A regenerator gets the record and the target's signature, not the source body, repository or path, and writes the function. A judge sees the original and the rebuild side by side and returns equivalent. If it says no, an adjudicator sees everything and decides whether the record was wrong or the rebuild was. Step through what each call can see:

The decision rules are blunt. Direct accept whenever equivalent is true, with no threshold on the judge's confidence. Adjudicated accept whenever keep is true. Nothing is executed. The regenerated Python must parse and other languages must be non-empty, and even that is stored as metadata, not used to decide. The reference repository's methodology note says it plainly: "an LLM consistency check, not formal program equivalence and not a substitute for execution-based verification." The abstract says "verifies".

The paper's check on the checker is a human annotation (Table 4). Direct accepts: 96% accurate descriptions, 84% correct reconstructions, 76% worth keeping. Adjudicated accepts: 88% accurate, 0% correct reconstructions (the judge had already rejected every one, and the annotators agreed) and 84% worth keeping. Rejects: 32%, 0% and 28%. That is a real gap between accepted and rejected. Two caveats (reasoned). No sample sizes are given, and every rate is a multiple of 4%, which fits 25 records per pool. And the headline figure, 92% of accepted descriptions judged accurate, is the unweighted mean of 96 and 88. That is only right if direct and adjudicated accepts are equally common, and the paper never says how many of each there were.

Filtering and deduplication. Accepted records then pass a deterministic low-value filter, three regexes plus a few rules, and 60,829 records are dropped (reported by the release's statistics). In the reference code one regex is \btrivial\b, which also matches "non-trivial". Not one of the 750,748 released cards has trivial, getter or setter in its title, workflow, inputs, outputs, invariants, error cases or anti-goals (measured). The rest are grouped by the key (task_family, intent_action, intent_target), inferred without a model, and one existing record is kept as each group's representative. It yields 750,748 purpose cards. The largest is readArguments, command-line parsing, with 761 records from 398 owners (measured). The most common skill in a million mined skills is parsing argv, which is about what I would have guessed.

The third type. The abstract leads with three record types. At extraction, 270 of 945,993 mapped records were labelled pattern (release statistics). The paper's granularity chart counts atomic-only, composite-only and mixed cards to 750,536 and leaves out the 212 pattern-only cards. The release adds a pattern pass that is not in the paper. Deterministic gates settle 736,209 cards, and DeepSeek-V4-Flash-0731 reviews 14,539 as proposer, verifier and adjudicator. 3,600 are confirmed (measured).

The scale arithmetic

The post's arithmetic is 1,006,822 ÷ 19,769 ≈ 50.9 skills per repository (reasoned). That treats the pool as if every repository contributed. The release says otherwise.

The edges table stores the owner in repo_name (NVIDIA, alibaba), with the repository as the first segment of each example's path. Counting owner and repository pairs across every card's examples gives 7,399 repositories, a lower bound, since the examples cover 918,083 of the 945,993 mapped records (measured). That is at least 37% of the pool. The yield per contributing repository is then at most about 128 records, not 50.9 (reasoned: 945,993 ÷ 7,399). An unused figure in the arXiv source, construction_outputs.pdf, charts 6,115 "rep groups" beside the 19,769 source repositories, close to the 6,103 owners I count.

The owners hint at why. Records per owner top out at 448, whether the owner has one repository or many. alibaba has 31 repositories in the release and 246 records; facebook has 13 and 65; NVIDIA has 17 and 402. That is what a cap of about 400 units applied per owner, not per repository, would look like (reasoned; the paper states neither the cap nor its unit). Add the 41.4% of the pool whose primary language the parser does not read, and 37% coverage stops being mysterious.

The same unused figure has one more pair of bars: 147,465 "retrieval records" from 778 "retrieval repos". The experiments "randomly sample 10% of the available retrieval records" as the retrieval pool. If those 147,465 are what was available, the agents in every table searched roughly 14,700 records, not a million (reasoned, and flagged as a reading of an unreferenced figure). Even at the full bank, 10% is about 100,000.

The evaluation

Five models (DS4-Flash, which is DeepSeek V4 Flash; Qwen3.5 27B; Qwen3.6 27B; Gemini 2.5 Pro; GPT 5.2) run in nine settings, with reasoning on and off where a model has both. Eight benchmarks: SWE-bench Verified, BigCodeBench, AIME 2026, HMMT 2025, GPQA, TerminalBench, LongCLI-Bench and AgentBench-OS. The loop drafts a solution, reviews it and revises. The skill condition retrieves records into the review step; the control runs the same loop without them.

I recomputed Table 1 from the LaTeX. It is consistent. There are 57 wins, 6 ties and 9 losses across 72 pairs, and the macro-average goes from 42.90 to 47.90, a 5.00-point gain that is 11.66% relative (the paper rounds it to 11.7%). SWE-bench Verified improves 9 of 9 and AIME 9 of 9. BigCodeBench is 6 of 9. LongCLI is 4 of 9, with five pairs flat, two of them at 0.00 both ways. The largest average gains are DS4-Flash's, +7.39 and +7.29. Table 1 reports no variance.

Of the baselines one would want, the paper has no skills (Table 1) and trajectory-derived banks (Table 2). It has no human-written skills and no retrieval over raw code. That second omission matters most, because it is the obvious question: does abstracting a function into a record beat retrieving the function?

Table 2 builds Trace2Skill, ExpeL and a SkillRL-adapted bank with Qwen3.5-397B-A17B on a held-out split and runs all four banks through the DS4-Flash reasoning loop, five runs each. Code2Skill wins all seven benchmarks and averages 49.5 against 31.0, 27.9 and 32.8. Its row matches Table 1's DS4-Flash reasoning row with skills to the printed precision, so Table 1's row without skills is the matched control. Put it back:

The same loop with no skills averages 41.6 over those seven benchmarks (reasoned, from Table 1). Every trajectory bank is below it. On SWE-bench Verified, Trace2Skill scores 6.0 and ExpeL 7.5 against 34.32 with nothing retrieved. The paper's best-of-three oracle, 40.1, is also below it. And on BigCodeBench, Code2Skill's 42.3 sits under the no-skill 43.27, though the paper says it "ranks first on all seven benchmarks". The Limitations section in the source, commented out of the compiled paper, says the baselines "are not construction-compute matched". What the table establishes is that three baseline banks as built here actively hurt this agent. MemHarness found raw memory replay can hurt more than no memory. That is a known failure, and it is not the same claim as code beating trajectories.

The ablation I would take away is RQ4, on BigCodeBench Instruct-Hard.

expand

Going from one record to ten grows the context from 2.1K to 17.8K characters and adds little. Rendering three records as summaries cuts the context by 88.9%, from 6,352 to 707 characters, keeps Qwen's score and lifts DS4-Flash from 28.40 to 31.80. With full records, DS4-Flash sits at or below its no-skill baseline. If you build on this bank, render the summary.

Two more results are thinner than their placement suggests. In coding RL, starting from a Qwen3-32B SWE-World checkpoint, resolve rate at step 150 goes from 24% with no skills to 31-32% with skills in the prompt or the verifier and 38% with skills in a post-generation reviewer. That is one checkpoint and no seeds, which the paper says itself. The AI-code result compares banks of 50 records each: 25 tested human implementations and 25 GPT-5.1 rewrites, two records apiece. On 400 LiveCodeBench tasks they score 93.00% and 93.50%, or 372 and 374. They disagree on 16 tasks, 7 against 9. An exact sign test on that split gives p ≈ 0.80 (reasoned). There is no no-skill number for that subset, so neither bank is shown to help.

Contamination

The paper does not discuss it; the word appears only in a cited title. It evaluates a fixed 475-instance subset of SWE-bench Verified's 500 without saying which 25 it dropped or why. The benchmark is drawn from twelve repositories. The release contains two of them, django/django with 109 records and astropy/astropy with 260 (measured), and they supply 231 and 22 of the 500 Verified instances. The crawl is from April 2026, years after the fixes the benchmark asks for, so any record from those repositories describes the patched code. Whether any such record landed in the 10% retrieval sample, and whether any of them encodes a benchmark fix, cannot be determined from what is released (reasoned). A skill bank mined from the same repositories a benchmark is drawn from needs a filter, and this one reports none.

What is released, and the licence

The dataset has cards, edges and pattern annotations. It does not have the evidence archive the paper leans on for auditability. No released field records accepted_stage, the judge's rationale, a commit or a licence (measured). The dropped config that the release's own status file counts as two files and 60,829 rows is not in the repository. The top-level card's configs point one directory above where the parquet sits, and the Hugging Face datasets-server reports no data files. The Quick Start loads ant-intl/CodeSkillBank, a name that does not exist in the organisation's listing. The paper's own Figure 2 example comes from AdguardTeam/dnsproxy. AdGuard's 280 released records come from two other repositories.

The licence is the real problem. The card's metadata says apache-2.0. Its own body says "The dataset card therefore uses license: other until the provenance audit and dataset-level licensing decision are complete." The LICENSE_REVIEW.md shipped beside the data says: "Do not replace license: other with a standard license until the dataset authors decide the dataset-level terms and complete an upstream-source audit." Someone replaced it. The code repository has no LICENSE file, and its README says "No license is implied". Its publication checklist still has "Confirm model-provider terms permit the documented workflow" unticked.

My reading, which is not legal advice: the records are model-written descriptions, and the prompt forbids quoting code, so they are further from copies than a code corpus is. But they paraphrase specific functions from repositories under every licence from MIT to GPL. Attribution is the one condition nearly all of those licences share, and it needs the provenance the release says it lacks: no URL, no commit, and an owner where the repository should be. Vendored code makes it worse. By my path heuristic, 16,701 example references (1.8%) sit in vendored directories or bundled libraries. There the credited repository is not the author, as with Bootstrap under oxml_xxe. Apache-2.0 on top of that is a promise the authors' own review file says they cannot make yet.

The take

The mechanism is worth stealing. Source-body-blind reconstruction is a cheap, scalable completeness test for any description of code. If the description cannot rebuild the function, something is missing, and the adjudicator is an honest admission that rebuilds fail for their own reasons. Retrieved records lifting a review step in 57 of 72 matched runs is a real result, and the summary-rendering ablation is directly usable.

What the sentence in the post claims, the release does not show. "Verified" is model agreement with no execution. "19,769 repositories" is the pool, and at least 7,399 appear in the bank. "Over 1 million" deduplicates to 750,748, and the evaluation searched a 10% sample. The comparison that would show code-derived skills beat alternatives uses baselines that lose to no skills at all. The retrieval-over-raw-code baseline, the one that would say whether the abstraction earns its keep, is not run. Nor, before an Apache licence goes on the card, is the provenance audit the authors wrote down as a precondition.

Beacon turns agent runs into skills. Code2Skill turns code into skills before any run exists. The second is the more interesting bet, and it deserves a comparison it could lose.

Source: Yongqi Tong, Pan Wang, Hang Wang, Jianshe Li, Xin Zhang, Jiang-Ming Yang and Wei Wu, Grounded Skill Synthesis from Code at Scale for Agentic Intelligence (Ant International, 4 September 2026). Dataset ant-intl/DeveloperSkills-Code2Skill as released on 14 August 2026; code at ant-intl/Code2Skill, commit 7f384a5. Figures 1, 3 and 6 are rendered from the vector originals in the paper's arXiv source for commentary. The funnel, round-trip and baseline interactives are my own. Their paper numbers come from Tables 1, 2 and 4 and Appendix C, and their release numbers are my counts, listed with method in the receipts table.