Grounded Skill Synthesis from Code at Scale for Agentic Intelligence
Yongqi Tong, Pan Wang, Hang Wang, Jianshe Li, Xin Zhang, Jiang-Ming Yang, Wei Wu
cs.SE · cs.CL
Quick verdict
Code-derived 1M-skill bank gives +11.7% rel across 8 benchmarks and 9 model settings; solid eval, but contamination and significance checks are missing.
Summary
The paper introduces Code2Skill, a fully automated pipeline that converts source code from 19,769 popular GitHub repositories into over one million verified skill records (CodeSkillBank). Selected functions, methods, and file-level components are abstracted into atomic-operation, composite-workflow, or recurring-pattern records capturing applicability conditions, steps, invariants, failure handling, anti-goals, and source provenance. Each record is verified by source-body-blind reconstruction: an LLM regenerates code from the skill alone, and a source-aware judge compares the reconstruction with the original, routing disagreements to adjudication. Across 72 protocol-matched evaluations (nine model settings, eight benchmarks spanning software engineering, terminal/OS control, and math/science reasoning), retrieved skills raise the macro-average score from 42.90 to 47.90 (11.7% relative) and win 57 of 72 comparisons, including all nine SWE-bench Verified pairs. Under a shared agent loop, Code2Skill outscores Trace2Skill, ExpeL, and SkillRL-Bank on all seven shared benchmarks (49.5 vs 31.0/27.9/32.8). Skills also help in coding RL, where post-generation review is the best interface (38% vs 24% resolve rate), and skills from AI-generated code perform comparably to human-derived ones (93.50% vs 93.00% on a 400-task LiveCodeBench subset). Main limitations: no decontamination analysis despite GitHub-derived skills being tested on GitHub-derived benchmarks like SWE-bench; no significance testing for the 72-comparison headline; verification is LLM-judged rather than test-based; and no explicit code release is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Source code from maintained repositories is a scalable, grounded substrate for skill synthesis that avoids the environment coupling of trajectory methods and the unverifiability of document-derived skills.
- Retrieved CodeSkillBank skills improve the macro-average score from 42.90 to 47.90 (11.7% relative) across 72 protocol-matched evaluations, winning 57 of 72 including all nine SWE-bench Verified pairs.
- Under a unified downstream interface, Code2Skill outperforms trajectory-derived banks (Trace2Skill, ExpeL, SkillRL-Bank) on all seven shared benchmarks, 49.5 vs 31.0/27.9/32.8 average.
- Source-body-blind reconstruction plus source-aware comparison is a scalable verification filter, yielding records with 92% human-judged description accuracy in the final bank.
- The same pipeline extracts usable skills from tested AI-generated code (93.50% vs 93.00% pass rate on LiveCodeBench), enabling the bank to keep expanding as AI coding grows.
Strengths
Novel grounded substrate with a falsifiable verification loop
Source-body-blind reconstruction (§3.2.3, Algorithm 1) turns skill faithfulness into a checkable code-comparison task — a genuinely different paradigm from trajectory- or document-derived skills — and the adjudicator design separates bad skills from failed reconstructions.
Scale with full provenance
1,006,822 accepted records from 19,769 active repositories (>500 stars; median 3,133 stars, 66.0% pushed within a year; Fig. 3), each retaining repository/file/symbol/span provenance plus acceptance status for audit and refresh (§3.1, §4.1).
Unusually broad evaluation program
Honest human audit of construction outcomes
Complete method disclosure
Weaknesses
No contamination analysis
Headline statistics lack uncertainty quantification
RQ2 comparison confounds
Grounding gate admits failed reconstructions
Framing occasionally outruns evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces Code2Skill, a fully automated pipeline that mines 19,769 popular GitHub repositories (>500 stars, active-maintenance filters; Fig. 3) and converts selected source units — functions, methods, CLI entry points, file-level components — into typed skill records: atomic operations, composite workflows, and recurring patterns (§3.2.2). Each record captures applicability conditions, ordered steps, invariants, failure handling, anti-goals, and span-level source provenance. Verification (§3.2.3, Algorithm 1) is a two-step round trip: a source-body-blind reconstructor regenerates code from the record alone, then a source-aware equivalence judge compares reconstruction to source, with an adjudicator resolving disagreements. The result is CodeSkillBank: 1,006,822 accepted records (§4). Central claims: (i) repository code is a scalable, grounded skill substrate avoiding trajectory-environment coupling (§1); (ii) retrieved skills lift the macro-average from 42.90 to 47.90 (+11.7% relative) across 72 protocol-matched evaluations, winning 57/72 (Table 1); (iii) under a shared loop, Code2Skill beats Trace2Skill, ExpeL, and SkillRL-Bank on all seven shared benchmarks, 49.5 vs 31.0/27.9/32.8 (Table 2); (iv) skills integrate productively into coding RL (Table 3); (v) AI-generated code yields equally usable skills (Fig. 7).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Code2Skill?
A fully automated pipeline from Ant International researchers that converts source code from GitHub repositories into reusable 'skill' records for AI agents — atomic operations, composite workflows, and recurring patterns — and verifies each record by having a language model reconstruct the code from the skill description alone, then comparing that reconstruction against the original source.
How large is CodeSkillBank and where does its data come from?
CodeSkillBank contains 1,006,822 accepted skill records mined from 19,769 popular GitHub repositories (over 500 stars, actively maintained), covering languages such as Python, TypeScript, JavaScript, Go, and Java. Each record keeps provenance down to source-code spans so it can be inspected and refreshed as the code evolves.
How much does CodeSkillBank improve agent performance?
Across 72 matched evaluations (nine model settings, eight benchmarks including SWE-bench Verified, BigCodeBench, AIME, HMMT, GPQA, and terminal benchmarks), adding retrieved skills raised the macro-average score from 42.90 to 47.90 — an 11.7% relative gain — improving 57 of 72 comparisons, including all nine SWE-bench Verified pairs. It also beat three trajectory-derived skill baselines on all seven shared benchmarks.
Is CodeSkillBank publicly available?
The paper's title page includes 'Website' and 'Dataset' link labels, but the reviewed text contains no explicit URL or release statement, so public availability of the bank and pipeline code could not be confirmed from the paper itself.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction ends inside Appendix D.2 (right after the LongCLI three-attempt protocol note); any later content — further appendix sections, acknowledgments, dataset URLs — was not available, so 'absence' claims apply only to the reviewed text.
- Title-page 'Website' and 'Dataset' appear only as link labels without URLs in the extraction, so the claimed public availability of the bank could not be verified either way.
- The extracted text opens with a hex-string artifact (BBAAD9C2...) before the date and title; the body otherwise extracts cleanly and every number cited in this review was legible in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.