Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
Jianlyu Chen, Yuyang Hu, Hongjin Qian, Jiawei Liu, Wenqing Wei, Xiaolong Chen, Defu Lian, Zhicheng Dou, et al.
cs.AI · cs.CL
Quick verdict
Distills 1,000 ML repos into 5k+ verified skills that lift a fixed GPT-5.5 Codex agent on 4 benchmarks; strong gains, but cost and contamination need scrutiny.
Summary
The paper argues that autonomous ML research agents lack an 'operational knowledge' layer between model backbone and harness, and proposes DisCo, which distills declarative sources (GitHub repositories, papers) into verified agent-facing skills — three-layer packages of SKILL.md entry points, references, and scripts — organized into skill graphs with progressive disclosure. Task-agnostic distillation yields the AREX-Skill Library: 5,353 skills from 1,000 widely used ML repositories, routed by an LLM-induced taxonomy of 20 areas and 178 capability families. Task-oriented distillation builds skills on demand for concrete tasks. Holding a GPT-5.5 backbone, Codex harness, and per-task runtime budget fixed, adding skills raises MLE-bench Any-Medal from 31.11% to 72.89% (75 competitions, three runs), PaperBench from 29.45% to 39.59% (20 papers), FrontierCS from 70.63 to 77.14 (188 tasks; 95% CI [3.41, 9.83]), and PassNet AS Score from 1.343 to 1.531, though Fast_1 slightly declines. Main limitations: one-time construction costs (~$40/repo; up to 24 GPU-hours exploration per MLE-bench task) are excluded from the matched-budget comparison; task-oriented skills are refined via trials on benchmark development problems (residual contamination risk); statistical rigor is uneven (SEM only on MLE-bench; no reported variance on PaperBench/PassNet). Code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Operational knowledge is a distinct layer K beyond the model Mθ and harness H (A_res = (Mθ, H, K)), and packaging it as skills measurably improves a fixed research agent.
- Task-agnostic distillation scales to a public library of 5,353 verified skills from 1,000 ML repositories, routed over 20 areas and 178 capability families via progressive disclosure.
- With backbone (GPT-5.5), harness (Codex), and per-task runtime budget held fixed, skills raise MLE-bench Any-Medal 31.11%→72.89% (+134.3%), PaperBench 29.45%→39.59% (+34.4%), FrontierCS 70.63→77.14 (+9.2%), and PassNet AS 1.343→1.531 (+14.0%).
- Gains are not explained by extra compute: on FrontierCS, per-task usage increases but gain–usage Spearman ρ ≤ 0.015, and no-sub-agent tasks still gain 5.54 points on average.
Strengths
Clean conceptual framing with formal grounding
Sec. 2.2 formalizes the agent as A_res = (Mθ, H, K) (Eq. 4) and splits operational knowledge into capability vs. usage policy, which the three-layer skill design (SKILL.md / references / scripts, Eq. 6) realizes directly; the harness-governs-how vs. knowledge-governs-what distinction is crisp and testable.
Unusually disciplined construction pipeline for LLM-generated artifacts
A.1: per-repo include/exclude evidence maps, live API inspection before claims enter skills, assertion-backed usability cases, native checks classified pass/skill_gap/native_fail/skip_unsafe/skip_not_selected, static gates (link integrity, self-containment, provenance, local-path leakage), localized repair loops, and a taxonomy induced via locator/judge dual calls over 100 batches of 10 repos with explicit convergence gates.
FrontierCS evaluation meets modern standards
Explicit contamination hygiene
Honest reporting of regressions and failure modes
Scale with verifiable structure
Weaknesses
Headline relative gain rests on a weak self-run baseline and cross-harness leaderboard comparison
Runtime-only budget parity overstates deployment cost parity
Uneven statistical rigor across benchmarks
PassNet aggregate masks a mixed per-metric picture
Verification quality is largely self-graded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets autonomous ML-research agents and argues the standard two-component view (backbone Mθ + harness H) omits a third layer: operational knowledge K — executable, domain-specific know-how (Eqs. 2–4, Sec. 2.2). DisCo instantiates K as Anthropic-style skills (SKILL.md / references / scripts, Eq. 6) grouped into skill graphs, produced by a four-stage distillation loop (scope → ground → construct → verify, Eq. 8) in two forms: task-agnostic (repository/paper anchors) and task-oriented (task anchors). Scaled over 1,000 widely used ML repos, this yields the AREX-Skill Library: 5,353 skills, 20 areas, 178 capability families, 2,209 exact assignments, 700 multi-family repos (Sec. 4.1, Appendix B). Central claims: with a GPT-5.5 backbone, Codex harness, and matched runtime budget fixed, skills lift MLE-bench Any-Medal 31.11%→72.89% (+134.3%), PaperBench 29.45%→39.59% (+34.4%), FrontierCS 70.63→77.14 (+9.2%), PassNet AS 1.343→1.531 (+14.0%) (Tables 1–4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the AREX-Skill Library?
It is a public collection of 5,353 agent-readable 'skills' — three-layer packages consisting of a SKILL.md entry file, reference documents, and executable scripts — automatically distilled and verified by the DisCo pipeline from 1,000 widely used ML GitHub repositories, and organized by a router over 20 areas and 178 capability families so an agent loads only the relevant branch during a task.
How much do distilled skills improve agent performance?
Under a fixed GPT-5.5 backbone and Codex harness with matched per-task runtime budgets, adding skills raised MLE-bench Any-Medal from 31.11% to 72.89% (+134.3%), PaperBench replication from 29.45% to 39.59% (+34.4%), FrontierCS from 70.63 to 77.14 (+9.2%, 95% CI [3.41, 9.83]), and PassNet AS Score from 1.343 to 1.531 (+14.0%), though PassNet Fast_1 slightly declined.
Is the cost of building the skills counted in the performance comparison?
No. One-time skill construction — e.g., up to 24 GPU-hours of exploration per MLE-bench task and about $40 per repository for the library — is reported separately and excluded from the matched runtime budget, so the headline comparison reflects runtime-only cost parity, not total cost per task.
Is code or data available for this paper?
Yes. The paper links a public repository (github.com/VectorSpaceLab/AREX-Skill) containing the skill library and router; the skill-construction protocols are detailed in Appendices A.1 and A.2.1–A.2.4, though rerunning headline benchmark numbers requires frontier-model API access, official benchmark graders, and GPU resources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt (head and tail preserved, middle marked '[...sampling gap...]'); sections in the gaps — likely including related work, parts of Section 3, some experiment subsections, and parts of Appendices A.2.1–A.2.3 — could not be assessed, so judgments about any additional ablations, statistical tests, or human-audit details located there were not attempted.
- The abstract supplied in the task metadata is cut off mid-sentence ('These gains come from adding dis'); the complete abstract embedded in the paper text was used instead.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.