SkillGym: Internalizing Human Skills into LLMs for Real-World Problem Solving
Zhilong Ge, Yuting Shao, Yutao Yang, Yuxuan Cai, Jie Zhou, Kai Chen, Bo Zhang, Qin Chen, et al.
cs.CL
Quick verdict
Turns human agent skills into 2,756 verifiable training envs; SFT yields big gains, but harness confounds and an abstract-body number mismatch temper claims.
Summary
SkillGym converts human-written agent skills (drawn from the OpenClaw/ClawHub registry) into executable, code-verified training environments for LLM agents. A skill-to-task pipeline pairs manually designed per-category templates with structured skill cards, instantiates Docker tasks with outcome verifiers, and labels tasks via contrastive executions: tasks a reference agent solves with the skill but fails without it become Skill-Dependent (1,081 of 2,756; the rest are Verifier-Passed fallbacks). The authors release 2,756 environments across 12 categories and 8,364 successful trajectories (17.4% of 48,152 trials; averaging 49 tool calls and 63.4k logged tokens), collected with four harness-model pairs (Claude Code with DeepSeek V4 Pro/GLM-5.2; Codex with GPT-5.4/Nex-N2-Pro). Supervised fine-tuning of Qwen3.5-35B-A3B yields SkillGym-Agent: under Claude Code, +199 Elo on GDPval-AA v2, +19.10 points on Terminal-Bench 2.1, and +28.13/+14.68 points on SkillsBench v1.1 with/without skills; skill-free, it still beats its skill-assisted base on both harnesses, supporting the internalization claim. Comparisons against Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro rely on publicly reported scores under different harnesses rather than controlled runs. Main limitations: no significance testing or variance estimates, a system-prompt confound in the Codex comparison, an abstract-versus-table inconsistency (12.38 vs 14.68), and reinforcement learning remains future work.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Human-written agent skills can be systematically converted into executable, code-verified training environments at scale (2,756 environments across 12 categories, released publicly).
- Supervised fine-tuning on verified multi-harness trajectories substantially improves Qwen3.5-35B-A3B: +199 Elo on GDPval-AA v2, +19.10 points on Terminal-Bench 2.1, and +28.13/+14.68 points on SkillsBench v1.1 with/without skills under Claude Code.
- Skill-free fine-tuned agents surpass their skill-assisted base under both harnesses, indicating internalized rather than merely instruction-followed procedural competence.
- The 35B SkillGym-Agent reaches 51.47% on skill-assisted SkillsBench v1.1, above reported (non-controlled) scores of Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro (Preview).
- Teacher and harness mixing improves execution-oriented metrics but can reduce GDPval-AA v2 Elo, so data-mixing benefits are capability-dependent.
Strengths
Novel and well-motivated resource direction
Turning registry skill documents into verifiable training environments is distinct from tool-learning corpora and skill libraries; Table 3 shows SkillGym is the only listed dataset combining human-skill origin, executable environment, code-based verifier, and long trajectories (49 tool calls and 63.4k tokens on average).
Epistemically careful acceptance protocol
Eq. 2 (§3.2) separates executable feasibility from empirical skill dependence, explicitly frames the contrastive label as configuration-relative evidence rather than proof, and retains 1,675 Verifier-Passed fallback tasks instead of discarding them.
Large, internally consistent released artifact set
Strongest evidence is controlled and consistent
Transparent appendix reporting
Weaknesses
No statistical uncertainty anywhere
Internal numeric contradiction in headline claims
Confounded and non-controlled comparisons
Single-shot contrastive labeling
Unexamined contamination and partial coverage
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SkillGym (arXiv 2609.27717v2; ECNU ICALK + Shanghai AI Laboratory) addresses a real gap: human-written agent skills (SKILL.md-style artifacts from the OpenClaw/ClawHub registry) serve as inference-time instructions but are rarely used as training signal. The paper makes three claims: (1) a skill-to-task pipeline converts registry skills into 2,756 executable, code-verified Docker environments across 12 major / 63 sub-categories; (2) it releases 8,364 verified successful trajectories (17.4% of 48,152 sampled trials; mean 49 tool calls, 63.4k logged tokens, 35.2 steps) from four harness-model pairs; (3) SFT of Qwen3.5-35B-A3B yields SkillGym-Agent with large same-backbone gains, and skill-free performance exceeding the skill-assisted base evidences internalized procedural competence. Claims are mostly well-scoped — the authors repeatedly flag that public-score comparisons are not controlled — but the abstract both leans on those comparisons ('exceeding reported scores for Claude Sonnet 4.6, GPT-5.4 Mini, and DeepSeek V4 Pro') and contains a numeric error (12.38 vs Table 4's 14.68 for SkillsBench w/o skills).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SkillGym?
SkillGym is a framework from East China Normal University and Shanghai AI Laboratory that converts human-written agent skills (sourced from the OpenClaw/ClawHub skill registry) into Docker-based training environments with code-based outcome verifiers. It releases 2,756 environments across 12 categories, 8,364 verified successful agent trajectories, construction code, and a fine-tuned 35B model on GitHub and Hugging Face.
How much does SkillGym training improve an LLM agent?
Supervised fine-tuning of Qwen3.5-35B-A3B produces SkillGym-Agent. Under the Claude Code harness it gains +199 Elo on GDPval-AA v2, +19.10 percentage points on Terminal-Bench 2.1, and +28.13/+14.68 points on SkillsBench v1.1 with/without skills. Under Codex the base starts far lower, so relative gains are larger (e.g., Terminal-Bench 10.11% to 46.07%), though that comparison also changes the system prompt.
Does SkillGym-Agent beat Claude Sonnet 4.6 or GPT-5.4?
On skill-assisted SkillsBench v1.1 it scores 51.47%, above the publicly reported 47.2% for Claude Sonnet 4.6, 41.4% for GPT-5.4 Mini, and 50.1% for DeepSeek V4 Pro (Preview). These are reported scores measured under different agent harnesses, so the paper does not treat them as controlled head-to-head results; on GDPval-AA v2 and Terminal-Bench 2.1, frontier models still lead.
Does SkillGym use reinforcement learning?
No. All reported training is supervised fine-tuning on verified successful trajectories. The environments ship code-based verifiers that could provide outcome rewards for RL, but the paper lists reinforcement learning as future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3-6 appear as garbled text fragments in the extraction; chart-level values could not be visually verified, so the review relies on Tables 1-11.
- GDPval-AA v2 Elo judgment counts and pairwise-comparison methodology are not in the extracted text, limiting assessment of that metric's reliability.
- The released model URL points to a Hugging Face datasets path (.../datasets/ecnu-icalk/SkillGym-Agent), so model availability could not be confirmed from the text alone.
- Table 11 public-reference rows use heterogeneous harnesses per §D.1; any cross-model statements here inherit that limitation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.