Skill Issue: Lessons from Optimizing Repository SKILLs for Coding Agents
Mykhailo Kozyrev, Andrei Kozyrev, Anton Podkopaev
cs.AI
Quick verdict
GEPA-mined repo SKILLs add +4.9pp on a paired agent score over 3 Kotlin repos; SkillOpt +0.1pp. Honest paper admits gains sit within agent variance.
Summary
This paper asks whether a repository SKILL — a plain .md file loaded into a coding agent's context — can be synthesized automatically with measurable benefit. Because bare repositories ship no benchmark, the authors mine tasks from three Kotlin repositories (kotest, ktor, koog) by reverting merged pull requests at a single frozen base commit, so each task re-introduces a real historical change and hidden FAIL_TO_PASS tests grade it; only repositories yielding at least 100 validated tasks qualify. The agent (Claude Code, Sonnet 4.6) is frozen and only the SKILL varies; each candidate is scored by pairing its rollout with the stored empty-seed rollout on the same task, anchored at 0.5. GEPA-found documents raise the mean held-out score by 4.9pp; SkillOpt's land at +0.1pp (Table 1: per-repo paired scores 0.525–0.567 vs 0.437–0.546; total spend $2,013.98, 69.2h). The authors candidly report these gains are not statistically significant on 20–26-task splits and cannot be separated from agent run-to-run variance. Qualitatively, a koog maintainer found real project knowledge in both documents, and on two live issues either SKILL roughly halved wall-clock time and cost. Limitations: three Kotlin repositories (two JetBrains-associated), one agent/model pair, a single maintainer reviewer, and no human-written SKILL baseline.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Tasks mined by reverting merged PRs at a single frozen base commit are harder (~53% baseline resolve) and stay current with the repository, unlike SWE-smith-style synthetic tasks that capable agents saturate.
- Across three Kotlin repositories, SKILLs found by GEPA raise the mean paired held-out score by 4.9pp, while SkillOpt's documents leave it at +0.1pp.
- Neither gain is statistically separable from the agent's run-to-run variance at 20–26-task splits; settling it would need more tasks than one repository's history supplies.
- A koog maintainer judged both optimized SKILLs to contain project knowledge otherwise only obtained by working in the repository, and would accept the GEPA document as a draft PR.
- On two live koog issues, the agent finished in roughly half the wall-clock time and at lower cost with either SKILL compared to none.
Strengths
Honest statistical framing
Abstract, §1 and RQ3 state explicitly that the +4.9pp (GEPA) and +0.1pp (SkillOpt) deltas are not statistically significant on 20–26-task splits and cannot be separated from the agent's run-to-run variance; §1 also quantifies why binary pass rates mislead (Sahoo et al.: 10.7% of 'passes' are lucky) and shows gskill's benchmark saturates at 100% with no skill under Claude Code + Sonnet 4.5.
Well-engineered, validated mining pipeline
Three-tier reverter (git apply --reverse, structural file undo, LLM rewrite with the patch re-derived from git), dynamic FAIL_TO_PASS validation with up to 30 regression guards, plus a static check that rejected 56/280 intermediate koog tasks — 36 of which had already passed dynamic validation (§A). Failure cases are documented: tracy kept 5/203 PRs, http4k <100 graded tasks from 700 PRs (§6, §C).
Paired seed-anchored objective with documented score search
Artifact-level evidence beyond the score
Full cost transparency
Weaknesses
Headline effect sits inside the noise
Narrow external validity
n=1 maintainer review
Missing natural baseline
SkillOpt collapse undiagnosed in the visible text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets repository-specific SKILL synthesis: optimizing a single .md file loaded into a coding agent's context, for one repository, with no external benchmark. Against prior work (gskill, SkillOpt) that evaluates on synthetic SWE-smith tasks a strong agent saturates (§1: 100% no-skill under Claude Code + Sonnet 4.5), the authors change the instrument twice: tasks mined as merged PRs reverted at one frozen base commit (§3.1), and a paired objective scoring each candidate against the seed's stored rollout on the same task (§3.2). Claims: (1) the mined tasks are harder (~53% baseline resolve) and repository-current; (2) GEPA finds documents worth +4.9pp on the paired held-out score while SkillOpt's land at +0.1pp; (3) neither gain is statistically separable from agent variance at 20–26-task splits; (4) the artifacts themselves pass a maintainer's reading.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper 'Skill Issue' claim about auto-generated SKILL files for coding agents?
It reports that SKILL documents found by the GEPA optimizer raise a paired held-out score by 4.9pp on average across three Kotlin repositories (kotest, ktor, koog), while SkillOpt's documents add 0.1pp. The authors state neither gain is statistically significant at 20–26 held-out tasks per repository, and supplement the numbers with a maintainer review and two live-issue experiments where a SKILL roughly halved the agent's time and cost.
How do the authors create benchmark tasks when a repository has none?
They mine the repository's merged pull requests and revert each change at a single frozen base commit, so the problem the PR solved reappears in the current codebase; tests that switch from passing to failing become the hidden FAIL_TO_PASS grading set. Repositories must yield at least 100 validated tasks to be usable — two of five attempted repositories failed this floor.
How much does it cost to optimize a repository SKILL this way?
The paper reports $2,013.98 total agent API spend and 69.2 hours of wall clock across three repositories and two optimizers, with a mean of $0.84 per agent rollout; individual optimization runs range from $182.48 to $484.81 (Table 1).
Do SKILL or AGENTS.md files actually help coding agents?
Evidence is mixed. This paper finds small, statistically insignificant paired gains from optimized SKILLs on its mined tasks, though a koog maintainer judged the documents to contain real project knowledge and the agent finished two live issues about twice as fast and cheaper with one. Related work it cites (Gloaguen et al., 2026) found generated AGENTS.md files produced non-significant changes overall.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: marked '[...sampling gap...]' sections skip parts of §4.1–4.3 (including RQ2/RQ3 detail and the detectability bound) and chunks of the appendices; judgments leaning on those parts are limited to what is visible.
- The excerpt ends mid-sentence in appendix H ('Enumerate every symbol t…'), so the final pages of the paper are not visible in the provided text.
- No code/data release statement is visible in the provided excerpt; per absence discipline it may exist in an unsampled portion, so codeAvailable=false reflects visibility, not verified absence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.