Who Maintains Agent Skills? A Longitudinal Study of Human-Governed, AI-Assisted Skill Maintenance
cs.CL · cs.AI · cs.HC · cs.SE
Quick verdict
Claims the first longitudinal study of public skill-file maintenance; rigorous and honest, but a purposive 5-repo sample and LLM coding limit generalization.
Summary
This paper reports the first longitudinal, coded study of how public agent-skill files (SKILL.md) are maintained in practice. The authors mine the full git histories of five purposively selected repositories (getsentry, trailofbits, obra/superpowers, anthropics, cloudflare), yielding 873 commits, 143 skill files, and 254 substantive post-creation edits between October 2025 and June 2026, coded under three pre-registered codebooks for governance, curation operation, and trigger evidence. Three findings emerge: every substantive edit is authored or merged through a named human account, while 62% (158/254) carry an AI co-author trailer with sharp per-organization variation; operations skew toward enhancement (60%) over correction (38%), with pruning rare (4.3%); and a pre-registered rule-likeness axis failed its reliability gate (κ=−0.02) and is reported as a negative result. A powered transfer-task study (13 skills, 143 tasks) finds no average judged-quality benefit of maintained over earliest skill versions (−0.09, 95% CI [−0.28, +0.10]). The paper is notable for pre-registration discipline, honest negative results, and layered reliability analysis (blind human recode κ=0.72 on the two-way collapse; cross-family recodes exposing leaf-label fragility and trigger-axis instrument dependence). Main limitations: a small purposive sample with upper-bound-like maintenance depth, LLM-coded labels of moderate reliability, and a null result conditional on a judging harness below its own pre-registered reliability gate.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Every substantive SKILL.md maintenance edit across five public repositories is authored or merged through a named human account, while 62% (158/254) carry an AI co-author trailer, with sharp per-organization variation.
- Maintenance is genuine curation dominated by additions and corrections: 60% enhancement versus 38% corrective under the classical two-way collapse, with pruning (consolidation/deprecation) at only 4.3% of edits.
- A pre-registered rule-likeness axis failed its reliability gate (κ=−0.02), so reliably coding rule-likeness from commit artifacts remains an open measurement problem.
- A powered transfer-task study (13 skills, 143 tasks) finds no average judged-quality benefit of maintained over earliest skill versions (−0.09, 95% CI [−0.28, +0.10]), a bounded null under the constructed harness.
Strengths
Fills a genuinely unmeasured gap
Automated skill-curation systems (SkillOpt, SkillOS, Trace2Skill) are evaluated only against automated baselines and treat manual maintenance as an unmeasured bottleneck (§1, Table 1); this is the first coded longitudinal account of human SKILL.md maintenance, with a released corpus and replay protocol (Appendix E).
Pre-registration and negative-result discipline
The rule-likeness axis is reported as failed (κ=−0.02, with PABAK 0.34 and Gwet's AC1 0.44 correctly diagnosing a base-rate paradox) rather than reported as a distribution (§4, Appendix A); the pilot's positive Wilcoxon (p=0.02) is explicitly downgraded to a powered null in §7 rather than cherry-picked.
Claims tiered to measured reliability
Careful construct validity on governance
Confounded sub-analyses defused rather than hidden
Weaknesses
Purposive five-repository sample with pooled headline
Fine-grained operation taxonomy is fragile yet feeds design advice
Trigger-evidence axis fails its own cross-family validation
The §7 null is conditional on a harness below its own gate
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks who maintains public agent-skill files and what that maintenance consists of, positioning itself against automated skill-curation systems (SkillOpt, SkillOS, Trace2Skill) that evaluate only against automated baselines and treat human maintenance as an unmeasured bottleneck (§1, Table 1). Four claims: (1) all 254 substantive SKILL.md edits across five repositories are human-authored or human-merged, 62% (158/254) with an AI co-author trailer (§6, Table 3); (2) maintenance is genuine curation, 60% enhancement versus 38% corrective with pruning at 4.3% (§4, Figure 1); (3) the pre-registered rule-likeness axis failed its reliability gate at κ=−0.02 and is reported as a negative result (§4, Appendix A); (4) a powered transfer-task study finds no average maintenance benefit (−0.09, 95% CI [−0.28, +0.10]; §7, Appendix D). Claims are consistently scoped as descriptive; the abstract's 'human-governed, AI-assisted loop' framing is a fair reading of the data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the 62% AI co-authorship figure actually mean?
It means 158 of 254 substantive SKILL.md edits carried a Co-Authored-By commit trailer naming an AI tool. The paper is explicit this is a commit-metadata signal, not a measure of true AI use: it undercounts undisclosed use (Anthropic reports ~80% of its production code is Claude-authored while its skills repo shows almost no trailers) and can overcount auto-injected trailers, though an audit found clear auto-injection in only 1 of 25 trailer-present commits.
Did the study find that maintaining skills is useless?
No. A powered transfer-task study (13 skills, 143 tasks, blind cross-family judging) found no average judged-quality difference between maintained and earliest skill versions (−0.09 on a 1–5 rubric, 95% CI [−0.28, +0.10]). The authors call this a bounded null under their constructed harness — individual skills moved both ways — not evidence that maintenance is inert in native workflows.
Which repositories were studied and over what period?
Five public AI-skill repositories — getsentry/skills, trailofbits/skills, obra/superpowers, anthropics/skills, and cloudflare/skills — selected purposively for sustained activity, covering October 2025 to June 2026, 873 commits, 143 SKILL.md files, and 254 substantive post-creation edits.
Is the data and code from this paper publicly available?
Yes. The paper releases the commit-level corpus, the three codebooks with a datasheet, mining and analysis scripts, pinned commit hashes for all five repositories, archived solver/judge logs, and a replay protocol for evaluating automated skill curators, via github.com/megagonlabs/who-maintains-agent-skills. The labeled-edit statistics reportedly recompute offline without any API key.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 are scrambled by PDF extraction (legend counts and axis labels interleave); every number cited in this review was cross-checked against the prose, captions, and Tables 2–6, and Figure 2's percentage labels in particular should be read via Table 3 instead.
- Pinned commit hashes and the release URL are quoted as printed in the text; the actual existence and contents of the released repository could not be verified offline (no network access during review).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.