SkillSpec: Intent-Masked Specification Reasoning for Agent Skill Correctness
Yizhuo Zhang, Bo Kang, Yi Yang, Zhiyu Duan, Zhouteng Ye, Shunkun Yang
cs.SE · cs.AI
Quick verdict
Claims Hoare-style intent-masked reasoning surfaces agent-skill defects (763 across 515 skills, 61.2% precision); promising, but no recall, thin baselines.
Summary
The paper frames correctness of LLM-agent skills — bundles of SKILL.md instructions plus scripts — as a specification-consistency problem and proposes SkillSpec, a Hoare-style detection pipeline. A unified graph aligns workflow units parsed from SKILL.md with code entities extracted via a Tree-sitter IR; for each node the system derives an ExpectSpec from declared intent and multiple FactSpecs under an intent mask exposing four visibility layers (self, neighbors, lineage, holistic). Cross-view reasoning flags candidate defects, which an OpenCode/Harbor sandbox validates via probes for code or scenario corroboration for workflows. On 515 skills from SkillsBench and top-downloaded skills.sh repositories, manual review confirmed 763 defects in 239 skills (46.4%), with 61.2% precision overall under GPT-5.6-Sol (67.1% code, 55.4% workflow); results hold across GPT-5.6, DeepSeek-V4, and a local Qwen3.8-27B. Ablations attribute gains to explicit specifications (candidate confirmation 28.4%→38.2% workflow, 56.4%→63.8% code on SkillsBench-Script) and multi-view masking (48.2% vs 42.9–45.8% single-view confirmation). End-to-end cost is $2.585 per skill. Main limitations: recall is unmeasured, no statistical significance testing or inter-annotator agreement is reported, unsure-case counts are undisclosed, baseline comparison is limited to a vanilla no-spec ablation on 38 skills, and workflow defects rely on LLM-based scenario judgment rather than execution.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 2
● low 4
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agent skill correctness can be operationalized as Hoare-style consistency between declared intent (ExpectSpec) and encoded behavior (multi-view FactSpecs) over a unified workflow-code graph with an intent mask.
- On 515 real-world skills, SkillSpec identifies 763 manually confirmed defects across 239 skills (46.4%) at 61.2% precision with GPT-5.6-Sol (67.1% code, 55.4% workflow), remaining effective across five backends including a local Qwen3.8-27B.
- Explicit specifications and multi-view intent masking each improve candidate quality: confirmation rates rise 28.4%→38.2% (workflow) and 56.4%→63.8% (code) vs vanilla, and joint multi-view reasoning reaches 48.2% vs 42.9–45.8% for single views.
- Specification reasoning is consistently reliable for code nodes while plain-text nodes are the bottleneck, with defects concentrating at intent-implementation boundaries (74.0% of workflow defects requirement-side, 91.6% of code defects implementation deviations).
Strengths
Novel problem framing with solid ecosystem grounding
Recasts skill correctness as Hoare-style specification consistency over a hybrid NL+code artifact; Sec II grounds it in real data (884,669 skills on skills.sh, 7,577 with >1,000 installs; 3,512 single-file skills, median SKILL.md 7,352 chars, 72.6% Markdown-only).
Exceptionally self-consistent result accounting
Fig 9 totals (6,933 candidates = 4,633 validated + 2,156 refuted + 144 unresolved, 97.9% conclusive) reconcile exactly with per-model validated counts in Table VI; Table V node rates recompute correctly (inline_code 243/2,689 = 9.0%, functions 417/7,077 = 5.9%); cost table sums to $2.585/skill.
Honest evidence-tiering and FP forensics
Cross-backend robustness including a local model
Concrete, verifiable case studies and release
Weaknesses
Ground truth is soft: no recall, opaque unsure tier, no agreement statistics
Baseline coverage is minimal
Workflow 'validation' is LLM judgment, not execution
Internal inconsistencies and un-ablated design choices
Purely descriptive statistics, single-run results
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SkillSpec (Beihang University, arXiv 2609.06052) attacks a genuinely new QA target: defects in LLM-agent skills — SKILL.md instruction files bundled with scripts and resources — where failure modes include intent conflicts and description drift that surface as silent failures because the host model compensates at runtime. The paper motivates scale with an ecosystem anatomy (Fig 1): of 884,669 skills on skills.sh, 7,577 exceed 1,000 installs; 3,512 are a single SKILL.md (median 7,352 characters) and 72.6% are Markdown-only. Three claims are staked: skill correctness is a Hoare-style specification-consistency problem; a unified graph plus an intent mask (Self/Neighbors/Lineage/Holistic visibility) with sandbox validation can detect such defects automatically; and real skills are defect-ridden — 763 manually confirmed defects across 239 of 515 skills (46.4%) at 61.2% precision. The 'first to integrate static detection with dynamic validation across workflow and code artifacts in agent skills' novelty claim is plausible but adjacent to concurrent semantic-fuzzing work (their ref [10]).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SkillSpec?
SkillSpec is a framework from Beihang University that automatically finds defects in LLM-agent 'skills' — SKILL.md instruction files bundled with scripts. It converts the skill repository into a graph, derives what each step should do from the declared intent and what it actually does from the code/instructions, flags mismatches as candidate defects, and then tries to validate each candidate in an isolated sandbox.
How well does SkillSpec work?
On 515 real-world skills, manual review confirmed 763 defects across 239 skills (46.4% of skills) with 61.2% precision using GPT-5.6-Sol — 67.1% precision for code defects and 55.4% for workflow defects. The approach stayed effective across several model families, including a locally run Qwen3.8-27B, at a cost of about $2.59 per skill.
What kinds of skill defects does it find?
Mostly requirement errors and omitted constraints in natural-language workflows, and implementation deviations in code. Paper examples include a 'security auditor' skill whose script always reports success with an empty findings list, and a stock-ranking bug where Python treats the valid value 0.0 as false ('or -9999'), corrupting gainers ordering.
Is the SkillSpec dataset or code publicly available?
The paper states a public release of both the source code (github.com/IainZhang/SkillSpec) and the annotated 515-skill defect dataset (huggingface.co/datasets/IainZhang/SkillSpec).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 2, 7-9 extract as scattered axis labels and numerals; distribution shapes and Fig 9's latency detail were read from captions and stated totals rather than rendered plots.
- Tables III, VI, and VIII are dense multi-part tables whose cell mapping was reconstructed from the notes lines; totals cited here were cross-checked arithmetically, but individual per-subset cells could in principle be misattributed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.