Who Holds the Pen? Let Specifications, Not Agents, Sign Off
Haiqing Li, Xin Ma, Yinhao Wu, Wenliang Zhong, Feng Jiang, Thao M. Dang, Xiao Hu, Hehuan Ma, et al.
cs.AI · cs.MA
Quick verdict
SpecHarness moves completion authority from agent to external runtime, cutting gaps on two benchmarks; gains entangle commitment with repair feedback.
Summary
The paper argues that LLM agents which both act and declare their own completion lack a specification authority boundary, producing two gaps: understanding-execution (requirements understood but unmet) and state-authority (self-claimed completion not proving the required state). On SkillsBench, an LLM compiler frozen on a task-disjoint development set extracts 509 source-grounded task directions from agent-visible materials across 87 tasks; across seven models only 79.6-86.4% of directions are satisfied, and ungated completion claims exceed official pass rates by 28.7-37.9 points. SpecHarness operationalizes the proposed State Authority Principle: visible specifications are compiled into source-linked obligations; agents propose actions, repairs, and finalization, but only admissible evidence from qualified validators can commit versioned obligation state, and finalization requires all fresh mandatory obligations satisfied. On SkillsBench, macro pass rises from 61.1% to 73.1%, and on GuideBench (1,042 tasks) from 86.2% to 91.3%, with paired bootstrap CIs and Holm-corrected McNemar tests significant for all seven models; costs are 1.53x tokens and 1.24x wall-clock. Adapted baselines (Agentic Rubrics, VeriMAP, AgentSpec, SatLM, RvLLM) fare worse. Main limitations: the comparison evaluates the full architecture including feedback and repair rather than isolating commitment under matched compute; the measurement surface is compiler output with only 69.4% fine-grained alignment to official tests; and no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM agents that self-declare completion lack a specification authority boundary, producing measurable understanding-execution and state-authority gaps: only 79.6-86.4% of 509 extracted directions are satisfied and ungated claims exceed official pass rates by 28.7-37.9 pp across seven models.
- The State Authority Principle — agents propose; an external runtime commits state only from admissible evidence of qualified providers — can be operationalized as SpecHarness, which compiles visible specifications into source-linked obligations with versioned state and evidence-gated finalization.
- SpecHarness improves all seven models on both benchmarks (macro SkillsBench pass 61.1% to 73.1%; GuideBench 86.2% to 91.3%), with paired bootstrap CIs and Holm-corrected McNemar tests significant throughout.
- The guarantee is deliberately scoped: only 44% of SkillsBench hard obligations receive preventive no-bypass mediation, ambiguous or subjective requirements remain advisory, and the guarantee excludes undeclared channels and compromised trusted components.
Strengths
Crisp conceptual formalization
The State Authority Principle is operationalized as a checkable state-write invariant (Eq. 8) with explicit admissibility (Eq. 9), atomic ledger updates (Eq. 10), freshness (Eq. 18), and a finalization gate (Eq. 12); Figure 2's taxonomy of post-hoc verification, completion gating, and runtime enforcement cleanly positions the contribution.
Protocol hygiene around evaluation independence
The compiler is selected on a task-disjoint 14-task dev set and frozen before execution (Appendix A.1, Table 8); official verifiers are used only post-hoc for alignment (573/585, Table 9) and final pass; the same frozen validators measure all conditions read-only while only SpecHarness consumes their evidence online.
Adversarial self-auditing of its own guarantees
Above-average statistical reporting
Honest cost and scope accounting
Weaknesses
Headline comparison is confounded with feedback and repair
Measurement surface defined by the compiler itself
No artifact release stated
Baseline adaptations may understate competitors
Raw S-A is definitional
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Who Holds the Pen? Let Specifications, Not Agents, Sign Off
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the state-authority gap in LLM agents?
It is the discrepancy between an agent's own completion claim and whether the specification-required state actually holds. In this paper, uncontrolled agents' completion claims were accepted 100% of the time while official verifiers passed only 52.9-71.3% of runs — a 28.7-37.9 percentage-point gap across seven models on SkillsBench.
What does SpecHarness do?
It is a runtime layer that converts agent-visible task instructions, guidelines, and skill files into source-linked obligations. The agent can only propose actions and completion; a qualified external validator must supply admissible evidence before the runtime commits versioned state, and the task can finalize only when all fresh mandatory obligations are satisfied.
How much does SpecHarness improve task success?
Across seven LLMs on 87 SkillsBench tasks, the macro official pass rate rose from 61.1% to 73.1%; on 1,042 GuideBench guideline-following tasks, from 86.2% to 91.3%. The cost was about 1.53x tokens and 1.24x wall-clock time versus uncontrolled agents.
Is code for SpecHarness available?
The paper text does not mention a public code or data release. Independent reproduction would require the OpenHands substrate, access to seven proprietary model APIs, and the authors' compiler prompts, development annotations, validator suites, and closure-audit tooling.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are available only as captions and surrounding text; diagram content (e.g., the SpecHarness workflow detail) could not be independently verified.
- The abstract is truncated mid-sentence in the extraction; the closing claim was reconstructed from the body and conclusion.
- Baseline implementations are adaptations by the authors; their fidelity to the original systems cannot be judged from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.