WiseSpec: Requirements-Driven Agents for Code Generation
cs.SE · cs.AI
Quick verdict
Requirements-refinement agent claims +13.17% avg %Resolved across 3 SWE-benchmarks and 2 LLMs; gains consistent, but no ablations, cost data, or code release.
Summary
WiseSpec is a requirements-driven agent framework for repository-level code generation, motivated by the observation that coding agents improve how tasks are solved while treating possibly defective task descriptions as reliable input. The framework has three components: (1) Requirement Generation iteratively retrieves code context along program dependencies and organizes it into a requirement DSL with nine primary and seventeen sub-attributes; (2) Requirement Quality Assessment reformulates requirement quality as execution-based evaluation, generating code and tests from the requirement and requiring all generated tests to pass; (3) Requirement Refinement diagnoses deficiencies as Conflict, Omission, or Ambiguity, applies alignment rules, and greedily retains the highest-scoring candidate while recording failed feedback as counterexamples. On 100-instance samples of SWE-bench-Lite, Verified, and Pro with DeepSeek-V3.2 and Qwen-Plus, WiseSpec achieves the best %Resolved in all six settings (e.g., 51% vs. 46% for Claude Code on Verified with DeepSeek; 35% vs. 24% on Pro), with %Applied near 100%, an average reported gain of 13.17%, and Wilcoxon p < 2.5×10⁻⁴; an Opus-4.8 run on Pro shows 56% vs. 53%. Main limitations: no ablations isolating the DSL, retrieval, or refinement loop; no cost, latency, or iteration-count reporting; self-generated tests as the quality proxy raise circularity concerns; and no code, seeds, or sampled-instance lists are released, limiting reproducibility.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 14 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The quality of task descriptions — not agent capability — is a fundamental bottleneck in repository-level code generation with LLMs (§1).
- Requirement quality can be quantitatively assessed by reformulating it as execution-based evaluation: code and tests generated from the requirement must all pass (§2.2).
- Iterative requirement refinement using a Conflict/Omission/Ambiguity taxonomy with alignment rules and greedy selection yields higher-quality requirements and better code (§2.3).
- WiseSpec achieves the best %Resolved in all six benchmark × LLM settings with an average improvement of 13.17% and Wilcoxon p<2.5×10⁻⁴ (§3, Table 1).
- The approach generalizes to stronger models: with Claude-Opus-4.8 on SWE-bench-Pro, %Resolved rises from 53% (Claude Code) to 56% (§3).
Strengths
Novel reformulation of requirement quality as execution-based evaluation
Section 2.2 converts an inherently fuzzy property (requirement quality) into a measurable proxy: generate code plus tests from the requirement and accept only if all generated tests pass. This is the paper's most interesting conceptual contribution, bridging requirements engineering and agent-based code generation.
Consistent empirical wins across all six benchmark × LLM settings
Table 1 shows WiseSpec with the best %Resolved in every cell: DeepSeek 39/51/35 vs. Claude Code 36/46/24 on Lite/Verified/Pro; Qwen 28/37/26 vs. 26/33/20. The SWE-bench-Pro DeepSeek result (35% vs. 24%) is the strongest single margin among strong baselines.
Includes statistical testing and a strong-model generalization check
Structured deficiency taxonomy with counterexample memory
Two-metric evaluation separating syntactic from functional correctness
Weaknesses
No ablation studies
Circular quality proxy via self-generated tests
Unexplained near-ceiling %Applied suggests an unreported retry confound
No cost, latency, or iteration-count reporting
Thin reproducibility: no artifacts, seeds, or sampled-instance lists
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a three-page ASE '26 short paper (single author, Tianjin University, advised by Junjie Chen; NSFC grant 62322208) proposing WiseSpec, a requirements-driven agent framework for repository-level code generation. The diagnosis (§1): coding-agent research improves how LLMs solve tasks but consumes possibly defective task descriptions verbatim. The remedy has three components: (i) Requirement Generation — dependency-guided iterative context retrieval organized into a requirement DSL with nine primary and seventeen sub-attributes (§2.1); (ii) Requirement Quality Assessment — reformulated as execution-based evaluation: code and tests are generated from the requirement, and the requirement passes only if all generated tests pass (§2.2); (iii) Requirement Refinement — deficiencies are typed as Conflict, Omission, or Ambiguity, alignment rules produce feedback, a greedy strategy retains the best-scoring candidate, and failed feedback is stored as counterexamples (§2.3). Headline claims: WiseSpec wins %Resolved in all six benchmark × LLM settings with an average improvement of 13.17% (Abstract, §3), gains of 2–29% in %Resolved and 11–63% in %Applied, and generalization to Claude-Opus-4.8 (53%→56% on SWE-bench-Pro).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WiseSpec?
WiseSpec is a requirements-driven agent framework for repository-level code generation, published at ASE 2026. It converts a raw task description into a structured requirement using a domain-specific language with 9 primary and 17 sub-attributes, scores the requirement's quality by generating code and tests from it (all generated tests must pass), and iteratively refines the requirement — diagnosing deficiencies as Conflict, Omission, or Ambiguity — before final code generation.
How much does WiseSpec improve over existing coding agents?
On 100-instance samples of SWE-bench-Lite, Verified, and Pro with DeepSeek-V3.2 and Qwen-Plus, the paper reports the best %Resolved in all six benchmark-model combinations, an average gain of 13.17 percentage points against the baselines' average (individual gains of 2–29 points; e.g., 35% vs. 24% for Claude Code on SWE-bench-Pro with DeepSeek), and Wilcoxon p<2.5×10⁻⁴. Against the strongest baseline alone the margins are smaller, and an Opus-4.8 run on Pro shows 56% vs. 53%.
Is the WiseSpec code or dataset available?
The paper text contains no code, artifact, or data release statement, and does not provide prompts, the DSL definition, alignment rules, sampling seeds, or the lists of sampled instances. Independent reproduction currently requires reimplementing the pipeline from the paper's high-level description.
What are the main criticisms of the paper?
Reviewers would likely point to: no ablation isolating the DSL, retrieval, and refinement components; no cost or iteration-count reporting for the multi-step loop; a possibly circular quality proxy (tests generated from the same requirement they certify); a single pooled p-value without per-comparison statistics or correction; near-100% %Applied suggesting an undisclosed retry mechanism; and no released code or instance lists.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's visual details (exact loop topology) could not be verified beyond its embedded text labels; the extraction contains no figure rendering.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.