SWE-Prometheus: Measuring Engineering Governance Improvements in Real-World Repositories
Jiajun Wu, Leixin Sun, Zihan Tan, Yitao Liu, Shuo Li, Jiaru Qian, Shanghaoran Quan, Chuangxin Zhao, et al.
cs.AI · cs.SE
Quick verdict
Benchmark for LLM agents improving repo governance; rigorous controls (no-op, template, gate ablation), but single-rollout LLM-judged scoring caps precision.
Summary
SWE-Prometheus reframes repository-level agent evaluation: instead of fixing a supplied issue, a coding agent receives a fixed snapshot of one of 60 real GitHub repositories and an open-ended brief to improve engineering governance across six dimensions (Tests & CI, quality gates, documentation, maintainability, reproducible environment, dependency & security). The evaluator reconstructs base and treated states, runs identical probes in a clean environment, applies characterization-test behavior gates graded by mutation testing, and scores headroom-normalized improvement (NGI) from paired evidence rated by two independent teacher models. Ten models, run once per repository, were compared on a 22-repository public subset: mean NGI spans 0.0568-0.5760 and observed breakage rates 0-22.7%. The auxiliary study on a frozen ten-repository batch is the paper's most valuable part: a no-op condition (median 0, SD 0.073), a repository-blind template scoring 0.272 whose gains stay in Tests & CI, quality, and documentation and never improve D5/D6, and matched per-repository analyses showing only Kimi-K3 and GLM-5.3-Flash reliably beat the template. Main limitations, largely self-acknowledged: one rollout per model-instance pair, a scaffold confound (Claude-Opus-5 on Claude Code, others on pi), teacher-judge agreement reported only on no-op evidence, 38 private instances, and repositories selected for governance headroom. A benchmark paper unusually honest about its own resolution limits.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Repository engineering governance improvement is a distinct, measurable agent capability that complements issue-level functional repair.
- On the 22-repository public subset, ten models' mean NGI spans 0.0568-0.5760 with behavior-breakage rates 0-22.7%, so improvement and delivery reliability must be reported as separate axes.
- A repository-blind template reaches mean NGI 0.272, improving Tests & CI, Quality Gates, and Documentation on nearly every repository but Reproducible Environment and Dependency & Security on none — proving part of the score is reachable without repository diagnosis.
- Only Kimi-K3 (9-1) and GLM-5.3-Flash (8-1) beat the stronger template on a majority of the matched ten-repository batch, and Kimi-K3 ranks first under all full-pool failure-handling policies (broken runs scored 0, −1, or −3).
- Behavior-breakage rates are only interpretable alongside mutation-tested gate strength, which ranges from detected to none across 36 of 60 instances.
Strengths
Diagnostic controls rarely seen in agent benchmarks
No-op (median 0, SD 0.073), repository-blind mechanical template (NGI 0.272), rule-based baseline (0.254), matched per-instance comparison (Table 19), and gate ablation jointly separate capability from measurement artifact; the Efficient-WAM case (assert True passes while Ruff still reports 117 errors, §5.7) makes the artifact-versus-effect gap concrete.
Execution-backed, paired evaluation design
Base and treated states are reconstructed independently and probed identically in clean environments; characterization gates validated on base are re-run on treated, and mutation testing grades gate strength (24 detected / 27 blind / 3 vacuous / 6 none, Table 5) so zero-breakage under a vacuous gate is not read as safety evidence.
Statistical self-honesty about resolution
Transparent provenance and bug disclosure
Actionable negative result localizing rubric weakness
Weaknesses
Judge validity under-evidenced
Single rollout with unbalanced denominators
Cross-scaffold confound
Coverage and gating limits of the release
No-op drift not decomposed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SWE-Prometheus (arXiv 2609.29465) proposes a benchmark for *repository engineering governance*: an agent receives a pinned snapshot of a real GitHub repository plus an open-ended brief and must diagnose and retrofit six dimensions — Tests & CI (D1), Code Quality Gates (D2), Documentation & Collaboration (D3), Structure & Maintainability (D4), Reproducible Environment (D5), Dependency & Security (D6). Unlike SWE-bench, there is no issue list, no reference patch, no hidden tests. The central claims: (1) governance improvement is a distinct, measurable capability; (2) ten models on a 22-repo public subset span mean NGI 0.0568–0.5760 with 0–22.7% behavior breakage (Table 1/15); (3) a repository-blind template reaches NGI 0.272, showing D1–D3 credit is available without diagnosis while D5/D6 never improve; (4) only Kimi-K3 and GLM-5.3-Flash reliably beat the template; (5) improvement and delivery reliability are separate axes — Kimi-K3 leads all three full-pool failure-handling policies (broken = 0/−1/−3, Table 18).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SWE-Prometheus?
A benchmark (arXiv 2609.29465) that evaluates whether LLM coding agents can improve a real repository's engineering governance — tests & CI, code quality gates, documentation, maintainability, reproducible environment, and dependency/security health — given only a fixed repository snapshot and an open-ended objective, with no defect list or reference patch.
How is SWE-Prometheus different from SWE-bench?
SWE-bench gives the agent a known issue and scores whether its patch passes target tests. SWE-Prometheus gives no issue: the agent must find governance gaps itself, and the evaluator scores paired before/after evidence from identical probes run in a clean environment, plus behavior gates that catch regressions to existing functionality.
Which models performed best on SWE-Prometheus?
On the 22-repository public subset, Kimi-K3 and GLM-5.3-Flash tied for the highest mean NGI (0.5760), but GLM-5.3-Flash broke existing behavior on 5 of 22 repositories versus 1 for Kimi-K3. When broken runs are counted as failures across the full pool, Kimi-K3 ranks first under every penalty policy examined (0, −1, −3).
What is the template baseline finding and why does it matter?
A patch that blindly adds standard governance files (CI workflow, lint config, docs) scores mean NGI 0.272 — beating several frontier models — but improves only Tests & CI, Quality Gates, and Documentation, never Reproducible Environment or Dependency & Security. This shows those three dimensions can earn score without any repository-specific diagnosis, and the authors propose tightening their rubrics.
Is the SWE-Prometheus dataset public?
Partially: 22 of 60 repositories form the public comparison set, with machine-readable per-instance records, frozen auxiliary manifests, and table-regeneration scripts promised in the release. 38 instances are kept private for internal evaluation, and the paper acknowledges the public split is selected for governance headroom rather than randomly sampled.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 3-6 are available only as captions in the extraction; figure-level distributions (e.g., per-instance NGI spread in Fig. 6) could not be independently verified.
- Two text spans were truncated (part of §2/§5.10 and Appendix D.5), and the abstract is cut mid-sentence.
- External release artifacts (Homepage, Hugging Face, GitHub links in Fig. 2) are referenced but their actual availability could not be verified from the text.
- Model names and the pi scaffold are reported as given; the paper provides no model-version or API details beyond scaffold labels.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.