SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents
Xin He, Yanlin Wang, Mingwei Liu, Jiachi Chen, Hongyu Zhang, Guanbin Li
cs.SE · cs.AI
Quick verdict
Claims functional tests miss review constraints; SWE-Gate benchmark (303 instances) supports, but single-run stats and synthetic-task realism need scrutiny.
Summary
The paper introduces SWE-Gate, a repository-level benchmark for software engineering agents that scores review-constraint compliance alongside functional correctness. Motivated by maintainers rejecting functionally correct patches during code review, the authors mine merged pull requests of seed repositories, extract atomic, objectively testable review constraints with LLM assistance plus rule-based filtering, and transfer constraint seeds into 75 compatible Python repositories, yielding 303 synthetic repair instances. Each instance pairs a functional test with a constraint test and three reference patches (mutant, non-compliant, gold) that must satisfy a four-row validation matrix executed in Docker. Evaluating GPT-5.5, GPT-5.4-mini, DeepSeek-V4-Flash, and GPT-4o-mini under a common Mini-SWE-Agent scaffold, 644 generated patches passed functional tests but 221 (34.3%) failed the constraint suite; joint success ranged from 4.3% to 52.8%. An input ablation shows that providing the constraint text raises constraint-following rates by 10.2–25.6 percentage points while slightly lowering functional success for three of four models. Main limitations: one generation per model–instance pair with no variance estimates or significance tests; fully LLM-assisted synthesis whose realism and contamination exposure are managed procedurally rather than measured; Python-only coverage with a single agent scaffold; and quality-assurance statistics deferred to supplementary material.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Functional test passage is an insufficient acceptance criterion: across four LLM backends, 221 of 644 functionally successful patches (34.3%) fail an accompanying executable review-constraint test.
- Review constraints mined from real PR review comments can be transferred into 303 validated, executable repository-level repair instances spanning 75 Python repositories.
- Providing the natural-language constraint raises constraint-following rates by 10.2–25.6 percentage points but does not improve functional success, which drops for three of four models.
- Constraint difficulty is category-dependent: scope generalization, lifecycle/resource cleanup, encoding/escaping, and schema/typing constraints are hardest, while sentinel-distinction and argument-ordering constraints are easiest.
Strengths
Deterministic dual-oracle design, not LLM-as-judge
Constraint compliance is decided by executing a separate constraint test inside the Table 1 validation matrix (R passes F; R+M fails F; R+M+N passes F fails C; R+M+G passes both), enforced in Docker during QA — a clear advance over the closest prior work [60], which relies on LLM-based verification the paper itself notes is subjective [61, 62].
Provenance-grounded, filtered constraint mining
Constraint seeds come from real merged-PR review comments with linked diffs and issues (Constraint Extraction section); rule-based filtering removes workflow/docs/test-only/formatting/naming requests, and a second LLM gate demands code evidence, a plausible functional-but-non-compliant repair, and separable oracles before a seed is retained.
Internally consistent, carefully hedged results
Controlled input ablation isolates constraint visibility
Weaknesses
Single-run evaluation with no statistical inference
Synthetic realism asserted by process, not measured
Contamination acknowledged but unmeasured
Narrow external validity and a missing iterative condition
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SWE-Gate (arXiv 2609.04167) is a benchmark paper targeting a real gap in repository-level repair evaluation: SWE-Bench-style suites accept any patch that passes functional tests, while real maintainers reject functionally correct patches during review for reasons such as backward compatibility or exception semantics — illustrated by the Pydantic PR #12657 example in the Introduction. The authors claim three things: (1) review-derived, objectively testable constraints can be mined from real PR reviews and transferred into executable repair tasks; (2) the functional and constraint dimensions are separable — 221 of 644 functionally successful patches from four LLM backends (34.3%, Table 3) fail the constraint suite; (3) explicit constraint text improves constraint-following (CFR +10.2 to +25.6 points, Table 4) without improving functional success. The core contribution is the dual-oracle evaluation protocol; the empirical results validate it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SWE-Gate benchmark?
SWE-Gate is a benchmark of 303 repository-level bug-fix tasks built from 75 open-source Python repositories. Unlike SWE-Bench, which scores a patch only by whether it passes functional tests, each SWE-Gate task also ships an executable 'constraint test' encoding a requirement raised in a real pull-request code review (e.g., backward compatibility or error-message content). A patch counts as fully successful only if it passes both suites.
What are the main results of the SWE-Gate paper?
Four LLM agents (GPT-5.5, GPT-5.4-mini, DeepSeek-V4-Flash, GPT-4o-mini) using the same Mini-SWE-Agent scaffold produced 644 patches that passed the functional tests; 221 of them (34.3%) failed the constraint test. Joint success rates ranged from 4.3% (GPT-4o-mini) to 52.8% (GPT-5.5). Telling the agent the constraint in advance raised constraint-following by 10.2–25.6 percentage points, but slightly lowered functional success for three of the four models.
Are SWE-Gate tasks real bugs and real review comments?
Partly. The constraints are mined from real review comments on merged pull requests, but each task is a synthetic transfer: an LLM pipeline moves the constraint plus a matching bug pattern into a different, compatible repository and generates mutant, non-compliant, and gold reference patches. Every instance must pass a Docker-validated four-case matrix, and released instances are manually inspected, so realism is engineered rather than inherited from a single real PR.
Is the SWE-Gate code and dataset available?
Yes. The paper states that the replication package, including code, data, and experimental results, is available at github.com/DeepSoftwareAnalytics/SWE-Gate.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's code blocks are scrambled by PDF extraction (patch/test labels displaced relative to their diffs), so the worked example could not be fully verified; the Table 1 validation-matrix logic and surrounding prose are internally consistent.
- Supplementary material (failure taxonomy, stage-wise rejection counts, domain mappings) is referenced but not part of the provided text, so QA statistics could not be verified from this extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.