Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead
Fengshuo Liu, Ying Liu, Ruize Sun, Lie Luo, Siyuan Guo, Siyuan Guo
cs.SE · cs.AI
Quick verdict
Auditing 254 SWE-bench submissions, the paper argues top-entry leaderboard gaps are statistically inseparable and scaffold-dependent - a rigorous, reproducible, mostly convincing evaluation audit.
Summary
The paper audits whether fine-grained rankings atop the SWE-bench Verified leaderboard are statistically supported, using 254 published submissions across four splits without running any models. Three findings emerge. First, resolution collapse: the top ten submissions share 285 solved and 51 shared-failed instances, leaving only 164 informative ones (effective size ratio 0.33; 0.07 for the top two). Second, convergence mechanism: frontier solution sets are nested at a median of 0.935 versus a 0.774 score-implied baseline, so leaders differ by subsets, not complementarity. Third, provenance: reconstructed model-scaffold metadata shows within-model scaffold ranges up to 29.8 percentage points, exceeding the 8.8-point top-thirty spread, with six of nine interaction tests surviving Holm correction (explicitly observational). Exact paired McNemar tests separate none of 29 adjacent Verified top-thirty pairs, while the larger, less-saturated Test split separates 14 of 23, showing the failure is saturation-specific. A leader-based rule yields three descriptive tiers (two after Holm); the authors stress non-rejection is not equivalence. They release a five-step audit protocol, show instance retirement adds no paired power, and estimate instance budgets. Limitations: single-run submissions, only 46% usable provenance tags, one benchmark family, and observational confounding of scaffold effects.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SWE-bench Verified's frontier has converged: the top ten submissions share 285 solved and 51 failed instances, leaving 164 informative ones, and solution sets are nested at 0.935 against a 0.774 score-implied baseline.
- No adjacent pair in the Verified top thirty is separable by exact McNemar at alpha = 0.05 (smallest adjacent p = 0.545), while the less-saturated Test split separates 14 of 23 pairs, so the ranking is unreadable specifically where it is read.
- A leaderboard score is a property of the model-scaffold pair, not the model alone: within-model scaffold ranges reach 29.8 percentage points with orderings reversing across models, though these are observational associations, not causal effects.
- Adding instances of the existing character buys almost no resolution (~26,000 needed at observed discordance rates); resolution comes from instances that break solution-set nesting, and instance retirement adds spread but zero paired power.
- Leaderboards should publish descriptive tiers with an explicit linkage rule, comparison-set-specific effective size, and machine-readable model-scaffold provenance, instead of strict ranks the data cannot support.
Strengths
Fully recomputable audit of public artifacts
Every number derives from per-instance harness verdicts the maintainers already publish (Section 2); the released pipeline ships a frozen manifest, seeds, and input hashes, and the paper correctly notes per-instance publication is what makes the paired tests possible at all.
Counter-test design avoids a blanket claim
Section 5 / Table 4 runs the identical McNemar procedure on all four splits: 0/29 separable on Verified and Lite versus 14/23 on the 2,294-instance Test split, localizing the failure to a converged comparison set rather than the benchmark family or the test itself.
Alternative explanations are tested, not just acknowledged
Statistical hygiene above the bar for this literature
Pre-empts the tempting wrong fix
Weaknesses
The model-scaffold analysis rests on fragile reconstructed provenance
One run per submission leaves run-to-run variance unquantified
Missing-verdict convention unexamined despite being the paper's own recommendation
Nesting is compared only to an independence null
External validity untested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does this paper prove the top coding agents on SWE-bench are equally good?
No. The authors explicitly state that failing to reject paired McNemar tests does not establish equivalence and define no practical equivalence margin. Their conclusion is narrower: the published per-instance evidence cannot statistically order the top thirty on SWE-bench Verified, so the leaderboard should present descriptive tiers rather than strict ranks.
Why can't the SWE-bench Verified leaderboard distinguish its top entries?
Because the leading systems solve almost exactly the same problems. The top ten all resolve 285 of the 500 instances and all fail the same 51, leaving only 164 instances that could tell them apart. On those, exact paired McNemar tests cannot separate any of the 29 adjacent top-thirty pairs at alpha = 0.05. On the larger, less saturated Test split the same test separates 14 of 23 pairs, showing the problem is saturation, not the statistics.
What is the model-scaffold issue the paper raises?
A leaderboard score comes from a model driven by a scaffold (the agent loop, tools, and control policy). Reconstructed public metadata shows the same model can score up to 29.8 percentage points differently across scaffolds - more than the entire spread among the top thirty entries - and scaffold orderings reverse depending on the model. So a score cannot be attributed to the model alone, though this analysis is observational.
Would adding more tasks to SWE-bench fix the ranking?
Mostly not. At the observed rates of disagreement between adjacent leaders, roughly 26,000 additional instances of the same character would be needed for paired statistical significance. What helps is curating instances that break the nesting of shared successes (an estimated imbalance rate of 0.2 needs about 900 instances for the reference pair); simply retiring solved instances widens the score spread but adds zero statistical resolving power.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are available only as captions and embedded text labels; visual content (e.g., the separability bars in Fig. 3 and the tier boundary in Fig. 2) could not be independently verified.
- The claimed reproducibility package (frozen manifest, seeds, hashes) and the GitHub repository were not inspected; codeAvailable reflects the paper's statement only.
- This is a v1 arXiv preprint (15 Sep 2026) and has not, to this reviewer's knowledge, completed peer review; several cited works ([5], [7], [13], [17], [18]) are themselves preprints.
- The extracted text appears complete (all sections and references present), but tables were reconstructed from flattened text and minor formatting could obscure details.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.