What Does an Agentic Software Engineering Benchmark Measure? Profiling Task Demands and Agent Behaviour Beyond What Category Labels Reveal
Radin Shayanfar, Keheliya Gallaba, Ahmed E. Hassan
cs.SE · cs.CL
Quick verdict
Claims SNC profiling shows benchmark labels hide task demands, backed by 14,922 agent trajectories; credible, though harness confound tempers RQ3.
Summary
The paper introduces the Spread–Novelty–Centrality (SNC) profile, three axes grounded in empirical software engineering metrics, computed over the gold patches of repository-level coding tasks: Spread (normalized patch entropy plus directory-tree radius), Novelty (added-versus-deleted line ratio), and Centrality (import-graph fan-in/out, 180-day churn, and size-times-complexity 'mass'). Applied to five agentic SE benchmarks totalling 2,487 instances (SWE-bench Verified, SWE-Gym Lite, FEA-Bench, FeatBench, FeatureBench) and 30 runs yielding 14,922 trajectories of Claude (Haiku 4.5/Sonnet 4.6/Opus 4.6 under Claude Code) and Qwen (3.5 9B/27B/397B under Qwen Code) on the Harbor harness, it reports three findings. RQ1: Scott-Knott ESD separates every benchmark pair on at least two axes, with separations traceable to curation choices (FeatBench's modify-only filter compresses Novelty; FeatureBench's test-first carve-out saturates it; FEA-Bench's new-component PRs sit at low Centrality). RQ2: on the 1,083 tasks Claude Opus 4.6 resolves, verbosity tracks problem-statement informativeness — over-production on hint-free FeatBench, under-production on FeatureBench (33.8% of its gold lines are comments/docstrings) — while exploration breadth is uniform. RQ3: resolved runs concentrate in low-SNC bins for all six configurations (BH-FDR-corrected χ², Cramér's V median 0.20), but success behaviours are family-specific: Claude resolves at gold parity (files parity share 0.17→0.41→0.54 with scale), Qwen over-produces at every scale (>2× band 0.6–0.7 on files), and under-editing marks failure for both. Limitations: Python-only benchmarks, one run per configuration, and a family contrast that entangles model with harness.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Nominal category labels are unreliable proxies for benchmark demands: every pair of the five benchmarks is statistically separated on at least two of the three SNC axes (Scott-Knott ESD, Figure 3).
- Curation decisions leave measurable SNC fingerprints: FeatBench's modify-only filter compresses Novelty, FeatureBench's test-first carve-out saturates it near 1.0, and FEA-Bench's new-component PRs yield the lowest FI-benchmark Centrality (Section 4.6).
- Agent patch verbosity tracks how much the problem statement reveals (over-production on hint-free FeatBench, under-production on FeatureBench where 33.8% of gold lines are comments/docstrings), while exploration breadth is uniform across benchmarks (RQ2, Figures 4/7).
- Resolved runs concentrate in low-SNC regions for every family and scale (all 42 SNC cells significant, Cramér's V median 0.20), whereas success behaviours are family-specific: Claude resolves at gold parity (files parity share 0.17→0.41→0.54 with scale), Qwen over-produces (>2× band 0.6–0.7 on files at every scale), and under-editing marks failure for both (RQ3, Figure 5, Table 2).
Strengths
SE-grounded metric design
Every SNC indicator traces to empirical SE precedent: Hassan (2009) change-complexity entropy, Nashid et al. (2025) patch proximity (radius), McCabe (1976) complexity inside 'mass', and 180-day churn windows per Shrikanth 2021/Zimmermann 2007 — far more principled than the aggregate length statistics the paper criticizes (Section 4.3–4.5).
Scale and breadth
2,487 instances across five benchmarks, 30 runs and 14,922 trajectories spanning two model families at three scales, all under one harness (Harbor); the FEA-Bench-to-Harbor conversion is itself a reusable community artifact (Section 3).
Statistical candor
Curation fingerprints are mechanistically actionable
Weaknesses
Harness-model confound in RQ3
Single run per configuration
Ad hoc axis reduction
Gold patch as demand proxy is partially circular
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a real gap: the vocabulary for comparing agentic software engineering (SE) benchmarks is nominal category labels ('bug fix', 'feature implementation') that conceal very different curation pipelines. It proposes the Spread–Novelty–Centrality (SNC) profile — Spread (normalized patch entropy + directory-tree radius), Novelty (|add|/(|add|+|del|)), Centrality (import fan-in/out, 180-day churn, size×complexity 'mass') — computed over gold patches, and applies it to five benchmarks (2,487 instances: SWE-bench Verified 500, SWE-Gym Lite 230, FEA-Bench 1,401, FeatBench 156, FeatureBench 200) plus 30 runs and 14,922 trajectories from six agent configurations (Claude Haiku 4.5/Sonnet 4.6/Opus 4.6 under Claude Code; Qwen 3.5 9B/27B/397B under Qwen Code) on the Harbor harness. Three claims: (1) labels are unreliable — every benchmark pair separates on at least two of three SNC axes (RQ1, Figure 3); (2) agent behaviour reveals demands the gold patch cannot, with verbosity tracking problem-statement informativeness (RQ2); (3) resolved runs concentrate in low-SNC regions uniformly, but success behaviours are family-specific (RQ3). Claims are scoped honestly and matched by the evidence presented.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SNC profile introduced in this paper?
A three-axis characterization of repository-level coding tasks, computed from each task's gold (reference) patch: Spread (how widely the change distributes across the codebase, via patch entropy and directory-tree radius), Novelty (the fraction of added versus deleted lines), and Centrality (import-graph fan-in/fan-out, recent 180-day churn, and 'mass' combining function size with cyclomatic complexity). It aims to describe the engineering work a benchmark task demands, beyond coarse labels like 'bug fix'.
Which benchmarks and models were studied?
Five agentic software engineering benchmarks totalling 2,487 instances: SWE-bench Verified (500), SWE-Gym Lite (230), FEA-Bench (1,401), FeatBench (156), and FeatureBench (200). Two model families at three scales each ran under the Harbor harness: Claude Haiku 4.5, Sonnet 4.6 and Opus 4.6 (via Claude Code) and Qwen 3.5 9B, 27B and 397B-A17B (via Qwen Code), producing 30 runs and 14,922 agent trajectories.
What is the main Claude vs Qwen finding?
Both families resolve more often on low-SNC tasks, but their success styles differ. Claude succeeds by producing patches at roughly the gold patch's scope — its parity share on files among resolved runs rises from 0.17 at the smallest scale to 0.41 at medium and 0.54 at the largest — while Qwen resolves by over-producing, with 0.6–0.7 of resolved runs in the >2× band on files at every scale. Editing too little predicts failure for both families.
What are the paper's main limitations?
All five benchmarks are Python-only and several indicators need language-aware tooling; only issue-resolution and feature-implementation task types are covered; the family contrast pairs different harnesses (Claude Code vs Qwen Code), entangling model and harness effects; and there is a single run per configuration, so no variance estimates are reported for the behavioural trends.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–9 are extracted as scattered axis labels and numbers; per-bin shares cited in this review come only from the prose, abstract and Table 2 (e.g., 0.17/0.41/0.54, 0.6–0.7, ≈0.8), not from reading plot pixels, and the Figure 6 radar plot values are not numerically recoverable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.