Passes Alone, Fails Together: Benchmarking Semantic Coordination in Parallel LLM-Agent Development
Haocheng Xia, Eugene Wu, Yongjoo Park, Yongjoo Park
cs.CL · cs.AI · cs.SE
Quick verdict
STALE benchmark: mined PR pairs almost never interfere after grading fix; scripted Django edits fail 97%; honest, rigorous, but narrow and idealized.
Summary
The paper introduces STALE, a benchmark for measuring semantic coordination failures in parallel LLM coding agents: patches that each pass their own tests individually but fail when merged. The metric Δ_blind counts only tests that pass with every patch alone and fail solely on the combined code. Three tiers are evaluated: synthetic tasks where a scripted patch changes d of 4 planted helper interfaces; 417 validated pairs of merged Django pull requests (834 runs); and 36 constructed tasks applying rename, signature, or return-type changes to 12 real Django helpers. After correcting a grading flaw (running the same combined test suite in every condition and discarding agent test-file edits), only 1 of 417 mined pairs showed interference. In the synthetic tier, blind interference rose linearly with staleness degree, and a roughly 130-token oracle message describing the finalized change cut mean interference from 2.50 to 0.04 (98%). On constructed tasks, blind GPT-5.5 runs interfered in 105/108 cases (97%), and the oracle message recovered 82% (93% among cleanly merging patches). Main limitations: reviewed PR history rarely supplies positive interference examples; communication is tested only with complete oracle descriptions of finished changes; trial counts are small and mostly one model; and the authors themselves stress the constructed rates do not estimate real-world prevalence.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Clean textual merges of independently correct LLM-agent patches can still fail behaviorally, and the Δ_blind metric with a single combined test suite isolates merge-induced failures from solo-run or grading artifacts.
- After correcting the grading procedure, only 1 of 417 validated mined Django PR pairs (834 runs) shows interference, indicating reviewed PR history rarely retains unresolved parallel changes.
- On 36 constructed tasks over 12 real Django helpers, blind GPT-5.5 runs produce interference in 105/108 runs (97%), and a complete oracle message describing the finished change recovers 82% (93% among cleanly merging patches).
- A roughly 130-token oracle message reduces mean synthetic interference from 2.50 to 0.04 (98% reduction), quantifying the information headroom of coordination.
- The constructed failure rates do not estimate how often such interference occurs in everyday parallel development (the authors' own explicit limitation).
Strengths
Corrected grading protocol is a transferable methodological contribution
Fig. 4 demonstrates how naive asymmetric testing (solo patches graded only on their own task's tests, merged patches on both) fabricates interference. The fix — one combined test suite in every condition, agent test-file edits discarded, and each patch required to pass its own gold tests before a pair counts as observable — should be adopted by any multi-agent merge benchmark.
Deterministic, fully documented mining pipeline
§3 specifies shared merge base, clean-merge filter, and fail-to-pass base validity, with complete rejection accounting: 4,262 candidate Django pairs → 417 valid (~36% textual conflicts, ~33% base drift, ~17% non-applying, ~14% accepted); the same procedure yields 11 SymPy, 12 xarray, 7 seaborn pairs.
Unusually honest negative result and scoping
Three-tier design separates mechanism from history and from real code
Reproducibility infrastructure
Weaknesses
Mined tier is a null result that limits the benchmark's discriminative value
The 97% constructed interference rate is close to tautological
Descriptive statistics only, with small per-cell samples
Oracle-only communication overstates recoverability
Model and run provenance gaps hinder exact reproduction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This 6-page EXPRESS '26 workshop paper introduces STALE (code: github.com/illinoisdata/STALE-bench), a benchmark for semantic coordination failures: patches produced by parallel LLM coding agents that each pass their own tests (solo) but fail when textually merged (blind). The core artifact is the Δ_blind metric — tests passing with every patch alone but failing only on the merged code — plus the indicator I_blind, evaluated over three tiers: synthetic (scripted changes to d of 4 planted Django helpers; 3 mechanisms × 4 degrees × 8 trials = 96 runs), mined (417 validated pairs of merged Django PRs from a shared base commit; 834 graded runs, plus 30 unrun pairs in SymPy/xarray/seaborn and 23 disjoint controls), and real-derived (36 constructed tasks on 12 real Django helpers; 108 blind + 108 communication runs). The central claims are: (i) a corrected grading protocol is necessary because naive asymmetric testing fabricates interference (Fig. 4); (ii) reviewed PR history contains almost no unresolved interference (1/417 pairs); (iii) scripted breaking changes to real helpers cause interference in 105/108 blind runs (97%), with an oracle message recovering 82%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the STALE benchmark?
STALE is a benchmark that measures whether software patches produced in parallel by LLM coding agents still work when merged. It runs the same hidden tests on each patch alone and on the combination, counting only failures that appear solely after merging (the Δ_blind metric). It ships three task tiers: synthetic interface-change tasks, 417 validated pairs of real merged Django pull requests, and 36 constructed tasks on 12 real Django helper functions. Code is public at github.com/illinoisdata/STALE-bench.
Can two AI-generated code patches conflict even when git merges them cleanly?
Yes. The paper's motivating example: one patch adds a price cache and makes every update invalidate it, while another adds a bulk-update path writing straight to the database. The merge has no textual conflict, but reads return stale prices. The authors call this a semantic coordination failure and detect it with tests that pass on each patch individually but fail on the merged code.
How often do parallel-agent merge failures happen in real projects?
In this paper, rarely in reviewed history: among 417 validated pairs of merged Django pull requests (834 agent runs), only one pair showed merge-induced test failures after the grading procedure was corrected. Because human developers and reviewers usually make patches compatible before merging, the paper states its constructed-task failure rate of 97% cannot be used to estimate real-world frequency; measuring pre-review parallel agent work remains open.
Can telling an agent about concurrent changes fix these failures?
Partially, in the idealized setting tested. A roughly 130-token message completely describing the already-completed concurrent change cut mean synthetic interference from 2.50 failing tests to 0.04 (98% reduction) and recovered 82% of constructed Django runs (93% among patches that merged cleanly). This is an oracle upper bound; messaging during ongoing, incomplete parallel work was not evaluated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-4 are known only through captions and surrounding text; exact plotted values (e.g., Fig. 3 curves) could not be inspected directly.
- The second model family referenced in Table 1 is never named in the text, and per-run provenance (seeds, per-harness splits) is not described, so run-level verification is not possible from the paper alone.
- The extracted text appears complete (references and Appendices A-C present), supporting high confidence; agent-side numbers are model-dependent and would vary under reproduction with different models.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.