Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair
cs.SE · cs.AI · cs.LG
Quick verdict
Rigorous, honest negative result: localized LLM repair loses to blind resampling and beats a random-span placebo only suggestively at matched attempts.
Summary
This controlled study asks whether test-derived fault localization makes an LLM repair attempt better than simply resampling a fresh solution. Across Qwen2.5-Coder-32B, Qwen3.6-27B, Gemma-4-26B and (as a declared replication) Mistral-Small-24B, over HumanEval+, MBPP+ and a LiveCodeBench subset, the author applies three arms to the same failed candidate: blind resampling, Ochiai-spectrum-localized infilling, and a same-length random-span placebo. Three findings emerge. Localization is rarely available: only 9.0% of 488 failing candidates expose a failing public test with a usable spectrum (36.3% with an augmented suite no deployment would have). Where localizable, localized infilling loses decisively to blind resampling at a matched 16 attempts (3:40 unlock discordance, p=3.0×10⁻⁹; 4.4% vs 10.1% per attempt), replicating in Mistral at −11.3 points. Against the placebo, localized infilling leads only in pooled analysis (11:1, Holm p=.019), resolving in no individual model under the designated primary analysis, so the location effect stays suggestive. Wider edits, adaptive policies and deduplication do not rescue it; verbatim span regeneration (48.9%) and low edit diversity explain the saturation. Limitations: small per-model samples (34–91), every resolving result depending on the non-deployable strong signal, unverified LiveCodeBench contamination cutoffs, and exact-equality scoring that breaks comparability with published EvalPlus numbers.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Only 9.0% of 488 failing candidates from three 26–32B code models can be localized from public tests; the binding constraint on test-guided repair is verifier strength, not localizer quality (§4.1).
- On the 177 strong-signal-localizable candidates, localized infilling loses decisively to blind resampling at a matched 16 attempts (3:40 unlock discordance, p=3.0×10⁻⁹; 4.4% vs 10.1% per attempt), and the loss replicates in a third model family (Mistral-Small-24B, −11.3 points, 0:25 discordance).
- Localized infilling beats a same-length random-span placebo only in pooled analysis (11:1, Holm p=.019, +3.2 points) and in no individual model under the designated primary analysis, so the location effect is suggestive only.
- The loss is mechanistic, not an artifact of degenerate edits: infilling regenerates the removed span verbatim in 48.9% of attempts, yields 0.23 distinct programs per attempt vs 0.83 for resampling, and wider or adaptive edits do not rescue it (§4.4, Tables 2–3).
- Re-priced as tokens, localization leads only below 372 generated tokens — the cost of one fresh attempt — so it is viable solely in a latency- or token-bound deployment regime (§4.3).
Strengths
Two-control design that genuinely separates confounds
The same-length, code-only random-span placebo (§3.2) isolates 'the location was right' from 'the edit was small', while the matched K=16 attempt budget isolates editing value from retry value — the two explanations prior work (Fan et al. 2023; Xu et al. 2024) left conflated.
Pre-specification honestly reported as failed
Appendix E reproduces the pre-specified decision rule (5-point placebo margin, beat resampling, two-family replication), states all three win clauses failed, and distinguishes 'pre-specified' from 'preregistered'; both analysis plans (kickoff unlock rule vs Wave 1B per-attempt primary) are reported with their roles labeled.
Falsification attempts instead of defense
Mechanistic account with measurable cause
Reusable methodology artifacts
Weaknesses
Every resolving result depends on a signal no deployment has
Underpowered and heterogeneous per-model cells
Confirmatory status of the placebo effect rests on a pooled analysis the authors' own plans reject
Population and scoring choices limit external comparability
Token-budget analysis is post-hoc normalization, not a controlled comparison
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed evaluation question: when a frozen code model's candidate fails tests, does repairing a test-implicated span (Ochiai SBFL + fill-in-the-middle) outperform (a) an equally sized edit at a random location and (b) simply resampling a fresh whole solution at the same attempt budget? The population is 488 dead candidates from one greedy generation per task across Qwen2.5-Coder-32B-AWQ, Qwen3.6-27B and Gemma-4-26B-A4B on HumanEval+ v0.1.10, MBPP+ v0.2.0 and a LiveCodeBench function-call subset, plus Mistral-Small-24B as a declared fourth replication. The claims are evaluative, not algorithmic: (1) localizability is scarce (9.0% public / 36.3% strong, §4.1); (2) localized infilling loses to matched-budget blind resampling decisively (§4.2); (3) the location effect over the random-span placebo is only suggestive (pooled 11:1, Holm p=.019; no per-model resolution, best Holm p=.087); (4) the loss is mechanistic (48.9% verbatim regeneration, 0.23 vs 0.83 distinct programs per attempt, §4.4/Table 3); (5) localization leads only below 372 generated tokens (§4.3). All five are stated within the bounds the data support.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does fault localization help LLMs fix code better than just trying again?
According to this study, usually no. On 177 localizable failing solutions from 26–32B code models, repairing only the test-implicated span at the same 16-attempt budget succeeded on 4.4% of attempts versus 10.1% for blindly resampling a fresh whole solution (3:40 unlock discordance, p=3.0×10⁻⁹). Localized repair led only where the budget cannot afford a single fresh attempt (below ~372 generated tokens), and only suggestively beat a random-span placebo.
How often can spectrum-based fault localization even be applied to LLM code failures?
Rarely. Only 9.0% of 488 failing candidates exposed a failing public test with a usable coverage spectrum, mostly because the largest group of failures (212 of 488) passed every public test and failed only hidden ones. Using an augmented test suite raised this to 36.3%, but no deployed repair loop has access to that suite.
Why does localized infilling perform so badly in this study?
The authors show the mechanism: conditioned on a fixed prefix and suffix, the models regenerate the removed span verbatim in 48.9% of attempts and produce only 0.23 distinct programs per attempt versus 0.83 for fresh resampling. Wider edits, adaptive width policies and deduplication narrowed but never reversed the loss.
Is the paper's code and data available?
The paper states a workshop supplement ships the harness source, unit tests, aggregate results, spans and a strict public analysis record; every reported number is regenerated from result files by checked scripts. Model weights and benchmarks are not redistributed, and random seeds and wall-clock durations were not preserved.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's five numeric labels (44, 212, 149, 177, 194) cannot be confidently assigned to the five funnel categories from the extracted layout (they exceed 488 in sum); I relied only on prose-stated counts (44 public-localizable, 212 public-invisible, 177 strong-localizable).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.