If It's Not Buggy, Don't Fix It: On the Dynamics of Iterative Bug-fixing with LLMs
Xietao Wang-Lin, Anton Isopoussu, Louis Mahon
cs.SE · cs.CL
Quick verdict
Finds iterative LLM bug-fixing damages correct code more than it repairs buggy code, with a causal bugginess steering vector; narrow models and domain.
Summary
This report studies what happens when an LLM is repeatedly prompted to find and fix bugs in the same single file without access to its own history, a strictly Markovian loop. Using 800 C++ submissions from CodeContests+ (20 problems x 40, with ground-truth test verdicts), the authors evaluate Gemini 2.5 Flash-Lite and Qwen2.5-7B-Instruct under whole-file edits versus search/replace blocks (SRBs), at greedy decoding and tau=0.7, for up to 100 turns. Damage to correct code meets or exceeds repair of buggy code in every configuration (e.g., Qwen SRB tau=0: beta=0.424 vs alpha=0.041); SRBs induce more cycles, longer transients, and failure modes such as identity edits and phantom search blocks, and loops often settle into pseudo-bug-fixing cycles where edits are undone and reapplied indefinitely. Mechanistically, a difference-of-means probe built from the model's confident Y/N verdicts reaches validation AUC near 1.0 in deep layers; logit-lens cosine checks and unseen-task steering suggest the direction is not merely the verdict readout, and negative steering halts the loop while positive steering increases edits and phantom failures. Limitations: two small/fast-tier models only, single-file competitive code (median 38-42 LOC), probes trained on the model's own verdicts rather than ground truth, SEM-only statistics, and no stated code release.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across two models and two edit formats, iterative history-blind bug-fixing damages correct programs at a rate at least as high as it repairs buggy ones, with the gap largest for search/replace blocks (Table 2).
- The repair loop frequently converges to pseudo-bug-fixing cycles where the same changes are added and removed indefinitely, far more often with atomic SRB edits than with whole-file edits (Figs. 4-6).
- A difference-of-means steering vector on Qwen2.5-7B-Instruct activations encodes 'bugginess' with validation AUC near 1.0 at deep layers, is causally upstream of editing propensity (gamma steering), and is not merely the Y/N verdict readout (logit-lens cosine and unseen-task controls, Sec. 5).
Strengths
Clean dynamical-systems formalization with a proper failure-mode taxonomy
Sec. 3 formalizes the no-history loop as a strictly Markovian system (Eq. 1-2, identity application for invalid SRBs); Sec. 4.2 gives an explicit attractor taxonomy (None, identity, phantom, degeneration, cycle, failed) with frequencies, transient lengths and cycle-length distributions (Figs. 4-6), and Appendix E ablates the reasoning trace, showing cycling is not merely a reasoning artifact.
Quantified damage-vs-repair asymmetry across a factorial grid
Table 2 reports repair (alpha) and damage (beta) rates with SEMs across 2 models x 2 edit formats x 2 temperatures x 2 starting states; the SRB asymmetry is stark (Qwen tau=0: beta=0.424+/-0.015 vs alpha=0.041+/-0.004; Gemini: beta=0.293 vs alpha=0.062), and the whole-file case is honestly characterized as 'similar'.
Mechanistic claims come with the right controls
Unusually transparent experimental protocol
Weaknesses
Narrow external validity
Probe classes are model beliefs, not ground truth
SRB vs whole-file comparison is confounded by the one-bug-per-turn instruction
Thin statistical treatment
Abstract overstates its own Table 2
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.10123, cs.SE/cs.CL) studies iterative, history-blind bug-fixing by LLMs as a discrete dynamical system, C(n+1) = f(C(n)) (Sec. 3, Eq. 1-2). Three headline claims: (1) LLMs claim to find bugs in bug-free code and damage correct programs at a rate at least as high as they repair buggy ones; (2) the loop frequently converges to pseudo-bug-fixing cycles, especially with atomic search/replace block (SRB) edits; (3) a linear 'bugginess' steering vector in Qwen2.5-7B-Instruct causally controls editing propensity, suggesting an internal representation of buggy code. Within the studied scope, all three find support; claim (3)'s interpretation is the weakest link because the probe's classes come from the model's own verdicts, not ground truth.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs break working code when asked to fix bugs?
In this study, yes: across two models (Gemini 2.5 Flash-Lite, Qwen2.5-7B-Instruct) and two edit formats, the rate at which iterative LLM bug-fixing turned correct programs incorrect was at least as high as the rate at which it fixed buggy ones. For example, with Qwen2.5-7B using search/replace blocks at temperature 0, correct code was damaged on 42.4% of transitions while buggy code was repaired on only 4.1%.
What is a pseudo-bug-fixing cycle?
It is a loop state where the model keeps claiming to find bugs and applies changes that are later reverted, cycling between the same code states indefinitely instead of terminating. The paper shows these cycles are far more frequent with atomic search/replace edits than with whole-file rewrites, and some cycles even alternate between passing and failing versions of the code.
What is the steering vector in this paper?
It is a direction in the model's activation space obtained by averaging activations of code the model confidently labels as buggy versus bug-free. Adding it to activations (positive gamma) makes the model edit code more aggressively; subtracting it (negative gamma) stops the editing loop entirely, suggesting an internal 'bugginess' signal controls the model's propensity to make changes.
Which models and data were used?
Gemini 2.5 Flash-Lite (via API) and Qwen2.5-7B-Instruct (run locally on one 24GB GPU), tested on C++ competitive-programming submissions from the public CodeContests+ dataset (20 problems, 40 submissions each, median 38-42 lines of code). The mechanistic probing and steering experiments were done only on the Qwen model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as caption text with axis labels in the extraction; plotted values (e.g., Fig. 4 attractor frequencies, Fig. 7 AUC curves and cosine values) could not be inspected beyond their in-text descriptions.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.