Large Language Models for Programming: Actually Fixing or Reimplementing Incorrect Code?
Alexandru Stefan Stoica, Traian Rebedea, Marian Cristian Mihaescu
cs.CL · cs.SE
Quick verdict
Finds LLMs rewrite buggy code instead of minimally fixing it and solve more from scratch; plausible, but 567 pairs from 7 users, no significance tests.
Summary
The paper asks whether LLMs minimally repair buggy competitive-programming submissions or quietly replace them. The authors collect ~3,000 Codeforces submissions from 7 users, pair each buggy submission with the same user's later Accepted fix, filter to 567 pairs with buggy-to-human-fix similarity of at least 60%, and evaluate gpt-5-nano, gpt-5-mini and gpt-5.1 (plus a no-reasoning variant) under three prompts: NaiveBugFix, GenerateScratch, and a GEPA-optimized bug-fix prompt. Correctness is decided by running generated C++ against public tests plus DeepSeek-R1-generated tests from Codeforces-R1; structural preservation is a Jaccard line-similarity ratio (generated fix vs human fix, both relative to the buggy code). Table I shows scratch generation solves more problems than bug fixing for most model/bin combinations (~0.15 average gap for nano), and reasoning effort strongly drives pass rates. GEPA raises simratio (0.93 vs 0.83 on the [0.9-1] bin for gpt-5.1) but lowers solve rates, suggesting a structure-versus-correctness trade-off. Limitations: only 567 pairs from 7 BSc-era students, an arbitrary 60% threshold, no significance testing or confidence intervals despite bins of 60-115 samples, no non-OpenAI models, and a test oracle partly generated by DeepSeek-R1 that may mislabel verdicts; contamination is acknowledged but uncontrolled.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 4
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-generated fixes modify more lines than the human author's own fix, with simratio below 1 for most model/similarity-range combinations (Table I).
- LLMs solve more problems when generating from scratch than when patching buggy submissions, with a ~0.15 average gap for gpt-5-nano (Section V, RQ2).
- A GEPA-optimized prompt that pushes minimal edits raises simratio (e.g., 0.93 vs 0.83 for gpt-5.1 on [0.9-1]) but lowers the solve rate, implying a structure-versus-correctness trade-off (Section V, RQ3).
- Reasoning effort dominates correctness: gpt-5.1 without reasoning solves far fewer problems (0.36-0.77) than gpt-5.1 with medium reasoning (0.77-0.97).
Strengths
Naturalistic paired data instead of implanted bugs
Section III-A builds pairs from real Codeforces submission histories (7 users, 2015-2025, ~3,000 submissions -> 567 pairs), so buggy/fix pairs reflect what humans actually changed, unlike DebugBench-style bug implantation ([9]).
Execution-based correctness with a reasoned similarity metric
Pass/fail is decided by running code against public plus DeepSeek-R1-generated tests (Section III-B.2), and the simratio metric extends Dai et al.'s [14] consistency rate by also penalizing deleted lines (Section III-B.1) - a clear, formally defined refinement.
Contamination-aware, leakage-aware design
Transparent, cheap-to-replicate protocol
Weaknesses
No inferential statistics anywhere
Small, demographically narrow dataset
Unvalidated LLM-generated test oracle
Metric circularity in the GEPA arm
Single-vendor, single-call scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This empirical study (cs.CL/cs.SE, arXiv:2609.29410, Sep 2026) asks a behavioral question rather than a leaderboard question: when GPT-5-family models repair a buggy competitive-programming submission, do they emit a targeted patch analogous to the human author's eventual fix, or do they quietly reimplement? Three RQs cover structural preservation (RQ1), scratch-vs-fix solve rates (RQ2), and the relationship between them (RQ3). The stated claims: generated fixes touch more lines than human patches; scratch generation solves more problems; and forcing minimal edits via prompt optimization trades correctness for similarity. The framing (Section I) is practically motivated - AI coding assistants that replace rather than repair undermine debugging workflows.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs actually fix bugs or do they rewrite your code?
This paper finds that on 567 real buggy Codeforces submissions, GPT-5-family models typically change more lines than the human author did to fix the same bug, and sometimes produce essentially new solutions. Prompting them to minimize edits increases structural similarity but lowers the chance the fix is actually correct.
Are LLMs better at writing code from scratch than at fixing bugs?
In this study, mostly yes: scratch generation passed more tests than the bug-fix prompts for most models and similarity ranges (an average gap of about 0.15 for gpt-5-nano). Models with reasoning enabled were much stronger overall, which narrowed the picture but did not reverse the finding.
What data does the paper use?
About 3,000 Codeforces submissions from 7 users (2015-2025). Each buggy submission is paired with the same user's later accepted fix, filtered to 567 pairs with at least 60% similarity, and correctness is checked against public plus DeepSeek-R1-generated tests from the Codeforces-R1 dataset.
Is the code and dataset available?
Yes. The authors state that the code, the GEPA prompt model, and the dataset are on GitHub, all experiments ran in Docker, and the whole study cost roughly $80 for ~20 million tokens, making replication comparatively cheap apart from OpenAI API access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 4 (anchor-construction diagram, GEPA flow) are not analyzable from the extracted text.
- The abstract tail is clipped mid-sentence in the extraction; minor omissions elsewhere are possible.
- Table I arrived flattened; cell-to-method/model mapping was reconstructed from the header text, so individual cell attributions carry small reconstruction risk.
- Per-bin sample sizes come from the Samples column; exact per-model denominators are not stated in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.