When Models Edit Too Much: On the Fidelity of Minimal Code Edits
Tongyao Zhu, Wei Hern Lim, Min-Yen Kan
cs.SE · cs.AI · cs.CL
Quick verdict
Controlled over-editing benchmark for LLM code repair; metric validation is strong, but single-token synthetic bugs limit real-world generalization.
Summary
The paper studies over-editing: LLM code repairs that pass tests but rewrite far more code than the bug requires. The authors build a controlled benchmark from 400 BigCodeBench problems by injecting one or two AST-level corruptions into reference solutions, so the minimal repair (the corruption's reversal) is known by construction. Repairs are scored on Pass@1, excess token-level Levenshtein distance relative to the gold patch, and added cognitive complexity; human validation (three annotators, 100 blinded pairs, κ≈0.9 for edit distance) supports the metrics. Over 20 frontier models across 50 matched settings show widespread over-editing: GPT-5.5 High reaches Pass@1 0.823 with excess distance 0.299 versus 0.074 for Claude Opus 4.7. A single preservation instruction cuts aggregate excess distance from 0.195 to 0.131, reduces added complexity by 26.6%, and raises Pass@1 by 2.3 points. Neither reasoning budgets nor model scale reliably reduce over-editing. Post-training Qwen3-4B, supervised fine-tuning overfits to seen corruption types (out-of-domain Pass@1 drops to 0.458), whereas GRPO-style RL with an execution-plus-edit reward achieves 0.782 out-of-domain Pass@1 with 0.050 excess distance while keeping general coding ability; the preference transfers to real Defects4J Java bugs with unchanged pass rates and smaller edits. Limitations include single-token synthetic bugs, LLM-based taxonomy annotation, and small human studies.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier LLMs over-edit by default: e.g., GPT-5.5 High passes 0.823 of repairs while its excess Levenshtein distance (0.2986) is over four times Claude Opus 4.7's (0.0745) under the generic prompt (Table 17).
- A single preservation clause reduces aggregate excess distance from 0.195 to 0.131, cuts added cognitive complexity by 26.6%, and raises Pass@1 by 2.26 points (95% CI [+1.49,+3.05]) across 50 matched settings.
- Reasoning effort and parameter scale (Qwen2.5-Coder 0.5B-32B, Figure 5) do not monotonically or reliably reduce over-editing among passing repairs.
- SFT overfits to seen corruption families (OOD Pass@1 0.458, LiveCodeBench −14.9), while GRPO-style RL with an execution-plus-edit reward reaches OOD Pass@1 0.782 with 0.050 excess distance and LCB +0.6.
- The learned minimal-edit preference transfers to single-method Defects4J Java bugs with unchanged pass rates and smaller edits (token edits 51.9→35.3 for 4B).
Strengths
Ground-truth minimal patches by construction
Injecting one or two AST corruptions from 14 predefined families (568 applications, Tables 9-10) into 400 BigCodeBench reference solutions, with retention only when the corrupted program fails the original tests, makes the minimal patch exactly known - a cleaner control than inferred real-bug gold patches.
Unusually thorough metric validation
Blinded human study (3 annotators, 100 pairs): excess Levenshtein agrees with the human majority at 94.8% (κ=0.897) for reviewability and 96.9% (κ=0.939) for faithfulness; a blinded audit of 100 high-excess repairs finds 82.3% genuinely unnecessary (CI 73.5-88.6%); Appendix B.1 shows CodeBLEU misranks minimal-vs-rewrite pairs, and D.3 confirms rankings with line-diff and syntax-tree-diff (Spearman ρ≈0.91).
Statistical robustness beyond the norm
Transparent failure reporting
Ablations defuse obvious confounds
Weaknesses
Construct validity: size-based fidelity penalizes valid alternatives
Taxonomy rests on LLM self-annotation
Thin real-bug external validity
Single-run training comparisons
Single-sample frontier decoding
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines **over-editing** - a functionally correct repair that changes more code than the minimal fix requires - and builds a framework to measure it. From 400 BigCodeBench problems, the authors inject one or two AST-level corruptions from 14 predefined families (568 applications; Tables 9-10), retaining a task only when the corrupted program fails the original tests. Because the bug is injected, the minimal patch is known by construction: it is the corruption's reversal. Gold patches are tiny (50.2% single-token; none touches more than two lines; Table 8). Four claims are advanced: (1) over-editing is widespread among frontier models; (2) a single preservation instruction reduces it; (3) reasoning budgets and model scale do not reliably help; (4) minimal editing can be learned, with RL beating SFT out of domain.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is over-editing in LLM code repair?
Over-editing is when a model fixes a bug correctly but rewrites far more code than the minimal fix requires - adding validation, restructuring data flow, or renaming variables - so the patch passes tests yet is larger and harder to review than necessary. This paper measures it by injecting known bugs into reference solutions and comparing each model's patch size against the known minimal repair.
Does telling an LLM to keep edits minimal actually help?
Yes. Adding one sentence asking the model to preserve the original code reduced average excess edit distance across frontier models from 0.195 to 0.131, cut added cognitive complexity by 26.6%, and raised Pass@1 by about 2.3 percentage points (95% CI [+1.49,+3.05]) - so the minimality instruction improved rather than traded off correctness.
Can minimal-edit behavior be trained into a model?
The authors find reinforcement learning works best: GRPO-style RL on Qwen3-4B with a reward combining test execution and edit distance reached 0.782 out-of-domain Pass@1 with only 0.050 excess edit distance while keeping LiveCodeBench performance, whereas supervised fine-tuning overfit to the corruption types it was trained on and lost general coding ability.
Is the benchmark and code for this paper public?
Yes. The paper links code at github.com/nreHieW/over-editing, and the framework is built entirely from public artifacts: BigCodeBench tasks, DeepCoder training samples, LiveCodeBench v6, and Defects4J. Compute needs are modest by training standards (an RL run of 100 steps for the 4B model takes about 3 hours on 8 A100 GPUs).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Two-column PDF extraction interleaves axis labels in Figures 2 and 4 and scrambles row ordering in parts of Table 17; all cited numbers were cross-checked against the table cells and remained legible.
- The extraction ends naturally at Appendix E.5 (AI Assistant Use) with no mid-sentence cutoff, so no truncation is suspected; figure images themselves were not available, only their embedded text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.