Better Understanding, Better Fixes? A Study of Hallucination in LLM-based Automated Program Repair
Xuemeng Cai, Jiakun Liu, Linhan Yang, Wei Ma, Lingxiao Jiang
cs.SE · cs.AI
Quick verdict
Rigorous empirical study of repair vs understanding hallucination in LLM-APR on 832 Defects4J bugs; but 'hallucination' partly repackages known overfitting.
Summary
This empirical study investigates hallucination in LLM-based automated program repair (APR) at two levels: repair hallucination in final patches and understanding hallucination in intermediate artifacts. Understanding is made observable through three tasks — triggering testcase identification, line coverage prediction, and additional testcase generation — each scored against execution-grounded oracles. The authors evaluate GPT-5, DeepSeek-R1, and Claude Sonnet 4.5 on 832 Defects4J bugs. Automatically, only 21.0%–55.9% of patches pass developer-written test suites; exact triggering-testcase identification reaches only 23.0%–40.9%; and 30.5%–48.4% of generated additional testcases are valid. Stratified manual analysis of 812 sampled repairs (open coding; Cohen's kappa 0.82 for repair, 0.49 for understanding) finds repair hallucinations in 72.7% of cases, dominated by incorrect causal localization (45.9%) and incorrect repair strategy (18.5%); 39 of 44 manually inspected low-scoring coverage predictions involve branching control flow. More accurate intermediate artifacts are generally associated with — but do not guarantee — successful repairs, and the paper names insufficient project-context grounding, imprecise causal localization, and unstable execution-path reasoning as contributing factors. Main limitations: the broad hallucination definition subsumes conventional patch overfitting, inflating headline rates; understanding-annotation agreement is only moderate; results come from a single Java benchmark; each task instance was sampled once; and the understanding–repair link remains correlational.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Only 21.0%-55.9% of patches generated by GPT-5, DeepSeek-R1, and Claude Sonnet 4.5 pass the developer-written test suite across baseline and three task settings on 832 Defects4J bugs (Table 2/Fig 8).
- Manual analysis of 812 stratified-sampled repairs finds repair hallucinations in 72.7% (590 cases), with incorrect causal localization (45.9%) and incorrect repair strategy (18.5%) dominant (Table 5).
- Understanding hallucinations are prevalent: exact trigger identification only 23.0%-40.9% (Table 6), branch structures in 39/44 low-scoring coverage predictions (Table 8), and only 30.5%-48.4% of generated additional testcases valid (Fig 11/Table 9).
- More accurate intermediate artifacts are generally associated with plausible patches, but the relationship is not deterministic in either direction (Tables 3, 4, Fig 11).
- Three empirically grounded contributing factors explain much of the observed hallucination: insufficient project-context grounding, imprecise causal localization, and unstable execution-path reasoning (Table 10/Sec 5.3).
Strengths
Execution-grounded three-task framework
Triggering testcase identification (Sec 3.3), line coverage prediction on both buggy and model-patched programs (Sec 3.4), and additional testcase generation (Sec 3.5) turn latent understanding into verifiable artifacts; the three-version validity schema (buggy/developer-fixed/model-patched) avoids circularly validating testcases against the model's own patch (Sec 3.5b).
Scale and model recency
832 filtered Defects4J bugs, three frontier models (GPT-5, DeepSeek-R1, Claude Sonnet 4.5, experiments started Nov 2025), baseline plus three task settings, with 812 manually annotated repairs and set-relation analysis over all 832 bugs per model (Tables 6, 7).
Disciplined qualitative pipeline
Evidence-backed taxonomy with worked examples
Honest causal discipline
Weaknesses
Definitional breadth inflates the headline rate
Moderate agreement on the central new construct
Single response per task instance
Input restriction may confound the grounding finding
Uneven manual sample sizes and no inferential tests
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This Empirical Software Engineering-style manuscript (arXiv 2609.04909) asks whether hallucination in LLM-based automated program repair (APR) extends beyond final patches. The authors define **repair hallucination** (patches that fail to compile, fail the developer test suite, or pass while overfitting/deviating from developer intent) and **understanding hallucination** (intermediate artifacts inconsistent with execution-grounded evidence). Three RQs structure the study: prevalence (RQ1), types and distributions (RQ2), contributing factors (RQ3). Central results: only 21.0%-55.9% of patches pass developer tests across 3 models and 4 settings on 832 Defects4J bugs; manual analysis of 812 sampled repairs finds hallucination in 72.7% (590 cases), led by incorrect causal localization (45.9%) and incorrect repair strategy (18.5%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'understanding hallucination' in LLM-based program repair?
It is this paper's term for model errors in intermediate repair artifacts rather than the final patch: misidentifying which tests trigger the bug, wrongly predicting which code lines execute under those tests, or generating invalid additional testcases — outputs inconsistent with execution-grounded evidence such as real test outcomes and instrumented coverage.
How often do LLMs hallucinate when automatically fixing bugs?
In this study on 832 Defects4J Java bugs with GPT-5, DeepSeek-R1, and Claude Sonnet 4.5, only 21.0%-55.9% of generated patches passed the developer-written tests, and manual analysis judged 72.7% of 812 sampled repairs to contain repair hallucinations — most often wrong causal localization (45.9%) or a wrong repair strategy (18.5%).
Which model repaired bugs best in this study?
GPT-5 performed best overall: the highest plausible-patch rate (55.9% in the line coverage setting), the best triggering-testcase identification (F1 0.607 for passing repairs versus 0.482 for Claude and 0.445 for DeepSeek), and the highest valid-testcase rate (48.4%).
Does better bug understanding lead to better patches?
The study finds a general association, not a guarantee: repairs that pass tests tend to have more accurate intermediate artifacts, but some passing patches coexist with understanding errors and some accurate artifacts accompany failed repairs. The relationship is explicitly correlational, not causal.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with three marked '[...sampling gap...]' regions (part of Section 3.1, Section 4.3 implementation/sampling details including Table 1, and the end of Section 5.2d into Section 5.3/Table 10 examples); no absence claims were made about those regions.
- Per-bug distribution shapes and CI bounds in Figs 7-11 are only recoverable from figure labels and in-text means cited in the text, not from raw figure data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.