Retrofitting Code Using LLMs to Support Exceptional Behavior
Linghan Zhong, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric
cs.SE · cs.CL
Quick verdict
Proposes retrofitting code with exception logic from tests; consistent large LLM gains across 5 models, yet ~25% of test-passing code differs from intent.
Summary
The paper introduces a new task: retrofitting existing Java methods with Exception Related Code (ERC)—throw statements plus the conditions guarding them—so that developer-supplied Exceptional Behavior Tests (EBTs) pass while normal behavior is preserved. The authors build EXCODER, which augments the LLM prompt via context engineering: static analysis supplies exception-constructor signatures and available symbols; dynamic analysis annotates exceptions thrown at runtime, per-EBT line coverage, and non-EBTs exercising the method. A benchmark of 304 evaluation methods from 75 GitHub Java projects is created by systematically stripping ERC, with EvoSuite/Randoop-generated tests added as stronger oracles. Across five models, EXCODER beats a context-free baseline on all metrics: with Qwen 2.5 Coder 32b, pass@1 on all user tests rises from 73.36% to 85.92% (+12.56 p.p.), and with GPT-5 Mini from 82.73% to 95.03%. Combining with self-repair lifts Iter-EXCODER to 96.22% pass@5 versus 84.47% for Iter-Base. Manual inspection (two inspectors plus Claude Code, Cohen's κ 0.77–0.87) shows only 65.13% of EXCODER's test-passing ERC are semantically equivalent to ground truth; the ~25% false-positive rate is unchanged by added context, dominated by too-lenient checks limited by EBT coverage. Limitations: Java/Maven only, stripped rather than naturally missing ERC, no significance testing, and only a weak no-context baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Retrofitting existing code with ERC given exceptional behavior tests is a novel task, distinct from prior work that avoids unhandled exceptions.
- EXCODER's static-plus-dynamic context engineering consistently improves LLM ERC retrofitting across five models and all metrics (e.g., +12.56 p.p. pass@1 for Qwen 2.5 Coder 32b on all user tests, reaching 85.92%).
- LLM self-repair cannot substitute for context: after 4 repair rounds, Iter-Base reaches 84.47% pass@5 (All User) versus 96.22% for Iter-EXCODER (Table V).
- Passing developer tests does not imply semantic equivalence: only 65.13% of EXCODER's passing ERC match ground truth, and the ~25% false-positive rate is not reduced by the added context, being bounded by EBT coverage (Table VI).
Strengths
Novel, crisply formalized task
Section II defines ERC retrofitting with explicit success criteria (pass all EBTs and all non-EBTs), cleanly separated from prior unhandled-exception-avoidance work (Section I, Related Work IX); the 49.7% coverage figure justifies the scope assumption.
Fully documented benchmark funnel
Section IV-A traces 1,099 methods from 150 Maven projects down to 304 evaluation methods across 75 projects with stated filters; Section IV-B enumerates removal rules and transparently keeps 38 (12.5%) intentionally non-compiling stripped methods (34 missing-return, 4 unreported-checked-exception).
Layered evaluation beyond pass@k
Honest headline-qualifying finding
Consistent, dissected gains
Weaknesses
No uncertainty quantification
Only a weak external baseline
Potential training-data contamination unaddressed
Abstract emphasizes pass rates over fidelity
Narrow scope and inference setup
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper formalizes a new task — retrofitting Exception Related Code (ERC: throw statements plus the conditions guarding them) into existing Java methods so that developer-supplied Exceptional Behavior Tests (EBTs) pass while non-exceptional behavior is preserved (Section II). Three claims are made: (1) the task is novel and distinct from prior work on avoiding unhandled exceptions; (2) EXCODER's static/dynamic context engineering improves LLM performance across five models; (3) passing tests does not imply semantic equivalence, quantified via manual inspection. Claims (1) and (2) are well supported; (3) is honestly reported but underweighted in the abstract.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'retrofitting code with exceptional behavior' mean?
It means automatically adding exception-related logic — throw statements plus the conditions (such as null checks) that decide when to throw — into existing methods, guided by developer-written tests specifying 'this input should cause this exception', while keeping the method's normal behavior unchanged.
How accurate is EXCODER?
On 304 Java methods from 75 GitHub projects, EXCODER with Qwen 2.5 Coder 32b raises pass@1 on developer test suites from 73.36% (plain prompt) to 85.92%; with GPT-5 Mini it reaches 95.03%. However, manual inspection found only 65.13% of the test-passing code was semantically identical to the developers' original exception logic.
Does passing the exceptional-behavior tests mean the generated code is correct?
Not fully. About a quarter of test-passing outputs were false positives, mostly 'too lenient' checks that throw under fewer inputs than the original code intended — a limit of what the provided tests can express, not of prompting.
Is the EXCODER code and dataset public?
Yes. The paper states that code, experimental scripts, and the benchmark dataset are publicly available at github.com/EngineeringSoftware/excoder.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3 and Figure 4 data points are interleaved/garbled in the PDF extraction, so figure-internal values could not be verified; all numbers cited in this review come from tables and running prose.
- The abstract ends mid-token in the extraction ('and 13'), but the value 13.15 p.p. is confirmed in Section I — an extraction artifact, not missing content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.