Efficiency Hallucination: Formalizing and Measuring Behavioral Calibration in LLM-Based Code Optimization
Sarah Wilson, Gail Kaiser, Patrick Musau
cs.SE · cs.AI
Quick verdict
Claims LLMs always edit already-optimal code and a prompt guardrail cuts over-edits 100%→55.6%; plausible pilot, but n=45, no stats, no runtime verification.
Summary
This pilot study formalizes the 'Efficiency Hallucination' — LLMs modifying already-optimal code while asserting unsubstantiated speedups — and attributes it to an 'Evaluation Trap': benchmarks like PIE and EffiBench reward correctness and speedups but never penalize pointless edits, making bluffing cost-free. Building on Kalai et al.'s Is-It-Valid framework, the authors run nine models (three each from GPT, Claude, Gemini) across 180 API trials on five paired optimal/sub-optimal LeetCode problems from EffiBench, comparing a control prompt with a penalty prompt requiring >90% confidence or the output ALREADY_OPTIMAL. Under control, all models edited optimal code in 100% of trials. The penalty prompt raises correct abstention to 44.4% (20/45) while preserving a 100% edit rate on sub-optimal code and zero false abstentions. Calibration is uneven: GPT-5.4-Mini abstains 5/5 while GPT-5.4 manages 1/5; per-problem abstention ranges from 89% (a two-pointer problem) to 11% (a Counter-based solution), suggesting surface complexity drives bluffing. Main limitations: tiny per-model samples (n=5, one draw per cell) with no statistical tests or confidence intervals; 'optimal' snippets are assumed ceilings without runtime verification of over-edits; sub-optimal variants were generated by Gemini, risking a self-recognition confound; and only five algorithmic contest problems, so generalization beyond toy code is untested. Code and data are stated to be on GitHub.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under standard optimization prompts, all nine tested LLMs modified already-optimal code in 100% of trials (45/45) with zero abstentions, confirming the Evaluation Trap.
- A single-sentence penalty prompt (>90% confidence or output ALREADY_OPTIMAL) raises correct abstention on optimal code from 0% to 44.4% while preserving a 100% edit rate on sub-optimal code and zero false abstentions.
- Calibration inverts capability within families: GPT-5.4-Mini abstains 5/5 while GPT-5.4 abstains 1/5, and Gemini-3 Flash Preview (60%) outperforms Gemini-3.1 Pro Preview (20%).
- Surface code complexity independently predicts bluff rate, with per-problem abstention ranging from 89% (two-pointer sweep) to 11% (Counter/nested-comprehension solution).
- Benchmarks like PIE and EffiBench supply no negative signal for non-improving edits, making bluffing at performance ceilings the dominant strategy as predicted by the IIV framework.
Strengths
Genuinely novel evaluation axis
The Optimal Baseline methodology (Section III-C) tests agents on code already at its performance ceiling — a failure mode structurally invisible to PIE, EffiBench, ENAMEL and SWE-Perf, which by construction only test improvable code (Section II-A).
Clean decoupling result
Table I shows the penalty prompt raises optimal-code abstention from 0% to 44.4% while sub-optimal edit rate stays 1.000 and False Abstention is 0.000 — the guardrail costs nothing on genuinely improvable code.
Transparent, internally consistent reporting
Unusually candid limitations
Actionable zero-cost mitigation
Weaknesses
No statistical inference anywhere
No runtime verification of ground truth
IIV formalization is analogical
Single operating point for the key knob
Narrow domain and behavior-proxy measurement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Efficiency Hallucination (arXiv 2609.14839)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is an Efficiency Hallucination in LLM code optimization?
It is a failure mode where a large language model, asked to optimize code that is already optimal, modifies it anyway and confidently claims a performance improvement that does not exist. The code still compiles and passes tests, so the fake optimization is invisible to normal test suites — it just wastes review time and adds unneeded complexity.
What is the Evaluation Trap?
The authors' term for the incentive structure of code-optimization benchmarks (PIE, EffiBench): they score functional correctness and runtime improvement but never penalize pointless edits to already-fast code. For a model facing optimal code, abstaining earns nothing while bluffing an edit is cost-free, so unnecessary modification becomes the rational strategy.
How well does the paper's penalty-prompt guardrail work?
In this 180-run pilot across nine models (GPT, Claude, Gemini families) on five LeetCode problems, adding a single instruction — only edit if >90% confident of a speedup, otherwise output ALREADY_OPTIMAL — raised correct abstention on optimal code from 0% to 44.4%, kept a 100% edit rate on genuinely improvable code, and produced zero false abstentions. Caveats: only 45 decisive trials, no runtime verification, and one threshold tested.
Which models were best at recognizing already-optimal code?
GPT-5.4-Mini abstained correctly in all 5 trials (100%), the only perfect score. Family averages were Claude 53% (8/15), GPT 53% (8/15), and Gemini 27% (4/15). Gemini-3.5-Flash abstained 0/5. Strikingly, lighter models often calibrated better than larger siblings — GPT-5.4-Mini (100%) vs GPT-5.4 (20%) — a pattern the authors call the Capability-Calibration Inversion.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The GitHub repository (sarah-wilsxn/efficiency-hallucinations) was not inspected; code/data availability is as stated in Section IV-A.
- Figure 1 is available only as embedded text labels; its visual content was assessed from the extracted per-model values in Tables II and III.
- Model identifiers (GPT-5.4, Gemini 3.5 Flash, Claude Opus 4.8) are proprietary APIs whose behavior may have drifted since the runs; exact sampling configurations are unreported.
- The extracted text appears complete (Sections I-VII, all three tables, references), but page-level layout and any supplementary material could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.