On the Lexical Superstition of Large Language Models for Code Comprehension: Re-evaluation on Code of Low Lexical Quality
Xin Shen, San-Zhuo Xi, Yali Du, Ming Li
cs.SE · cs.CL · cs.LG
Quick verdict
Claims pervasive lexical overemphasis in LLM code comprehension via Face/Off renaming; well-controlled and honest, but main results lack significance tests.
Summary
The paper asks whether large language models overweight identifier names when comprehending code. It introduces Face/Off, a framework applying semantics-preserving identifier renaming (verified via AST identity and alpha-equivalence checks) along a progressive spectrum: Perturbative (CodeBERT-sampled similar names), Anonymous (systematic placeholders), and Confusing (identifiers borrowed from an unrelated donor snippet). Across four tasks - code-document matching, code search, code summarization, and code completion - on eight models plus an embedding API, performance generally declines as lexical information degrades: CodeBERT's search MRR falls from 93.05 to 37.06 under Confusing naming (-60.2%), HumanEval Pass@1 drops up to 34.2% (LLaMA3:70B), and outputs track donor meanings, with GPT-3.5-turbo falling to 52.39% (near chance) in a two-document selection task. Prompt-based and fine-tuning interventions narrow but never eliminate the gap, while a type-inference control (317 targets, Holm-corrected McNemar) shows effects are small and insignificant when types are locally recoverable. Main limitations: headline tables lack significance tests or confidence intervals, summarization metrics are lexically coupled to identifiers, the model lineup is dated (GPT-3.5-turbo, GPT-4o-mini, LLaMA3:70B), and no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across the evaluated models and four code-comprehension tasks, performance generally decreases as identifier information moves from informative to absent to misleading, with Confusing naming producing the largest drops.
- Misleading identifiers directionally steer outputs toward donor meanings, evidenced by near-chance two-document selection (GPT-3.5-turbo 52.39%) and donor-reference similarity above random-pair baselines.
- Representative prompt-based and fine-tuning interventions reduce but never eliminate the naming-condition gap, indicating lexical overemphasis is not readily suppressed.
- A type-inference control on 317 locally-determinable targets shows naming effects become small and statistically insignificant when the answer does not require the name, bounding the generality of the overemphasis claim.
Strengths
Strictly controlled perturbation design
Eligible renamings are verified by AST identity modulo renamed tokens (§3.2.1); the type-inference control adds parsing, compilation, binding-occurrence, and alpha-equivalence checks (§4.3); the external-identifier study (§4.2) includes a dead-code insertion control (Table 7).
Boundary-condition control with proper statistics
§4.3 (317 ManyTypes4Py targets, project-clustered CIs, Holm-corrected McNemar) shows naming effects shrink to -3.2..+2.2pp and lose significance when types are locally recoverable - a falsification-aware design that strengthens the causal reading of the main effects.
Directional evidence beyond aggregate drops
Prompt robustness check
Unusually candid reporting
Weaknesses
No inferential statistics for headline results
Lexically coupled summarization evidence
Dated model lineup vs. 'current LLMs' framing
No code/data availability statement
Thin completion-side directional evidence
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper investigates whether LLMs overweight identifier names ('lexical overemphasis') during code comprehension. Its instrument is Face/Off (§3.3): semantics-preserving identifier renaming along a progressive spectrum - Perturbative (CodeBERT-sampled similar names), Anonymous (func0/var0 placeholders), Confusing (identifiers borrowed from a random donor snippet), plus a non-semantics-preserving Shuffling condition confined to §7.2. Eight models (GPT-3.5-turbo, GPT-4o-mini, LLaMA3:70B-Instruct, DeepSeek-Coder-V2 via prompting; CodeBERT, GraphCodeBERT, CodeT5, DeepSeek-Coder-1.3B via fine-tuning) plus Text-embedding-3 are tested on four tasks: code-document matching, code search, and code summarization (600 CodeSearchNet samples per condition), and code completion (filtered HumanEval, Pass@1). Three RQs structure the evidence: sensitivity magnitude (RQ1), directional influence (RQ2), and persistence under interventions (RQ3). The claim is deliberately bounded: a type-inference control (§4.3) shows the effect vanishes when the answer is locally recoverable.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Face/Off framework for evaluating LLMs on code?
Face/Off is an evaluation method that renames identifiers in code while verifying the program's behavior is unchanged (via AST identity and alpha-equivalence checks). It creates a spectrum of naming conditions - similar-but-different names (Perturbative), neutral placeholders like func0 (Anonymous), and identifiers copied from an unrelated program (Confusing) - to measure how much a model's answers depend on the words in names rather than on what the code actually does.
Do LLMs rely too much on variable and function names when reading code?
This paper finds substantial evidence of overreliance ('lexical overemphasis'): across four comprehension tasks and eight models, accuracy dropped whenever descriptive names were removed or replaced, with the largest drops when misleading names from unrelated code were inserted - e.g., code-search MRR for CodeBERT fell 60.2% and GPT-3.5-turbo dropped to near chance (52.39%) when choosing between two descriptions. However, when the answer (e.g., a variable's type) was recoverable from local code without the name, naming effects became small and statistically insignificant.
Can misleading variable names trick GPT models into writing wrong code?
Yes, in the tested settings. The paper's case study shows GPT-3.5-turbo generating code that matches misleading names (an even/odd counter) instead of the actual implementation (mean absolute deviation), and in a two-document selection task its accuracy fell from 92.84% to 52.39% - essentially random - when the code carried identifiers from a different program.
Does fine-tuning or prompt engineering fix the problem?
Only partially. Adding warnings and code-sanitization steps to prompts, or fine-tuning on anonymized code, improved scores under misleading names (e.g., CodeT5's ROUGE-L under Confusing naming rose from 11.69 to 14.95) but never closed the gap to performance on normally named code in the evaluated configurations.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 are not analyzable from the extracted text; Figure 6 similarity values were read from embedded axis labels and could not be fully verified.
- No code or data availability statement appears in the extracted text; if one exists in an appendix or external project page it was not captured.
- Review based on arXiv v1 preprint ('Manuscript under review'); a camera-ready revision may address statistical reporting gaps.
- Prompt templates and few-shot examples are abbreviated (Figure 3), so exact prompt-level replication fidelity is uncertain.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.