Talking to Itself While Coding: What Makes Comments Help Code Generation?
Dangfeng Pan, Zhensu Sun, Cenyuan Zhang, David Lo, Xiaoning Du
cs.SE · cs.AI
Quick verdict
Claims comments help codegen only by carrying correct solution content; well-controlled prefill experiments (+17.2% lift) largely support this causal claim.
Summary
The paper asks whether natural-language comments LLMs emit alongside code causally affect code-generation accuracy (pass@1). Observational analyses on LiveCodeBench v6 across 88 models show comment volume and surface form do not reliably track pass@1 (max within-model |rho|=0.53), and a WHY/WHAT intent analysis of 13 models via an LLM judge (anchor-class kappa 0.72-0.74) yields no correlation surviving Bonferroni correction. The core contribution is a controlled prefill intervention: comment blocks written by stronger sources (Grok-4.1-Fast, Gemini-3.1-Flash-Lite, Claude-Opus-4.7) are attached to four weak recipients. Comments from source-passing solutions lift recipient pass@1 by 17.2% on average (12 pairs, all Holm-significant McNemar), while comments from failed solutions show no reliable gain, wrong-topic comments cost 20.8%, random text 17.9%, line shuffling retains 68% of the lift, and single-commitment corruption of correct comments removes it. A RepoClassBench replication (C#/Java) is direction-consistent. Across 14 models and 10 prompt variants plus planning pipelines, self-elicitation recovers at most ~24% of the external gain, with effects varying in sign by model. Limitations: greedy single-sample decoding, the corrupted condition tested for one source only, floor effects on the source-fail subset, and the intervention being a measurement instrument rather than natural commenting. Code and outputs are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across 88 models on LiveCodeBench v6, neither comment volume/surface form nor (across 13 models) WHY/WHAT intent reliably predicts pass@1.
- Prefilling a weak recipient with a strong source's comments from a passing solution raises pass@1 by 17.2% on average across 12 (recipient, source) pairs, with all cells Holm-significant (paired McNemar).
- Comments matched in form but describing failed solutions give no reliable gain, while wrong-topic comments reduce pass@1 by 20.8% and length-matched random text by 17.9%.
- Corrupting one algorithmic commitment in a correct comment block removes the lift on the same problems, and shuffled line order retains 68% of it.
- Across 14 models and 10 prompt variants plus self-planning/two-stage pipelines, prompting recovers at most ~24% of the external-comment gain, with effects varying in sign by recipient model.
Strengths
Clean causal decomposition of the comment property space
Section 4.1 varies six prefix conditions (source-written, source-correct, source-wrong, shuffled, wrong-topic, random-text) plus single-commitment corruption (Table 2) and four style rewrites (Table 9), pinning content correctness — not form, order, or topic — as the driver of the +17.2% average lift.
Statistical discipline above empirical-SE norms
Paired exact McNemar against matched-prompt per-cell baselines with Holm-Bonferroni per family (m=60 Table 1, m=24 Table 3, m=126 Table 4, m=12 Table 14); paired bootstrap CIs (median width 4.4%); the WHY trend (rho=+0.61, raw p=0.027) is honestly reported as failing the corrected threshold 0.0167.
Explicit confound handling
Generalization effort beyond Python single-file
Strong reproducibility disclosure
Weaknesses
All results rest on greedy, single-sample decoding
The cleanest causal leg covers one source only
Observational sections mix heterogeneous pipelines
Self-elicitation null leans on small recipients
The external prefix carries a full solution plan
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether self-emitted natural-language comments causally affect an LLM's own pass@1 in code generation, and which property — volume, form, intent, order, topic, or content correctness — is responsible. Three claims anchor it: (1) an observational null: across 88 models on LiveCodeBench v6 (Section 3.1), comment volume and surface form do not track pass@1 (max within-model |rho|=0.53 on tok-cmt%), and across 13 models intent does not survive Bonferroni (Section 3.2: WHY rho=+0.61, raw p=0.027 vs corrected threshold 0.0167); (2) an interventional result: prefilling a weak recipient with a strong source's comments from a passing solution lifts pass@1 by 17.2% on average over 12 (recipient, source) pairs, all Holm-significant (Table 1), while failed-solution comments give no reliable gain and wrong-topic comments cost 20.8%; (3) a self-elicitation gap: across 14 models x 10 prompt variants plus SP/TS planning (Section 4.2), prompting recovers at most ~24% (Qwen3-8B dense +4.4% vs +18.1% external). The conclusion — comments help because they carry correct solution content — is convincingly established for the prefill instrument, though the abstract's framing invites broader practical reading than the design licenses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do comments actually help LLMs write better code?
According to this study, only when the comments describe a correct solution. Prefilling a weak model with comment blocks written by a stronger model for problems that model's own solution passed raised the weak model's pass@1 by 17.2% on average across 12 model pairings, while comments describing failed solutions gave no reliable gain and comments about a different problem lowered pass@1 by 20.8%.
How much do correct comments improve code-generation accuracy?
On LiveCodeBench, correct comment blocks lifted four small recipient models (Gemma-4-E4B-IT, Qwen3-8B, Seed-Coder-8B, CodeGeeX4-9B) by an average of 17.2 percentage points, with per-pair lifts ranging from about +14% to +26% depending on the source model (Gemini-3.1-Flash-Lite, Claude-Opus-4.7, or Grok-4.1-Fast).
Can I get the same benefit just by prompting my model to write comments?
The paper finds mostly no: across 14 models and 10 comment-related prompt styles (none, rare, dense, WHAT, WHY, mix, Google-style docstrings, top-of-file plans, inline reasoning), the best case recovered only about 24% of the external-comment gain, and the same prompt could help one model while significantly hurting another.
Do comments need to be in the right order to help?
Largely no. Shuffling the lines of a correct comment block retained 68% of the lift on average, suggesting comments act as mostly independent local hints rather than a step-by-step derivation the model follows in order.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as captions and embedded code listings in the extracted text; visual layout of the sub-experiment diagram is not verifiable.
- Appendix tables (7, 10-19) were extracted as dense running text; cell alignment was cross-checked against in-text numbers where possible, but fine-grained per-row values were not independently verifiable beyond those stated in the prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.