Style, Not Self: Surface Cues Explain Zero-Shot Code Attribution by Large Language Models
Ehsan Barkhordar, Surendrabikram Thapa
cs.AI · cs.CL · cs.SE
Quick verdict
Rigorous negative study: zero-shot LLM code self-recognition tracks length/style cues, not self-knowledge; caveats: mostly small models, short MBPP code.
Summary
The paper asks whether commercial LLMs can recognize their own code zero-shot. Five models (GPT-5, Claude Haiku 4.5, Gemini 2.5 Flash, Grok 4 Fast, DeepSeek-V3) generate solutions to MBPP, HumanEval, and DS-1000, with seven additional generators for cross-model attribution on MBPP. Models evaluate in four tasks: pairwise self-recognition, individual self-recognition, attribution to a named model, and blind quality judgment. In the individual task, balanced accuracy stays within 49-58% across all 15 model-benchmark combinations; raw accuracies of 38-67% reflect response policy, not ability (GPT-5 never answers 'yes'). Pairwise accuracy correlates at r=0.93 with how often the evaluator's own solution is longer, and judges mostly pick the longer, better-documented solution. Attribution to a named model succeeds on some pairs (up to 76.2%) yet is consistently inverted on others (down to 7.8%), matching family stereotypes rather than style knowledge. An AST-based normalization that strips docstrings, comments, type hints, and local names preserves Pass@1 and drives ten of twelve re-tested results to chance; the other two follow residual length differences. A trained classifier still attributes normalized GPT-5 pairs at 93-96%, showing the signal exists but is out of reach zero-shot. Claude Haiku's self-preference also vanishes. Limitations: mostly small or fast models, short MBPP-scale code, all-cues-at-once normalization, and a six-month gap between experimental rounds.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Zero-shot individual self-recognition is at or near chance: balanced accuracy lies between 49% and 58% for all 15 model-benchmark combinations, and raw accuracies of 38-67% reflect response policy rather than ability.
- Pairwise self-recognition accuracy is largely a function of surface cues: across 14 evaluator-opponent combinations it correlates at r=0.93 with how often the evaluator's own solution is the longer one.
- Attribution to a named model succeeds on some pairs and is consistently inverted on others, indicating judges separate styles but attach wrong family-based priors rather than knowing the named model's style.
- An AST normalization that preserves Pass@1 removes most zero-shot attribution signal (ten of twelve re-tested results at chance) while a trained classifier still attributes the normalized code, so the signal exists but is not zero-shot accessible.
- The one clear self-preference effect (Claude Haiku favoring its own solutions against DeepSeek, +11.8 points) is tied to a taste for documentation and disappears after normalization.
Strengths
Heuristic baselines on identical items
Section 3.3 scores seven non-LLM rules (length, docstring, type hints, comments) on the exact same pairs as the judges — the control most prior self-recognition studies omit. The baselines often match or beat the LLMs: the length rule scores 97.5% vs GPT-5's own 81.9% on GPT-5 vs Grok, and 89.0% vs the best judge's 81.6% on Claude Haiku vs DeepSeek (Table 2).
Statistical and reporting hygiene
Wilson 95% CIs, exact binomial tests vs 50%, Holm step-down correction within tables and across the twelve normalized re-tests, exact McNemar for self-preference, position-balanced accuracy, and label-consistency analysis (Table 14) that cleanly separates random from inverted judges. Section 4.3 explicitly shows always-answering 'no' scores 66.7%, above Claude Haiku's raw 65.7% on MBPP.
Behavior-preserving normalization as a causal-style probe
Classifier contrast quantifies the gap
Transparency about a self-discovered confound
Weaknesses
Capability ceiling untested
Ecological validity of short code
All-cues-at-once normalization
Temporal and version confound in re-runs
Sparse opponent coverage in Task 1b
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether commercial LLMs can recognize their own code, or attribute code to a named model, without training — the zero-shot prerequisite for self-preference bias and for generator-monitor collusion in AI-control pipelines (Section 1, citing Greenblatt et al. 2024). Five core models (GPT-5, Claude Haiku 4.5, Gemini 2.5 Flash, Grok 4 Fast, DeepSeek-V3-0324) generate solutions to MBPP sanitized (257 problems), HumanEval (164), and DS-1000 Pandas (first 150); seven additional generators cover MBPP for third-party attribution. Four tasks are studied: pairwise self-recognition (1a), individual yes/no self-recognition (1b), target identification (2), and blind self-preference (3). Headline claims: (i) zero-shot self-recognition is at or near chance — balanced accuracy 49-58% across all 15 model-benchmark combinations (Section 4.3); (ii) pairwise accuracy is largely a length/documentation artifact, correlating r=0.93 with the rate at which the evaluator's own solution is longer (Section 4.2); (iii) a Pass@1-preserving AST normalization eliminates ten of twelve re-tested results (Section 4.5); (iv) target attribution fails in structured ways, including consistent inversions that indicate style discrimination with a mislabeled prior, not an inability to separate styles (Section 4.4, Table 14).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Can LLMs recognize code they wrote themselves?
According to this study, zero-shot (just asking the model, no fine-tuning) largely no: balanced accuracy across 15 model-benchmark combinations stayed within 49-58%, and apparent pairwise success was mostly explained by simple cues such as picking the longer or better-documented solution rather than recognizing one's own style.
What is self-preference bias in LLM judges, and did this paper find it?
Self-preference is a judge favoring its own output. The study found one clear case — Claude Haiku 4.5 chose its own MBPP solutions 75.2% of the time against DeepSeek-V3 — but this tracked a taste for documentation (84% choice of documented solutions) and vanished after code normalization removed surface cues.
Why do some studies report high LLM self-recognition while others find none?
The paper's methodological findings help explain the disagreement: an early prompt that listed author names in the same order as the solutions let judges score 86-99% from prompt structure alone, and single raw accuracies in yes/no tasks conflate response policy with ability, since always answering 'no' scores 66.7% by construction.
What does this mean for using LLMs to monitor other LLMs (AI control)?
Moderately reassuring for zero-shot settings: tested models could not reliably tell their own short code from others', and one collusion channel — a monitor recognizing its generator's style — was driven by removable surface cues. But the authors warn redaction is not anonymization: a trained classifier still attributed normalized code, and stronger or fine-tuned monitors may behave differently.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and appendix tables are available only as extracted text and numbers; visual layout of plots could not be verified.
- The released GitHub repository (footnote 1) was not independently inspected; artifact completeness is taken from the paper's claim.
- arXiv v1 (24 Sep 2026) only; no journal peer-review status is available.
- Prompt templates in Appendix A may have minor formatting artifacts from text extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.