What is the Difference Between Me and You? Benchmarking the Quality Gap Between Human-Written and AI-Generated Code
Cristina Improta, Pietro Liguori, Domenico Cotroneo
cs.SE · cs.AI
Quick verdict
Claims AI code is compressed, templated, differently defective and vulnerable vs human across 787k pairs; large-scale, well-controlled, largely credible.
Summary
The authors compare 787,562 function pairs (285,249 Python, 221,795 Java, 280,518 C), each pairing a human function mined from 34k+ open-source repositories with implementations generated from its docstring by OpenAI GPT models (ChatGPT for Python/Java, gpt-oss-20B for C), DeepSeek-Coder-33B, and Qwen2.5-Coder-32B. They measure structural complexity (NLOC, cyclomatic complexity, Halstead, Maintainability Index), n-gram naturalness (cross-entropy/perplexity with identifier normalization), defects (Pylint/PMD/Clang-Tidy mapped to ODC), and vulnerabilities (Semgrep mapped to CWE with severity). AI code is roughly half the size and branching of human code, clusters apart stylistically, and concentrates repetitive boilerplate defects (e.g., unused-argument) while human code concentrates mature-codebase issues. Security is language-dependent: in Java, LLMs reach up to 19.56% vulnerable samples versus 3.00% human; in C, human code carries more high-severity findings (13,565 vs 5,941 for gpt-oss). Size-adjusted correlations show complexity metrics carry little signal after controlling for NLOC, while naturalness separates authors. The authors release CQBench (27,346 issue-prone tasks) and show Claude Opus 4.8 still triggers defects on roughly two-thirds of a 600-task subset. Limitations: no statistical precision estimate for the static analyzers, single-rule dominance in Python defects (protected-access), model heterogeneity inside the OpenAI label (gpt-3.5-turbo vs gpt-oss-20B), and associational-only RQ4 analysis.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AI-generated code is structurally compressed: roughly half the NLOC and cyclomatic complexity and one-third to one-half the Halstead volume of paired human functions across all three languages (RQ1).
- AI code is stylistically templated: after identifier normalization, human code sits at the highest cross-entropy in every language, and dendrograms separate human from the LLM cluster (RQ1, Figs. 2-4).
- Defect and vulnerability profiles differ in kind: human code concentrates mature-codebase issues while AI code concentrates repetitive boilerplate; LLMs produce more and more severe security findings in Python and Java, but fewer high-severity findings than humans in C (13,565 human vs 5,941 gpt-oss Critical+Error) (RQ2-RQ3).
- After controlling for whole-sample NLOC, complexity metrics carry little signal about defect and vulnerability counts, while naturalness separates authors (RQ4, Figs. 9-12).
- CQBench (27,346 issue-prone tasks) remains challenging for a post-construction frontier model: Claude Opus 4.8 triggers defects on roughly two-thirds and vulnerable findings on 30.2% of a 600-task subset (§7, Table 11).
Strengths
Function-for-function paired design at scale
787,562 docstring-conditioned pairs across 34k+ repositories make observed differences attributable to authorship rather than task selection or repository norms — directly fixing the core confound of repo-mined AI-attribution studies (§1-2, Table 1 vs Mao et al.).
Taxonomy-grounded comparability
Rule-level mappings of Pylint/PMD/Clang-Tidy findings to ODC and Semgrep findings to CWE+severity (Table 3, §3.2.3-3.2.4, released in the replication package) make four authors and three languages directly comparable instead of comparing raw tool-specific rule counts.
Confound-aware reporting
Size-adjusted RQ4 design
Exceptional reproducibility package
Weaknesses
Static-analyzer precision never statistically estimated
Docstring-only generation may manufacture part of the 'compression' finding
OpenAI label pools two different models
C corpus quality anomalies under-analyzed
RQ4 'explain' framing outruns effect sizes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether AI-generated code differs from human code in the quality dimensions that dominate lifecycle cost, not just functional correctness. It compares 787,562 function pairs — 285,249 Python, 221,795 Java, 280,518 C (Table 4) — where each human function mined from 34k+ GitHub repositories (CodeSearchNet/HMCorp for Python/Java; TheVault for C) is paired with implementations generated from its docstring by OpenAI GPT models (ChatGPT gpt-3.5-turbo; gpt-oss-20B for C), DeepSeek-Coder-33B, and Qwen2.5-Coder-32B. Four RQs cover structural complexity and n-gram naturalness (RQ1), ODC-mapped defects from Pylint/PMD/Clang-Tidy (RQ2), CWE-mapped severity-stratified Semgrep findings (RQ3), and size-adjusted correlations linking the two (RQ4). Central claims: AI code is structurally compressed (roughly half NLOC/CCN, one-third to half Halstead volume) and stylistically templated; defect profiles differ in kind; security is language-dependent with a C reversal in high severity; and complexity metrics lose signal once size is controlled. A released benchmark, CQBench (27,346 tasks), is demonstrated on Claude Opus 4.8.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is AI-generated code worse than human-written code according to this study?
Not simply worse — different. Across 787,562 paired functions in Python, Java, and C, AI code was about half the size and branching of human code, more templated in style, and had different defect and vulnerability profiles. In Python and Java, LLM code had more security findings (up to 19.56% vulnerable samples in Java vs 3.00% for human code), but in C, human code actually carried more high-severity findings (13,565 vs 5,941 for gpt-oss).
What is CQBench?
CQBench is a public benchmark of 27,346 'issue-prone' programming tasks (10,354 Python, 10,103 Java, 6,889 C) derived from the study's failure data, released with frozen analyzer rules, integrity hashes, baseline outputs from the OpenAI, DeepSeek, and Qwen models, and an evaluation pipeline for testing whether new models produce code with fewer detected defects and security weaknesses.
Which AI models were compared in this study?
The study compared human code against code from OpenAI models (ChatGPT gpt-3.5-turbo for Python/Java, gpt-oss-20B for C), DeepSeek-Coder-Instruct 33B, and Qwen2.5-Coder-Instruct 32B, plus a demonstration with the newer Claude Opus 4.8 on a 600-task subset of the released benchmark.
Does this paper show AI code is more insecure than human code?
It depends on the language. In Java the gap was widest (DeepSeek-Coder: 19.56% vulnerable samples vs 3.00% human) and in Python LLMs also produced more findings, but in C human code had more high-severity (Critical+Error) findings (13,565 vs 5,941 for gpt-oss), with LLMs shifting residual risk toward race conditions and command injection.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with several skipped middle passages (end of §2.1 related work, start of §3.2.3, the RQ1 size/Halstead results and Table 5 values, parts of §5 analyzer configuration, the C subsection of §6.2 and the opening of §6.3); judgments leaning on those passages carry extra uncertainty.
- The excerpt ends mid-sentence in §8 (Discussion); §9 Threats to Validity and §10 Conclusion are not present in the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.