On the Relation between Code Quality and Machine Learning Performance: A Large-scale Empirical Study
Marius Mignard, Steven Costiou, Anne Etien
cs.SE · cs.LG
Quick verdict
Large Kaggle study finds ML-specific—not general—code quality tracks performance; statistically rigorous but correlational, possibly one-rule-driven.
Summary
The authors conduct a large-scale empirical study of 265,363 Python notebooks submitted to Kaggle competitions, testing whether code quality relates to ML performance and whether popularity or author expertise reliably signal quality. From an 8M-notebook January 2026 Meta Kaggle snapshot they filter competition-linked, Python ≥3.6, ≥30-line, tool-runnable notebooks; they measure general Python quality with Pylint and ML-specific quality with SonarQube under a 34-rule profile, and express performance as within-competition z-scores of private leaderboard scores. Using non-parametric tests, effect sizes, and G*Power sensitivity analyses (α=0.005), they find general Python quality decoupled from performance (violation density ρ=0.053, negligible) while ML-specific violation density shows a consistent small negative association with performance (ρ=−0.132), robust across competition types and expertise strata. Upvotes are unrelated to quality or performance; competition experts outperform non-experts (Rrb=0.224) with fewer ML violations (Rrb=−0.278) yet more Pylint errors and refactor issues; code experts are indistinguishable. Main limitations: a correlational design whose confounder control relies on stratification alone, no multiple-comparison correction across dozens of tests, per-rule attribution deferred despite rule S6740 appearing in 87.25% of notebooks, and best-per-team sampling that conditions on the outcome. A Zenodo reproduction package is provided.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- General Python code quality (Pylint) is decoupled from ML performance, with negligible or non-significant correlations across all stratifications tested.
- ML-specific violations (SonarQube 34-rule profile) show a consistent small negative association with performance (ρ=−0.132 overall in the Best-Per-Team dataset), persisting across competition types and expertise levels.
- Neither popularity (upvotes) nor code-track expertise carries information about code quality or performance.
- Competition expertise correlates with better performance (Rrb=0.224), fewer ML violations (Rrb=−0.278), and slightly more Pylint error and refactor violations.
- Software engineering best practices can be adopted from the earliest ML development stages at no measured performance cost.
Strengths
First-at-scale question on a genuinely new corpus
Section 2.2 states no prior dataset joins static-analysis violations, private competition scores and social metadata per notebook; Section 3 builds one from 265,363 competition-linked Kaggle notebooks with a fully documented three-filter pipeline (Fig. 1: 8M → 355,760 → 309,291 → 297,417 → 265,363).
Disciplined statistical framework
Section 4.1: exclusively non-parametric tests, rank-biserial effect sizes with 95% CIs and CLES, Cohen's revised thresholds (Table 1), and a priori G*Power sensitivity analyses for every single test at α=0.005, power 0.95–0.99 (Tables 20–29), with d→r conversion documented.
Confounders measured, not hand-waved
Dual-tool design yields a conceptual finding
External validation and reproducibility
Weaknesses
Headline effect may be carried by one rule
No multivariate confounder control
Best-per-team sampling conditions on the outcome
No multiple-comparison correction
z-score reliability for small competitions unaddressed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This empirical software engineering paper (Univ. Lille / Inria, CRIStAL) tests two assumptions at platform scale: that code quality and ML performance are unrelated, and that Kaggle's social signals (upvotes, expertise tiers) proxy quality. The material is 265,363 competition-linked Python notebooks from a January 2026 Meta Kaggle snapshot, filtered from 8M through three documented stages (Fig. 1): competition submissions minus Getting Started (355,760), Python ≥3.6 with ≥30 SLOC (309,291), tool-runnable under JupyText / Pylint 4.02 / SonarQube 25.11 (297,417), with computable z-scores (265,363). Quality is deliberately dual: Pylint for general Python, SonarQube with a bespoke 34-rule data-science profile (Appendix A). Performance is the private leaderboard score z-standardized within competition (Section 3.3.4). Three RQs: popularity↔quality (RQ1), violations↔performance (RQ2), expertise↔violations (RQ3). The headline: quality is not one construct — general Python quality is decoupled from performance, ML-specific violations carry a consistent small negative association, and social signals are uninformative except competition expertise.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does writing clean code improve machine learning model performance?
According to this study of 265,363 Kaggle competition notebooks, general Python code quality (measured with Pylint) shows no measurable relationship with model performance, but adherence to ML-specific practices (e.g., seeding random generators, specifying dtypes and hyperparameters, avoiding inplace=True) is associated with modestly better leaderboard results (violation density ρ=−0.132, a small effect). The relationship is correlational, not proven causal.
Are highly upvoted Kaggle notebooks better quality?
No. In this dataset upvote count correlates negligibly with both code quality and competition performance (e.g., ρ=0.015 with the Pylint quality score and ρ=−0.079 with the ML quality score, both rated negligible). The authors conclude popularity should not be used as a quality proxy when selecting notebooks to reuse.
Do Kaggle expert badges (Master, Grandmaster) indicate better code?
Partly. Competition-track experts do achieve better scores (Rrb=0.224) and commit fewer ML-specific violations (Rrb=−0.278), but they produce slightly more Pylint error and refactoring violations than non-experts. Code-track experts show no measurable advantage in either quality or performance.
What tools and data did the study use?
Pylint 4.02 for general Python quality and SonarQube 25.11 community edition with a custom profile of 34 data-science/ML rules (pandas, NumPy, scikit-learn, TensorFlow, PyTorch), run on notebook-to-script conversions of 265,363 competition-linked Python notebooks from the January 2026 Meta Kaggle snapshot, with performance taken from private leaderboard scores standardized within each competition.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with two sampling gaps: Section 4.4 (RQ1 methodology), Table 4, and parts of Sections 3.2.3, 5.1, 6.4 and Appendices A–B fall in the gaps; this review cites only figures from visible sections and tables.
- Several effect-size label cells appear as '-' in Tables 8, 10 and 14; this may be a labeling choice or an extraction artifact rather than the authors' omission.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.