GitScholar: A Dataset for Predicting AI Research Impact from GitHub Engagement
Emilien Guandalino, Lorenz K. Müller, Beatrice Alessandra Motetti, Konstantin Berestizshevsky, Lukas Cavigelli
cs.DL · cs.AI · cs.LG · cs.SI
Quick verdict
A large, carefully time-stamped GitHub-arXiv dataset; the 12% early-precision gain is credible but figures lack numeric tables and error bars.
Summary
The paper introduces GitScholar, a publicly released heterogeneous graph linking GitHub activity from roughly 444,000 repositories to over 558,000 AI arXiv papers (cs.AI/LG/CL/CV), plus 820,000+ authors and ~11M papers from Semantic Scholar for citation and author features. All nodes, edges, and engagement metrics are timestamped, enabling backtesting at arbitrary past dates; README arXiv mentions are recovered via regex plus a binary search over commit history. In a top-1000-precision experiment (predicting one-year citation counts at weekly lags with a 2-layer GNN trained on log-MSE), adding GitHub features to a citation+author baseline yields up to 12% absolute precision gain, peaking one to two weeks after publication; a LightGBM appendix run shows similar patterns. Coverage analysis shows near-complete GitHub coverage of the top 1% most-cited AI papers, while filtering out non-mentioned papers removes 23% of AI papers but almost none high-impact. Against a 2019 bot-filtered TweetPap Twitter snapshot, stars correlate 0.272 with one-year citations versus 0.035 for likes, and the star-citation correlation rises to 0.374 for 2024-25 papers. Limitations: headline gains are read from figures without numeric tables, confidence intervals, or seed variance; author features in the baseline are thin; the Twitter comparison spans different eras; and citation-count impact is itself an approximation.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GitScholar links GitHub engagement from ~444,000 repositories to 558,000+ AI arXiv papers with per-feature timestamps that allow reconstructing the graph at any past date.
- Adding GitHub features to a citation+author GNN improves top-1000 one-year-impact precision by up to 12% absolute, peaking one to two weeks after publication.
- GitHub achieves near-complete coverage of the top 1% most-cited AI papers; filtering out non-mentioned papers removes 23% of AI papers but almost none that are high-impact.
- GitHub stars correlate more strongly and robustly with one-year citations than Twitter likes (0.272 vs 0.035, filtered 2019 data), and the star-citation correlation rose ~44% to 0.374 by 2024-25.
Strengths
Leakage-safe temporal design
All entities/edges are timestamped; README link histories are recovered by binary search over commits; train (2021-03-21 to 2023-03-20, N=101,428), validation (to 2023-06-20, N=18,115) and test (2024-06-18 to 2025-06-17, N=94,525) splits are separated by publication date with a deliberate one-year maturity buffer before test-start (Sections 2.2-2.3, 3.1).
Clean ablations isolate the GitHub contribution
Four matched feature sets (Citation / +Author / +Git / +Git+Author) are trained identically; the GitHub-only model beats author-only at early lags and the combined model peaks at +12% absolute precision over the citation+author baseline (Section 3.1, Figure 2).
Transparent Twitter contamination analysis
Findings are model-agnostic
Reusable, extensible release
Weaknesses
Headline numbers are figure-only with no uncertainty
The 'strong academic baseline' is thin
Twitter comparison is era-confounded
Mention coverage conflates platform truth with harvesting method
Objective/evaluation mismatch and unquantified filtering
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GitScholar is a dataset paper with supporting experiments: a heterogeneous, fully timestamped graph linking GitHub engagement (~444,000 repositories whose README mentions 'arxiv') to 558,000+ AI arXiv papers (cs.AI/LG/CL/CV), plus 820,000+ authors and ~11M papers drawn from the Semantic Scholar corpus for citation and author features (Sections 2.1-2.2). Three claims are tested: (1) GitHub features improve early identification of high-impact papers - up to 12% absolute precision gain over a citation+author baseline (Section 3.1, Figure 2); (2) GitHub coverage of the top 1% most-cited AI papers is near-complete, with 23% of AI papers unmentioned (Section 3.2, Figure 3); (3) GitHub stars correlate with one-year citations more strongly and more robustly than Twitter likes (0.272 vs 0.035 filtered, 2019; 0.374 for 2024-25 papers) (Section 3.3, Tables 1-2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the GitScholar dataset?
A public dataset (Hugging Face: huawei-csl/GitScholar) that links GitHub activity - stars, forks, issues and pull requests from ~444,000 repositories whose README mentions arXiv - to more than 558,000 AI papers on arXiv (cs.AI/LG/CL/CV), with author and citation features from Semantic Scholar. Every entity and interaction is timestamped, so users can reconstruct the data as it looked on any past date, which enables backtesting prediction models.
How much does GitHub activity improve AI paper impact prediction?
In the paper's experiments, adding GitHub features to a model that uses citations and author information raises the precision of identifying the future top-1000 most-cited papers by up to 12 percentage points (absolute), with the largest gains one to two weeks after publication - exactly when citations are still too sparse to be informative.
Do influential AI papers always have a GitHub repository?
Nearly, according to the paper: among AI papers published 2024-2025, about 23% were never mentioned in a GitHub README within their first year, but coverage of the top 1% most-cited papers is near-total. For mainstream AI papers, absence from GitHub is therefore itself a meaningful (negative) signal, unlike in physics where many impactful papers are simply absent.
Is GitHub a better signal than Twitter/X for predicting paper impact?
The evidence is historical: using the 2010-2019 TweetPap dataset with bots removed, GitHub stars correlated 0.272 with one-year citations versus only 0.035 for Twitter likes. For 2024-25 AI papers the star-citation correlation reached 0.374. The two platforms were never compared on the same papers and time period, so the paper cannot strictly prove present-day superiority over X.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-6 (precision curves, coverage plot, correlation densities, GNN vs LightGBM) carry the headline results but only captions are extractable from text; numeric values could not be independently verified.
- No numeric precision table exists anywhere in the text; the 12% magnitude and curve shapes rely on the authors' prose description.
- Training/analysis code availability is not stated; only the dataset is confirmed released, so 'codeAvailable' reflects the public dataset.
- The Appendix A LightGBM comparison is figure-only, so the claimed architectural robustness could not be checked numerically.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.