CodeGraph: Open-Taxonomy Knowledge Graph for Source Code with Wikidata Grounding
Federico Pennino, Andrea Gurioli, Stefano Zacchiroli, Maurizio Gabbrielli, Paolo Ferragina
cs.SE · cs.CL · cs.IR
Quick verdict
158M-node LLM-annotated, Wikidata-grounded code KG: impressive scale, unusually honest QA, but grounding precision and downstream utility remain unvalidated.
Summary
The authors present CodeGraph, a pipeline that annotates source files with open-taxonomy semantic labels — application domains, algorithms, paradigms, and design patterns — using Qwen3-Coder-30B-A3B-Instruct, then grounds the extracted concepts in Wikidata via a three-stage procedure: deterministic SPARQL lookup, a Deep Research Agent (Qwen3.6-27B) for ambiguous cases, and a two-hop parent rollup. Applied to Stack-Edu, the pipeline yields a typed property graph of 158M nodes and 1.02B edges over 145M files in 14 languages, with ~63K consolidated concepts and 19.8K grounded Wikidata entities, at a cost of ~400,000 GPU-hours on the LEONARDO supercomputer. Quality assurance combines a small human gold set (127 files, 292 judgements, 17 annotators; Fleiss' κ = 0.153) with a six-model verifier bake-off won by Devstral-2 2512 (F1 88.5%), deployed as a corpus-wide silver filter that brackets precision between 68% and 88%. Strengths: unprecedented scale, released code and graph, and unusual candour about weak inter-annotator agreement. Weaknesses: entity-linking precision to Wikidata is never validated; the two use cases (semantic search, algorithm selection) are conceptual only, with no retrieval or selection experiments; internal tallies of AlgorithmCategory (268K) and AlgorithmComplexity (12.9M) nodes are hard to reconcile with 19,660 algorithms; and the 1,536-versus-256 comparison-set discrepancy in Section 5 is unexplained. Low κ (-0.111 for paradigms) leaves two of four axes exploratory despite headline coverage.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An open-taxonomy LLM pipeline can annotate ~145M source files across 14 languages with domains, algorithms, paradigms, and design patterns at 400K GPU-hour cost.
- A three-stage procedure grounds 89-98.6% of consolidated concepts per axis into Wikidata (19,807 entities), yielding a dual local/global conceptual layer.
- A human gold set plus a six-model verifier bake-off yields a silver LLM-as-judge filter that brackets corpus-wide annotation precision between 68% and 88%.
- CodeGraph is the first large-scale open-taxonomy knowledge graph for source code, two orders of magnitude larger than GraphGen4Code (Table 1).
- The same computational primitives recur across substantively different application domains (Figure 10), enabling portfolio-style algorithm selection.
Strengths
Unprecedented scale with a genuinely released resource
145M :File nodes across 14 languages, 1.02B edges (§6); full graph on HuggingFace and pipeline code with all four prompt templates and JSON schemas released on GitHub (§4.1, Data Availability), enabling independent checking of §6 statistics.
Unusually candid reliability accounting
§5 reports Fleiss' κ = 0.153 overall, including paradigm κ = -0.111 (below chance) and pattern κ = 0.024, and explicitly demotes those axes to exploratory rather than hiding them; the 68-88% precision bracket is presented as a range, not a point estimate.
Sensible staged grounding architecture
Headline numbers are internally consistent where checkable
Weaknesses
Wikidata linking accuracy is never measured
Both use cases are unevaluated
Internal count anomalies in the derived layers
Unexplained sample-size chain in the QA protocol
Tiny gold set, no uncertainty quantification
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CodeGraph (CIKM '26, arXiv 2609.29474) is a resource-and-method paper: an LLM pipeline that annotates the Stack-Edu corpus (~167M files; 145M retained across 14 languages after excluding ~22M Markdown) with four open-taxonomy semantic axes — application domains, algorithms, paradigms, design patterns — grounds the extracted vocabulary in Wikidata, and materialises a typed property graph of 158,202,834 nodes and 1,020,887,480 edges (§6). The claims are: (1) the first open-taxonomy LLM annotation at corpus scale; (2) an innovative three-stage grounding procedure; (3) a calibrated quality-assurance protocol; (4) a KG two orders of magnitude larger than GraphGen4Code (Table 1). Given Table 1's positioning against CodeSearchNet, CodeXGLUE, CoIR, Project CodeNet, The Stack v2, and GraphGen4Code, the "first large-scale open-taxonomy code KG" claim is plausible, though "first" assertions are inherently hard to verify exhaustively.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CodeGraph?
CodeGraph is a knowledge graph built over the Stack-Edu corpus: about 145 million source-code files across 14 programming languages, annotated by a code-specialised LLM (Qwen3-Coder-30B) with application domains, algorithms, programming paradigms, and design patterns, and linked to Wikidata entities. It contains roughly 158 million nodes and 1.02 billion typed edges and is described as the first large-scale open-taxonomy knowledge graph for source code.
How accurate are CodeGraph's annotations?
The paper brackets annotation precision between 68% and 88%: 68.4% of candidate labels were judged correct by a small human gold set (127 files, 17 annotators, inter-annotator agreement Fleiss' kappa = 0.153), while an LLM verifier (Devstral-2 2512, F1 88.5% against humans) accepts about 88% of pipeline output. Paradigm and design-pattern labels showed the weakest human agreement and are released as exploratory signals.
Is CodeGraph's data and code publicly available?
Yes, partially: the pipeline source code (including the four extraction prompt templates and JSON schemas) is on GitHub (github.com/halykoss/CodeGraph), and the full graph is on HuggingFace (huggingface.co/datasets/halykoss/CodeGraph). The underlying Stack-Edu corpus requires accepting BigCode license and opt-out terms before download.
How much compute did building CodeGraph require?
Approximately 400,000 GPU-hours on 256 NVIDIA A100 GPUs on the LEONARDO supercomputer (EuroHPC/CINECA, via an ISCRA allocation), distributed across the LLM extraction stage, the Deep Research Agent grounding stage, and the corpus-wide verifier deployment — roughly 65 days of wall-clock time on the full allocation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figures 1-12) were assessed from embedded text labels and axis tick values in the extracted text only; visual layout, legends, and some per-bar details could not be inspected.
- Model naming relies on the extracted text: the 'Qwen3.6-27B' agent model is cited via a Qwen3.5 blog entry [22], so the exact verifier/agent model identity could not be cross-checked.
- Table 2's six verifier rows and Figure 7/8 percentages were read from plain text; formatting-level details (rounding, bold markers) could not be visually confirmed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.