ArticleMiner: Ontology-Guided Knowledge Graph Construction from Scientific Publications
Md Abrar Jahin, Craig A. Knoblock, Jay Pujara
cs.AI
Quick verdict
Ontology-module framework for paper-to-KG extraction beats same-LLM few-shot on 4 tasks, but headline gains are confounded by extra evidence access; candid.
Summary
ArticleMiner is a framework that converts scientific publications and their supplementary files into knowledge-graph-ready records. Its core design separates a shared five-stage orchestration (parse, extract, self-correct, reconcile, serialize) from a bounded, human-authored per-task ontology module — a six-tuple specifying vocabulary, canonicalization, derivation rules, executable validators, an identity key, and RDF bindings. Four modules are instantiated (geochemistry, drug-discovery chemistry, materials science, machine learning) and evaluated on 163 papers, including a new USGS-expert-curated geochemistry benchmark (28 papers, 87,759 measurements). Against same-LLM few-shot baselines across five closed backbones, point estimates favor ArticleMiner on all four tasks: Sonnet 4.6 reaches 79.3 vs 64.5 tuple-F1 on DiSCoMaT, and the GeoChem paired subset shows +62.3 sample-F1. Component ablations attribute 16.3 F1 points to ontology grounding on DiSCoMaT (Haiku). However, the paper is candid that the GeoChem headline gap is confounded — ArticleMiner reads supplementary spreadsheets the baseline never sees — and that paired summaries lack complete run provenance, so no formal significance is claimed. GeoChem ablation rows are uninterpretable (identical 51.0 F1 across all removals), ChemTables/MLTables have only 9/15 papers with very wide confidence intervals, and downstream RDF/query quality is never evaluated. Code, modules, and ground truth are released under CC-BY-4.0.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Applying bounded, human-authored ontology modules during extraction, normalization, and validation — rather than only after table recovery — improves publication-to-KG extraction under a fixed LLM backbone, with task-dependent gains (RQ1).
- A six-component task-module interface (vocabulary, canonicalizer, derivations, validators, identity key, RDF bindings) over one shared orchestration enables cross-domain reuse, demonstrated on four domains without OWL reasoning or SHACL execution.
- Multi-backend evidence fusion is a per-task choice: it helps on supplementary-heavy GeoChem (54.3 vs 46–48 F1) but hurts on PDF-only ChemTables (20.8 vs 26.2) and MLTables (47.6 vs 56.3).
- On GeoChem, ArticleMiner reaches composite S≈76 and sample-F1 60.8 vs ≤4.1 for same-LLM few-shot, but the paper concedes the gap partly reflects supplementary-file access rather than domain guidance alone.
Strengths
Exceptional transparency about confounds and weak statistics
The paper repeatedly disarms its own headline numbers: §4.1 states the GeoChem comparison 'includes access to supplementary spreadsheets and is not an isolated test of ontology grounding'; §6 admits paired run provenance is incomplete so 'those summaries are descriptive and do not establish formal statistical significance'; App. I concedes MinerU lacked OCR weights. This candor is rare and raises trust in the reported point estimates.
New expert-curated GeoChem benchmark is a real community asset
§3.1 and App. K document 28 peer-reviewed papers with USGS-coordinated ground truth on a 209-column schema: 1,236 unique sample identifiers, 87,759 non-null measurements, explicit censoring conventions (−L vs −99999 vs blank), reference-material handling, and CC-BY-4.0 release — the largest benchmark in the paper by observation count.
Clean, auditable module interface with precisely scoped semantics
Unusually broad empirical grid with honest negative results
Weaknesses
Flagship GeoChem comparison is not a controlled test of the core claim
GeoChem component ablations are non-informative
Statistical support for headline deltas is thin
Evaluator design has circularity and leniency channels
Knowledge-graph claims outrun the evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: ArticleMiner (ISWC 2026 workshop)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ArticleMiner?
A framework from USC ISI that extracts knowledge-graph-ready records from scientific papers and their supplementary files. Instead of one universal system, a shared pipeline (PDF/spreadsheet parsing, LLM extraction with vocabulary grounding, retries, reconciliation, JSON-LD output) is reused across domains, while each task supplies a small human-written 'ontology module' defining allowed vocabulary, normalization rules, validity checks, identity keys, and RDF mappings.
How much better is ArticleMiner than just prompting an LLM?
On the same underlying model, it beats few-shot prompting on all four tested domains — e.g., Sonnet 4.6 scores 79.3 vs 64.5 tuple-F1 on materials-science tables (DiSCoMaT), with gains of roughly +8.8 to +15.5 F1 on the other external tasks. On the new geochemistry benchmark the sample-F1 gap is huge (60.8 vs 0.8–4.1), but that comparison is confounded because only ArticleMiner reads supplementary spreadsheets; the authors themselves state the gain cannot be attributed to domain guidance alone, and no formal significance is claimed.
Is the code and data available?
Yes. Source code, the four ontology modules, prompts, output schemas, model outputs for eight backbones, ablation grids, and the new expert-curated geochemistry ground truth are released under CC-BY-4.0 at github.com/Abrar2652/articleminer-iswc26. Source-paper PDFs retain their original licenses and are not redistributable; the paper also notes there is no complete manifest linking every historical result to its configuration.
What are the main limitations of the paper?
Two benchmarks have only 9 and 15 papers with very wide confidence intervals; there are no significance tests and paired runs are incompletely documented; the geochemistry headline gain is confounded by supplementary-file access and its component ablations are not cleanly controlled; and knowledge-graph outputs (Turtle, SPARQL query quality, cross-paper entity resolution) were not evaluated — the reported evidence stops at tuple- and sample-level extraction scores.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 were extracted as text/schematics; visual rendering and exact plot values beyond labeled numbers could not be verified.
- The text appears complete through Appendix K, but appendix tables and prompt listings may have lost formatting in extraction.
- This review relies on the paper's self-reported numbers; the released code and outputs were not executed to verify Tables 4–5.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.