From Terminology to Diagrams: Visual-Instruction Generation for Scientific Diagram Understanding
Raul Ortega, José Manuel Gómez-Pérez
cs.CV · cs.AI · cs.CL
Quick verdict
Terminology-driven pipeline builds 194K-diagram, 1.4M-instruction dataset, SOTA on diagram QA; gains credible, but contamination and noise risks unquantified.
Summary
The authors introduce a terminology-driven framework for building visual-instruction data for scientific diagrams and use it to construct SciGram: 194,071 web-retrieved diagrams paired with 1.4M synthetic instructions across three subsets — SciGram-Align (582,213 caption pairs), SciGram-VIT (737,887 MCQs), and SciGram-M3 (47,506 curated from TQA/SQA/AI2D/ARC/OpenBookQA train sets). The pipeline filters 4,820 curriculum terms via a weirdness index plus embedding clustering, generates 5.5M atomic facts with LLaMA3-8B, retrieves diagrams from DuckDuckGo, and synthesizes captions and MCQs with Qwen2-VL-7B. Fine-tuning LLaVA on SciGram yields LLaVA-SciGram OV, which sets new best results on TQA diagram questions (80.12 DMC), SQA visual-support questions (97.62), and AI2D opaque labels (83.45), while training on far fewer instructions than LLaVA OV's 7.8M mixture. Ablations (Tables 5-6) show every subset contributes, and a 200-question probe suggests gains concentrate on visual-support questions (~10 points) rather than pure language priors. Limitations: the dataset deliberately trades precision for coverage (light filtering, residual noise acknowledged; a 600-item human audit finds only moderate agreement, Gwet AC1 0.59); benchmark deltas lack significance testing; web-retrieved diagrams may overlap benchmark test sets; and baseline comparisons mix own-prompt and literature-reported numbers (Appendix H). Dataset and models are released; code release is not explicitly stated in the text.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A terminology-driven pipeline (curriculum terms to atomic facts to web diagram retrieval to synthetic captions/MCQs) can produce 1.4M usable visual instructions grounded in 194K web diagrams.
- LLaVA-SciGram OV establishes new best results on diagram-centric QA: 80.12 on TQA diagram MCQs, 97.62 on SQA visual-support questions, and 83.45 on AI2D opaque labels.
- These results are achieved with far fewer instructions (1.4M) than LLaVA OneVision's 7.8M mixture, and ablations show each SciGram subset contributes.
- The gains reflect improved visual grounding rather than language priors alone, per a 200-question probe showing ~10-point gains concentrated on visual-support questions.
- Coverage-over-precision web data with synthetic labels is an effective supervision signal despite residual noise.
Strengths
Well-motivated, fully documented generation pipeline
Section 3 traces every stage with numbers: weirdness-index filtering (t=2 vs BNC) plus RoBERTa clustering yields 4,820 terms; LLaMA3-8B produces 5,508,218 atomic facts; DuckDuckGo retrieval with a 'diagram' suffix and light filtering (images with ≥5 supporting facts, perceptual-hash dedup) yields 255,657 images; Qwen2-VL-7B captions (3x per diagram, Levenshtein 0.4196 diversity) and MCQs (5.7% duplicates removed). Anchoring retrieval in curriculum concepts rather than free web crawl is the paper's core and defensible idea.
Ablations actually isolate each component
Table 5 compares each SciGram stage against its LLaVA OV counterpart (alignment: AI2D 59.07 vs BLIP's 45.43; mid-stage: SQA IMG 81.91 vs 74.57; final: SQA IMG 95.24 vs 87.31 with 48K vs 3.2M instructions), and Table 6 tests all seven subset combinations, with the full pipeline best (80.12/97.62/83.45) and a no-text-only variant (SciGram-M3*) included.
Statistically literate treatment of annotator agreement
Grounding analysis goes beyond benchmark scores
Release and evaluation transparency
Weaknesses
No uncertainty quantification on decisive margins
Benchmark-contamination risk unaddressed in the provided text
Deliberate noise acceptance with only moderate audit signal
Mixed evaluation provenance across baselines
Unreconciled pipeline quantities
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This COLM 2026 paper (arXiv 2609.00948, Ortega & Gomez-Perez, Expert.ai) proposes a terminology-driven framework that converts middle-school science curricula into diagram-grounded visual-instruction data, and releases **SciGram**: 194,071 web-retrieved diagrams with 1.4M synthetic instructions in three subsets - SciGram-Align (582,213 caption pairs), SciGram-VIT (737,887 MCQs), and SciGram-M3 (47,506 items from the TQA/SQA/AI2D/ARC/OpenBookQA train sets, answer choices shuffled). Central claims: (1) the pipeline is a general, effective strategy for diagram supervision despite noisy web data and synthetic labels; (2) LLaVA-SciGram OV reaches 80.12 on TQA diagram MCQs, 97.62 on SQA visual-support questions, and 83.45 on AI2D opaque labels - new bests among compared systems; (3) this uses far fewer instructions than LLaVA OV's 7.8M mixture; (4) a 200-question probe attributes gains to visual grounding rather than language priors.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SciGram dataset?
SciGram is a dataset of 194,071 scientific diagrams collected from the web, paired with about 1.4 million synthetic visual instructions (image captions and multiple-choice questions) covering life, earth, and physical sciences at middle-school level. It comes in three subsets (Align for vision-language alignment, VIT for instruction tuning, M3 curated from existing QA datasets) and is released by the authors alongside fine-tuned LLaVA models.
How much does fine-tuning on SciGram improve diagram QA performance?
Per the paper, LLaVA-SciGram OneVision 7B reaches 80.12% on TQA diagram multiple-choice questions, 97.62% on ScienceQA questions with visual support, and 83.45% on AI2D with opaque labels - each a new best among the compared published results - while training on about 1.4M instructions versus the 7.8M in LLaVA OneVision's mixture.
Is the synthetic data in SciGram reliable?
It is intentionally coverage-over-precision: captions and questions are generated by Qwen2-VL-7B with light filtering. A 600-item audit by four experts found moderate agreement (Gwet AC1 0.59) and the authors explicitly accept residual noise as a trade-off, so individual labels (especially MCQ answer keys) should not be treated as gold-standard.
Are the SciGram models and code open source?
The paper states that the SciGram dataset and the LLaVA-SciGram models are released for research use. An explicit code release is not stated in the reviewed text, though prompts, hyperparameters, and evaluation details are documented in the appendices.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked gaps; content inside gaps (including part of Section 5/experimental setup, the start of Section 6, and the remainder of the ethics statement, which cuts off mid-sentence) was not visible to this review.
- Appendix F quality results and several headline figures (Figures 3, 4, 10-19) are images with no extractable numeric rates, so per-dimension approval percentages and the grounding-probe per-type bars could not be verified from text.
- Some table rows (notably mid-stage rows of Table 5) are compressed or misaligned by PDF extraction; only cells that parse cleanly against their column headers were cited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.