When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning
Hamed Babaei Giglou, Sören Auer, Jennifer D'Souza
cs.AI
Quick verdict
Controlled 13-model study: LLM scale mainly raises precision in OL, non-monotonically; sound design but tiny test sets and no significance tests limit claims.
Summary
This paper presents a controlled benchmark of how LLM parameter scale affects ontology learning (OL). Using the OntoLearner RAG pipeline — fixed embedding model (Qwen3-Embedding-4B), top-10 retrieval, identical prompts and decoding — 13 models are evaluated on term typing, taxonomy discovery, and non-taxonomic relation extraction over four ontologies (MatWerk, MDS, OBI, MFOEM) from materials science and biomedicine. Models span a Qwen3.5 dense sweep (0.8B–27B), Qwen3.5 MoE variants (35B-A3B, 122B-A10B), matched Qwen3.6 releases, and proprietary GPT-5.5/5.6 variants. Within the dense lineage, scale mainly raises precision rather than recall, with the largest jump between 9B and 27B (e.g., MatWerk term typing F1 36.13→58.14). Effects are non-monotonic: dense 27B beats the 122B MoE on term typing, while MoE leads on taxonomy discovery; Qwen3.6-27B regresses versus Qwen3.5-27B on MatWerk. GPT-5.5 generally beats the newer GPT-5.6 variants, whose Luna shows extreme conservatism (100% precision, 5.26% recall on MFOEM). Non-taxonomic extraction is scale-resistant, especially on MDS (≤1.27 F1 open-weight, 4.12 best GPT). Main limitations: tiny evaluation sets (as few as ~19 items per task), no significance testing or variance estimates, undisclosed decoding parameters, and an incomplete task×domain matrix.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Within the Qwen3.5 dense lineage (0.8B–27B), increasing parameter count improves precision rather than recall, with the largest gains between 9B and 27B parameters.
- The effect of scale is non-monotonic and task-dependent: dense 27B models beat the much larger 122B MoE on term typing, while MoE models lead on taxonomy discovery.
- Architecture and model lineage can outweigh nominal parameter count, shown by the Qwen3.6-27B regression on MatWerk term typing and GPT-5.5 outperforming the newer GPT-5.6 variants.
- Non-taxonomic relation extraction resists scaling, staying below ~20 F1 even for frontier models and near zero on the abstract MDS ontology (best 4.12 F1).
Strengths
Genuinely controlled design that isolates scale
Same embedding model (Qwen3-Embedding-4B), top-10 retrieval, prompt templates, and decoding across all 13 models (Section 4); the within-family Qwen3.5 dense sweep (0.8B–27B) plus matched Qwen3.5/3.6 27B-dense and 35B-A3B pairs is the cleanest scale manipulation in the OL literature, directly answering the gap documented in Table 1.
Precision/recall decomposition yields a mechanistic finding
Small Qwen3.5 models (<9B) accept nearly every candidate — recall 90.57–100% with precision 9.06–25.33% on term typing (Section 6) — reframing small-model failure as decision-boundary calibration rather than missing knowledge, which is actionable for practitioners.
Transparent reporting of unfavorable anomalies
Strong reproducibility infrastructure
Weaknesses
No uncertainty quantification on very small test sets
Incomplete task×domain matrix and unexplained subsampling
Single fixed retriever with no ablation
Key experimental parameters not specified in-paper
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This workshop paper (WOP @ ISWC'26) asks whether nominal LLM parameter scale reliably predicts ontology-learning (OL) quality — a variable the prior LLMs4OL line never isolated, as Table 1 documents study-by-study. Three research questions structure the work: within-family scaling (RQ1), task- and domain-sensitivity (RQ2), and architecture/lineage effects (RQ3). The headline claims: (1) inside the Qwen3.5 dense lineage, added parameters raise precision far more than recall, with the largest step between 9B and 27B; (2) scale effects are non-monotonic across tasks — dense 27B beats a 122B-total MoE on term typing while MoE wins taxonomy discovery; (3) lineage and alignment choices can outweigh size (Qwen3.6-27B regresses on MatWerk term typing; GPT-5.5 beats the newer GPT-5.6 variants); (4) non-taxonomic relation extraction is scale-resistant, worst on the abstract MDS ontology. The claims are appropriately hedged in the text, and the controlled design gives them more force than prior family-confounded comparisons.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does a bigger LLM always perform better at ontology learning?
No. In this controlled study of 13 models, scaling within Qwen3.5's dense family mainly improved precision rather than recall (largest gains between 9B and 27B), a dense 27B model beat a 122B-parameter MoE on term typing while the MoE won at taxonomy discovery, and non-taxonomic relation extraction barely improved at any scale — especially on the abstract Materials Data Science ontology.
What tasks and datasets were used in the study?
Three ontology-learning tasks — term typing, taxonomy discovery, and non-taxonomic relation extraction — on four ontologies: MatWerk and MDS (materials science) and OBI and MFOEM (biomedical). All 13 models ran through the same RAG pipeline with the same embedding model, top-10 retrieval, prompts, and decoding settings.
Which model performed best overall?
Proprietary GPT-5.5 achieved the top scores (e.g., 76.19 F1 on MatWerk term typing) and generally beat the newer GPT-5.6 variants (Luna, Terra, Sol). Among open-weight models, Qwen3.5-27B was strongest on term typing and Qwen3.5-122B-A10B on taxonomy discovery.
Is the code and data available?
Yes. The study uses the open-source, MIT-licensed OntoLearner library (GitHub and readthedocs), with benchmark collections on Hugging Face and archived experimental resources on Zenodo (DOI 10.5281/zenodo.21719027).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 3's per-task item counts are garbled in the PDF extraction; item counts cited in this review were derived from recall granularity (e.g., 5.26% = 1/19) rather than read directly from the table.
- Figure 1 appears only as extracted axis labels; the plotted curves could not be inspected, so statements about the figure rely on the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.