CQ4OE: A benchmark for assessing LLM-assisted ontology generation from competency questions
Jiayi Li, Ziyuan Wang, Daniel Garijo, María Poveda-Villalón
cs.AI
Quick verdict
CQ4OE gives CQ-axiom-provenance gold standards for LLM ontology generation: an open, carefully annotated benchmark, but small and statistically untested.
Summary
CQ4OE is a benchmark for evaluating LLM-based ontology generation from competency questions (CQs). The authors build CQ-aligned gold OWL ontologies for six source ontologies (Wine, AWO, ODRL, SAREF4WATR, VGO, SWO), linking each CQ to the explicit, implicit, and derived terms and to the TBox axioms required to answer it, via a four-phase annotation pipeline with triple-review adjudication (85.6% of 118 CQs accepted unchanged). Two tasks are defined: CQ2Term (term-level, 99 CQs) and CQ2Onto (ontology-level, 118 CQs), scored across five targets — term recovery, property characteristics, domain/range triples, TBox axioms, and HermiT-based hierarchy closure — with alignment-conditioned and global views plus CQ-conditioned coverage. Nine LLMs from the DeepSeek, Qwen3.6, and Gemma families are evaluated under zero-shot, iterative, and multi-agent strategies (54 term-level runs, 162 ontologies). Key results: vocabulary recovery far outpaces structure (class F1 averaging 59.7% vs property 31.8%; global triple F1 12.4% vs 36.4% alignment-conditioned); closure F1 averages 16.7%; full axiom coverage reaches only 2.1%; multi-agent repair raises axiom coverage from 18.3% to 24.2% while hierarchies stay flat. Main limitations: every metric depends on an empirically tuned term-alignment step; gold sufficiency was judged by one team overlapping the source ontology and MASEO baseline; no significance testing; contamination from public CQs is deferred to future versions.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CQ4OE is, to the authors' knowledge, the first ontology-generation benchmark providing CQ-to-axiom provenance, with CQ-aligned gold standards over six ontologies and 118 CQs.
- LLMs recover explicit vocabulary far more reliably than they assemble correct ontology structure (class F1 averaging 59.7% vs property F1 31.8%; hierarchy-closure F1 averaging 16.7%).
- Global metrics hide requirement-localization failures that CQ-conditioned coverage exposes (e.g., Water: 70.6% term F1 but 0% full coverage; Wine: 100% at-least-one but 0% full coverage).
- Generation strategy matters less than domain: structural F1 varies by under 3 points across strategies, and multi-agent repair raises axiom coverage from 18.3% to 24.2% while leaving hierarchies shallow.
Strengths
First benchmark with CQ-to-axiom provenance
Section 3.2 links each CQ to explicit/implicit/derived term sets and a sufficient axiom set (an axiom is retained only if its removal makes the CQ unanswerable), directly fixing the reference-ontology/CQ misalignment documented in prior analysis; 85.6% of 118 CQs passed triple-review unchanged, and the 17 revisions track ontology complexity (10 of 17 in SWO, none in Wine/AWO).
Two-view, multi-target evaluation design that exposes failures single scores hide
The alignment-conditioned vs global split plus CQ-conditioned coverage (Table 2, Section 4.3) separates vocabulary misalignment from structural error: Triple-AC 36.4% vs Triple-G 12.4% overall; VGO/ODRL drop from 70.0% to 25.8% and 59.7% to 19.1%; Water reaches 70.6% term F1 with 0% full coverage and Wine 100% at-least-one with 0% full coverage.
Unusually complete reproducibility package
Differentiated empirical findings beyond a ranking
Weaknesses
Term alignment is a single point of failure for all ontology-level metrics
No statistical uncertainty anywhere
Potential circularity between gold standard, baseline, and evaluators
Scale and contamination exposure
Task CQ sets differ, complicating cross-task claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CQ4OE (arXiv:2609.26029) is a benchmark paper from the Ontology Engineering Group (UPM) targeting LLM-based ontology generation from competency questions (CQs). It claims four contributions: CQ-aligned gold standards with per-CQ term and axiom provenance, a multi-target evaluation framework, an automated explainable reporting pipeline, and nine-model baselines. The core empirical claims are that vocabulary recovery far outpaces structural adequacy, that global scores mask requirement-localization failures, and that generation strategy matters less than domain. The gap analysis in Section 2 is credible: OAEI, BioASQ, OntoAxiom, CORAL, and Plu et al. each target different tasks, and the claim that no prior resource offers CQ-to-axiom provenance is plausible, though it should be re-verified against very recent concurrent work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CQ4OE?
CQ4OE is an open benchmark (available on GitHub, Zenodo, and HuggingFace) for evaluating LLMs that generate OWL ontologies from competency questions. Its gold standards link every class, property, and TBox axiom to the specific competency question that requires it, and it ships an automated evaluation pipeline with five metric targets, per-run Markdown reports with CSV traces, and a public leaderboard.
What did the CQ4OE experiments find?
Nine LLMs from the DeepSeek, Qwen3.6, and Gemma families recover vocabulary much better than structure: average class F1 was about 59.7% versus 31.8% for properties, hierarchy-closure F1 averaged 16.7%, and only about 2.1% of competency questions received all their required axioms. Zero-shot, iterative, and multi-agent strategies performed within 3 points of each other on structural metrics.
Is CQ4OE affected by training-data contamination?
Potentially yes: all six source ontologies and their published competency questions are public and likely appear in LLM pretraining corpora. The authors argue the impact is limited because mean structural F1 stays low (26.7-33.7%) and the provenance annotations are newly created, but fully contamination-controlled ontologies are deferred to future versions of the benchmark.
How large is the CQ4OE benchmark?
It covers six ontologies (Wine, AWO, ODRL, SAREF4WATR, VGO, SWO) spanning three size tiers. From 255 published competency questions, 110 were retained and 8 newly authored, yielding 118 CQs for the ontology-level task (CQ2Onto) and 99 for the term-level task (CQ2Term), each annotated with term and axiom provenance by a triple-review protocol.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-3 were extracted as numeric text dumps; values are readable but visual layout and per-cell model attribution in Figure 3(a) could not be fully verified against axis labels.
- The GitHub/Zenodo/HuggingFace resources are claimed in the Resource Availability Statement but could not be inspected from the text; the high reproducibility rating assumes the stated contents exist as described.
- This is an arXiv v1 preprint (September 2026); no peer-review verdict is available.
- Number citations rely on the provided text extraction, which may contain minor OCR artifacts.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.