CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
Jiacheng Cai, Zijin Hong, Zheng Yuan, Huachi Zhou, Qinggang Zhang, Xiao Huang
cs.CV · cs.AI
Quick verdict
Five-path cross-modal RAG benchmark (1,863 QA, 3.5 avg hops) with traceable KG-guided construction, but only two RAG baselines and LLM-only validation.
Summary
The paper introduces CrossModalQA, an open-domain benchmark for multimodal retrieval-augmented generation (RAG). It contains 1,863 text-only QA pairs built from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images, organized into five reasoning-path categories: vision-to-text, text-to-vision, vision-to-text-to-vision, multi-image intersection, and image-set reasoning, with an average depth of 3.50 hops. Construction uses a multimodal knowledge graph: articles are selected from visually rich domains under quotas, triples are extracted with GPT-5.1, entities are linked to Wikidata and grounded with verified images, and questions are generated from sampled subgraphs by GPT-5.4, then filtered by rule-based checks and LLM verification. Experiments with two multimodal RAG baselines (M2RAG and adapted mKG-RAG variants), all using GPT-4o as generator, show every RAG setting underperforms closed-book GPT-4o (M2RAG 14.2 EM vs 24.3 overall). Oracle experiments with gold text and images raise GPT-4o to 66.1 EM, indicating retrieval completeness contributes more than generator scaling; image-set questions remain hardest (29.2 EM even with full gold evidence). Main limitations: only two RAG baselines are evaluated, quality control relies on LLMs without reported human validation, and the roughly 10k-item corpus strains the 'open-domain' framing.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CrossModalQA is the first multimodal RAG benchmark covering all five cross-modal reasoning-path categories (V-to-T, T-to-V, V-to-T-to-V, multi-image intersection, image-set).
- Every question requires composing distributed textual and visual evidence, averaging 3.50 reasoning hops with over 40% of questions at four or more hops.
- Evaluated multimodal RAG systems underperform closed-book GPT-4o because incomplete retrieval omits necessary evidence and introduces distracting context.
- Retrieval completeness contributes more to end-to-end accuracy than generator scaling, and multi-image retrieval plus joint multi-image interpretation are the primary bottlenecks.
Strengths
Complete reasoning-path taxonomy
Table 1 shows prior multimodal RAG benchmarks cover at most three of five modality-transition categories, while CrossModalQA instantiates all five with balanced per-mode counts (366/383/411/391/312), enabling per-path diagnosis no prior benchmark supports.
Auditable, controlled construction
Algorithm 1 specifies mode-specific edge/image/hop budgets for subgraph sampling, and Figs. 6-7 reproduce the generation prompts; each question retains its supporting subgraph, making hop structure (avg 3.50, Fig. 3) and evidence provenance explicit and checkable.
Diagnostically valuable negative result
Dual-generator oracle ablations
Precise metric formalization
Weaknesses
Thin baseline coverage
LLM-only quality control
Open-domain framing vs corpus scale
Hop-count definition may inflate depth
No statistical testing or release statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces CrossModalQA, a benchmark for multimodal retrieval-augmented generation (RAG): 1,863 text-only QA pairs over 4,987 Wikipedia articles and 4,431 Wikimedia Commons images, organized into five reasoning-path categories — V-to-T (n=366), T-to-V (n=383), V-to-T-to-V (n=411), multi-image intersection (n=391), and image-set (n=312). Its central claims: (1) it is the first benchmark covering all five modality-transition paths (Table 1); (2) questions average 3.50 hops with >40% at four or more (Fig. 2); (3) evaluated multimodal RAG systems underperform closed-book GPT-4o because incomplete retrieval injects distractors; (4) retrieval completeness matters more than generator scaling, with multi-image reasoning the chief bottleneck. Claims (1)-(3) are substantively supported by the experiments; the strength of (3)'s generalization is limited by baseline breadth.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CrossModalQA?
CrossModalQA is a benchmark of 1,863 text-only question-answer pairs built from 4,987 Wikipedia articles and 4,431 Wikimedia Commons images. It tests whether multimodal retrieval-augmented generation (RAG) systems can discover and combine text and image evidence across five reasoning-path types (vision-to-text, text-to-vision, vision-text-vision, multi-image intersection, image-set), with an average depth of 3.5 reasoning hops.
Why do the RAG baselines score worse than GPT-4o with no retrieval at all?
Because retrieval is incomplete. Baselines like M2RAG retrieve some relevant items (71.0 Hit@5) but miss much of the required evidence (only 32.2 Recall@5), and the extra irrelevant context misleads the answer generator — M2RAG scores 14.2 EM overall versus 24.3 for closed-book GPT-4o.
How were the benchmark questions created?
A knowledge graph was extracted from Wikipedia articles with GPT-5.1 and grounded in verified Wikimedia Commons images via Wikidata; connected subgraphs were sampled under per-question-mode budgets, GPT-5.4 generated questions from those subgraphs, and rule-based checks plus LLM verification filtered out questions that did not match the intended reasoning path or require multimodal evidence.
Is the CrossModalQA dataset publicly available?
The reviewed paper text (through Appendix D) does not state a download link, code repository, or license. The underlying sources (Wikipedia, Wikidata, Wikimedia Commons) are public, and the appendix documents the sampling algorithm and generation prompts, but availability of the final dataset must be confirmed with the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2, 4 and 5 were extracted as raw number streams; cell-level attributions in those heatmaps rely on the stated row/column ordering, so individual heatmap cells (not the aggregate statistics cited in text) carry minor alignment risk.
- No code or data release statement appears anywhere in the extracted text (through Appendix D, page 20); availability may be announced on a project page not captured in the PDF text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.