Fine-Grained Multi Image Object Hallucination Benchmark
Joonki Min, Chaeyun Kim, Hyungwook Choi, Yejin Kim, Kihyun Kim, Yohan Jo, Joonseok Lee
cs.CV · cs.AI · cs.LG
Quick verdict
MIOH probes multi-image object hallucination (4 tasks, 3 patterns, 29 models); sound design and findings, but no significance tests or release statement.
Summary
The paper introduces MIOH, a benchmark for object hallucination in multi-image settings for multimodal LLMs. It crosses four object-centric tasks (existence, counting, attribute, position) with three reasoning patterns (comprehensive, comparative, selective) into 26 question types, and adds three controlled adversarial pressures: number of images, perceptual difficulty (small/occluded objects), and contextual bias (co-occurrence priors). Questions are generated from COCO-ReM, PACO, and SVG, validated by three annotators, yielding 3,484 questions over 11,732 images. Evaluating 29 models, overall accuracy averages 36.1%; Gemini-2.5-Pro (64.4%) and GPT-5 (63.1%) lead, while selection reasoning (34.2%) and counting (25.4%) are weakest, and raising the image count to eight collapses existence accuracy to 30.0%. An ablation comparing joint multi-image questions with decomposed single-image questions shows multi-image contexts amplify hallucination beyond error accumulation, which the authors attribute to integration-stage failures. Main limitations: no significance tests, confidence intervals, or reported correlation coefficients behind the accuracy–robustness claims; varying MCQ chance levels across question types; and no explicit public release statement for the benchmark.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing object-hallucination benchmarks cannot diagnose how visual complexity and reasoning demands trigger hallucination in multi-image settings.
- MIOH enables fine-grained diagnosis by crossing 4 object-centric tasks with 3 reasoning patterns (26 question types) under 3 controlled adversarial pressures.
- Even frontier models degrade sharply: GPT-5 reaches 63.1% and Gemini-2.5-Pro 64.4% overall (29-model average 36.1%), with selection reasoning (34.2%) and counting (25.4%) weakest and Existence collapsing to 30.0% at 8 images.
- Multi-image contexts amplify hallucination beyond single-image error accumulation, indicating integration-stage rather than purely perceptual failure (Fig. 5 ablation).
- High accuracy on easy questions does not guarantee adversarial robustness; open-source leaders Qwen2-VL-7B and Qwen2.5-VL-7B lose 27.2% and 21.8% under pressure.
Strengths
Factorial diagnostic design goes beyond binary POPE-style probing
Sec. 3 and Table 1 define 26 question types crossing 4 object-centric tasks with 3 reasoning patterns; Sec. 4.2 shows this matters empirically (Selection averaging 34.2% vs. high Existence-Comprehensive scores), a distinction single-format benchmarks cannot make.
Annotation-quality-aware data sourcing
Sec. A.1 builds on COCO-ReM, PACO, and SVG instead of raw COCO, explicitly citing RePOPE's evidence that label errors distort hallucination scores, and caps counting at ≤5 objects per image after observing COCO-ReM degradation above ~10 instances.
Human validation pipeline is documented
Controlled ablation isolating integration from perception
Broad, current model coverage
Weaknesses
No inferential statistics anywhere
Uncontrolled MCQ chance levels
Vague difficulty-filter thresholds and possible circularity
No stated code/data release
Unexplained gaps and unclear handling of single-image models
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Structured Preliminary Review: MIOH (arXiv 2608.30653)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the MIOH benchmark?
MIOH (Multi-Image Object Hallucination benchmark) is an evaluation suite for testing whether multimodal large language models hallucinate objects when reasoning over several images at once. It combines four tasks (existence, counting, attribute, position) with three reasoning patterns (comprehensive, comparative, selective) into 26 question types, and adds controlled pressures: more images, harder-to-see objects, and contextually misleading scenes. The final set has 3,484 multiple-choice questions built from 11,732 images.
How well do GPT-5 and Gemini-2.5-Pro handle multi-image object hallucination?
On MIOH they are the best performers but far from perfect: Gemini-2.5-Pro scores 64.4% overall and GPT-5 63.1%, against a 29-model average of just 36.1%. Both drop substantially when the image count rises to eight (Existence accuracy 55.3% and 56.4% respectively, versus an easy baseline of 62.4% averaged over models).
Why do multimodal LLMs hallucinate more with multiple images?
The paper's ablation suggests it is not just weaker perception: asking one joint question across images consistently yields worse results than asking separate per-image questions and combining answers. The authors interpret this as an integration-stage failure — models struggle to maintain and combine object representations across images, not merely to see them.
Is the MIOH benchmark publicly available?
The reviewed paper text does not state a repository, license, or release commitment for the questions or evaluation code; this is flagged as a reproducibility gap, and the construction is documented in enough detail (COCO-ReM, PACO, SVG sources, filtering rules, stratified sampling) that release should be straightforward for the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present in the extraction only as captions and embedded axis/tick labels; visual details of Fig. 4 (accuracy–robustness scatter) and Fig. 5 (per-model ablation bars) could not be inspected beyond their caption text.
- A few Table 2 row labels appear compressed by extraction (e.g., a bare 'InternVL3.5-8B' row alongside its Instruct/Pretrained/MPO variants); variant identities were inferred from surrounding rows.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.