Vroom-Vroom at SHROOM-Visions: A Multi-Judge Committee for Detecting Hallucinated Spans in Vision-Language Outputs
Toqeer Ehsan, Nico Penttilä, Richard Schmidt, Arash Hajikhani, Victoria Palacin
cs.CL · cs.AI
Quick verdict
Five-model VLM committee wins 3 of 4 SHROOM-Visions languages; strong aggregation design, but vote threshold tuned on test split and no significance tests.
Summary
This shared-task paper presents the authors' SHROOM-Visions 2026 system for detecting hallucinated character spans in vision-language model outputs in English, French, Italian, and Chinese. Exploratory analysis of the SHEEP benchmark highlights severe annotator disagreement: 80-86% of hallucinated characters are flagged by only one of three annotators. The main system LoRA-fine-tunes four vision-language judges (Gemma-4, Mistral-Small, Qwen3-VL, Qwen3.6) on distinct data subsets and adds a few-shot Gemma-4 judge; all predicted spans are projected onto characters, characters flagged by at least two judges are retained with averaged confidences, and adjacent runs are merged back into spans. A parallel track trains activation probes on Qwen3-VL hidden states, but they trail the committee. On the challenge set, the votes>=2 committee ranks first in French, Italian, and Chinese and second in English (average Cor 0.5730, IoU 0.4943), beating every individual judge on correlation metrics and reducing MAEp via finer probabilities. Judge flags rise monotonically with annotator counts (Spearman 0.39). Key limitations: the vote threshold was tuned on the labeled test split, no significance testing is reported, English scores are weakest, and the probes contribute little despite extensive development.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A five-judge VLM committee aggregated by character-level voting (at least 2 votes) ranks first in French, Italian, and Chinese and second in English on the SHROOM-Visions challenge set.
- The committee outperforms every individual judge on Cor and IoU (sole exception: English Cor_lbl), because aggregation recovers recall lost by fine-tuned judges while suppressing single-judge false positives.
- Disagreement among diverse models tracks human annotator disagreement: mean annotator count rises monotonically from 0.109 at zero votes to 2.072 at five votes (Spearman 0.39 character-level, 0.51 record-level).
- Averaging committee confidences yields finer probabilities than any single judge's three-class outputs, reducing MAEp to 0.305-0.336 versus up to 0.475 for individual judges.
- Activation probes can detect hallucinations from VLM hidden states but trail the committee on correlation metrics, and their performance depends strongly on the underlying model.
Strengths
Principled character-level aggregation
Section 3.3 projects heterogeneous judge spans onto characters, sums votes, and averages confidences divided by N (preserving magnitude rather than normalizing by vote count), then merges maximal same-label runs - cleanly handling span-boundary mismatch; votes>=2 yields the best Cor and IoU in all four languages on the test split (Table 1).
Committee demonstrably beats its own members
Tables 1-2 show the >=2-vote committee leading every judge on Cor and IoU across languages (sole exception: English Cor_lbl vs Qwen3-vl-ft-3shot, 0.3872 vs 0.3904); Table 3 adds the best Cor on every hallucination label and the lowest MAEp (0.305-0.336 vs up to 0.475), attributed to averaging out the judges' coarse three-class confidences.
Analysis grounded in annotation ambiguity
Strong reproducibility posture
Weaknesses
Aggregation tuned on evaluation data
No statistical rigor
Probe track underdelivers relative to its narrative weight
Committee composition is under-justified
Weakest-language diagnosis is thin
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SHROOM-Visions shared task and what did this paper's system achieve?
SHROOM-Visions is a 2026 shared task on detecting hallucinated character spans in vision-language model outputs across English, French, Italian, and Chinese, scored by character overlap and confidence correlation against human annotations. This paper's five-model voting committee ranked first in French, Italian, and Chinese and second in English, placing on the podium in every language and metric.
How does the multi-judge committee detect hallucinations?
Five vision-language models (four LoRA-fine-tuned, one few-shot) each mark unsupported spans with labels and coarse confidence values of about 0.33, 0.67, or 1.0. All spans are projected onto characters; a character is kept if at least two judges flag it, its probability is the average of the flagging judges' confidences, and adjacent kept characters are merged back into spans.
Did activation probes work better than the model ensemble?
No. Probes reading Qwen3-VL hidden states detect hallucination above chance and match individual judges on IoU, but score below the judge committee on the correlation metrics and did not improve the ensemble when added. The authors conclude probe performance depends strongly on the base model and may exploit surface features rather than mechanistic hallucination signals.
Is code or model access available for this paper?
Yes. The committee code is on GitHub (toqeerehsan/vlm_hallucination_detect), fine-tuned checkpoints are on Hugging Face under QSTS-VTT, and prompts plus LoRA and probe hyperparameters appear in the paper's appendices. The SHEEP dataset itself is distributed by the task organizers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 illustrate annotated examples whose span highlights are not recoverable from the extracted text, so the qualitative claims about annotation inconsistency (Appendices A-B) rest only on the prose.
- Official leaderboard or competitor scores are not included in the text, so the claimed first-place rankings in three languages cannot be independently verified from the paper alone.
- Table 3's label-wise blocks were extracted as flattened text; label-to-number alignment was inferred and individual cells may be misread.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.