Two-Token Features and Small-Large Ensembles for VLM Hallucination Detection
cs.CL
Quick verdict
Solid SHROOM-Visions 2026 system report: a 4B two-token probe ensembled with a 400B judge ranks 6th-8th across four languages; candid but single-seed.
Summary
This shared-task paper describes team eye-be-am's system for SHROOM-Visions 2026, which asks for character-level detection and five-way categorisation of hallucinated spans in vision-language-model outputs across English, French, Italian and Chinese. The system fine-tunes a 4B Qwen3.5-VL whose classifier head concatenates hidden states of the current and previous token at mid-network layer 16 of 32, selected by a validation sweep worth +0.039 Cor over the embedding layer; both this detector and a ~397B zero-shot VLM judge receive PaddleOCR text injected into their prompts. Synthetic training data (~67,500 rows generated by the large model from CC6M captions) and OCR grounding each add +0.010 mean Cor-lbl, bringing the best small detector to 0.467 mean Cor on validation. On the hidden test set the ensemble reaches an official mean Cor 0.487 / Cor-lbl 0.387, ranking 6th-8th of 21-28 teams per language. Appendix E is notably candid: the official entry is a per-language best across uploads rather than one runnable system; the test OCR arm is confounded with 10% extra training data; an EN leaderboard discrepancy is unexplained; and synthetic categories are badly mismatched (mischaracterization ≤6% synthetic vs 38-44% real). Single-seed training (seed 42), per-language swings up to 0.023 Cor, and no significance testing limit statistical confidence.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A two-token head reading mid-network layer 16 of 32 turns a 4B VLM into a competitive character-level hallucination detector, and layer choice is the largest single lever (+0.039 val Cor over the embedding layer).
- OCR grounding and synthetic data generated by the large model each add ≈+0.010 mean Cor-lbl on validation (Table 3).
- On the hidden test set, ensembling the small detector with the 397B zero-shot judge at α=0.8 adds +0.031 mean Cor over the small model (+0.028 against the best small model ever uploaded) and +0.130 over the judge.
- The official entry reaches mean Cor 0.487 / Cor-lbl 0.387, ranking 6th–8th per language on Cor-lbl.
Strengths
Layer sweep with isolating controls
§3.2/Table 1 sweeps seven layers holding architecture, data, schedule and seed fixed (+0.039 mean val Cor from L=0 to the L=16 plateau); permuting the previous-token slot in time costs 0.049 Cor while duplicating the current token costs ≤0.008, cleanly attributing the gain to positional context. App. D bounds the same-layer advantage (+0.017 head-only) as shrinking to 0.000 once LoRA and synth data are added.
Exemplary self-auditing (App. E)
The authors disclose that the official entry is a per-language Cor-lbl best across uploads rather than one runnable system, that the test OCR arm is confounded with 10% extra training data, that the honest ensemble gain is +0.028 (vs the best uploaded small model at 0.457 test Cor) rather than +0.031, that EN figures changed between upload time (0.460/0.347) and the leaderboard (0.466/0.357), and that synthetic categories are badly mismatched.
Rare compute honesty (App. B)
Quantified negative results (App. F)
Theory-linked diversity measurement
Weaknesses
No statistical rigor
Headline score is a composite
Unverified val–test explanation
Synthetic category mismatch unresolved
Single-family, single-engine design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a shared-task system paper: team eye-be-am's entry for SHROOM-Visions 2026 (Vázquez et al., 2026), which requires character-level marking of hallucinated spans in VLM outputs, five-way categorisation (invention, mischaracterization, OCR, miscounting, other), across EN/FR/IT/ZH, scored by Spearman Cor and label-aware Cor-lbl against per-character annotator gold. The paper makes three testable claims: (1) a two-token head reading mid-network layer 16 of 32 of a 4B Qwen3.5-VL is a competitive detector, layer choice being the largest single lever (+0.039 mean val Cor from the embedding layer, Table 1); (2) OCR grounding and 397B-generated synthetic data each add ≈+0.010 mean Cor-lbl (Table 3); (3) a fixed α=0.8 ensemble with a 397B zero-shot judge yields an official mean Cor 0.487 / Cor-lbl 0.387, ranking 6/28, 6/21, 8/21, 7/22 (Table 4). One phrasing overclaim: the Conclusion's 'top third of every language' does not hold for IT (8/21 = 38th percentile).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SHROOM-Visions 2026 shared task?
A shared task where systems must mark, at character level, which spans of a vision-language model's answer are hallucinated relative to an image and prompt, and categorise each span (invention, mischaracterization, OCR, miscounting, other) in English, French, Italian and Chinese. Ranking uses two Spearman correlations per language: unlabelled Cor and label-aware Cor-lbl.
How well did this system rank in SHROOM-Visions 2026?
On the hidden test set it scored 0.487 mean Cor and 0.387 mean Cor-lbl, placing 6th of 28 (English), 6th of 21 (French), 8th of 21 (Italian) and 7th of 22 (Chinese) on Cor-lbl — a mid-to-upper-field result, not a podium finish.
What is the two-token feature in this paper?
For each generated token, the detector concatenates the hidden states of that token and the previous one, both read from layer 16 of 32 of the language model, and a small MLP predicts a 6-way label (five hallucination categories plus none). A control where the previous-token slot is permuted in time costs 0.049 Cor, showing the slot carries positional context.
Is code or data available for this system?
The paper quotes its prompts verbatim, lists hyperparameters and compute (≈4 GPU-hours per small-model training run; ≈363 GPU-hours for synthetic-data generation on 8 H100s), but announces no public code or model release. The underlying SHEEP benchmark data is described in a separate dataset paper by the task organisers.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.