SKstars at SHROOM: Visions Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted Vision--Language Models
Ali Athar, Imran Ahsan, Joon-Yong Jung
cs.AI · cs.CV
Quick verdict
A shared-task system paper: zero-shot 72B + LoRA 7B ensemble for VLM hallucination span detection; honest analysis, mid-pack result (15th of 29).
Summary
This system paper describes SKstars' English-track entry to SHROOM-Visions 2026, a shared task requiring fine-grained hallucination detection in vision-language model outputs: marking hallucinated character spans, assigning one of five categories, and estimating per-span confidence. The pipeline merges zero-shot predictions from Qwen2.5-VL-72B-Instruct (4-bit quantized, three RTX A6000 GPUs) with those of a Qwen2.5-VL-7B-Instruct LoRA-adapted on 3,417 English training instances (rank 16, 47.5M trainable parameters). The 72B spans anchor the output; spans also proposed by the 7B model (overlap IoU ≥ 0.3) have confidence raised from 0.33 to 0.67, 7B-only spans are discarded, and confidence values are then redistributed to match training marginals (71.2/20.7/8.1%). On the hidden English test set the system scored Cor+Lbl 0.2902 (15th of 29 teams), Cor 0.3642 and IoU 0.3151 (18th on both). Strengths include a candid error analysis — the component models were over- and under-confident in opposite directions (98.8% vs. 93.9% of spans at extreme confidence values) — and detailed reproducibility reporting. Key limitations: the 50-example development diagnostic (IoU ≈ 0.51 vs. official 0.3151) supports no statistical conclusions; the decision to discard all 7B-only spans rests on a 0.513-vs-0.509 IoU difference; no component ablation is reported on the test set; and the abstract's multilingual claim is untested, since only English was evaluated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Combining a large zero-shot model (Qwen2.5-VL-72B) with a smaller LoRA-adapted model (Qwen2.5-VL-7B) via agreement-guided ensembling provides a practical framework for fine-grained hallucination localization.
- The two component models exhibit complementary calibration failures — 98.8% vs. 93.9% of spans at extreme confidence values — which agreement-based confidence only partially corrects.
- Development-set improvements (IoU ≈ 0.51 on 50 examples) do not transfer to the hidden test set (IoU 0.3151), partly due to distribution shift from 400 human-written test items absent from training.
- Matching marginal confidence frequencies to training statistics does not yield instance-level calibration, as reflected in the official Cor of 0.3642.
Strengths
Evidence-driven ensemble motivation from diagnosed failure modes
Sections 3.1-3.2 and Table 3 document complementary calibration failures — 72B raw places 98.8% of spans at confidence 1.0 while 7B raw places 93.9% at 0.33 — and the agreement rule (overlap IoU ≥ 0.3 raises 0.33→0.67) follows directly from this diagnosis rather than from ad hoc design.
Candid, specific self-criticism
Section 5 and the Limitations section name the exact weaknesses: the 50-example diagnostic has no CIs, the asymmetric ensemble 'likely discarded genuine hallucination spans', the 0.3 threshold was untuned, and marginal matching 'does not guarantee that a particular span receives an accurate probability'. This is unusually honest for a system paper.
Above-average reproducibility reporting
Plausible qualitative account of the dev-test gap
Weaknesses
Load-bearing design decision justified by a statistically meaningless difference
No component ablation on the official test set
Confidence 'adjustment' is not calibration
Multilingual claim unsupported by any experiment
Prevalence prior injected into the prompt
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: SKstars at SHROOM-Visions — Agreement-Guided Ensembling of Zero-Shot and LoRA-Adapted VLMs
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SHROOM-Visions?
SHROOM-Visions 2026 is a shared task on fine-grained hallucination detection in vision-language model outputs. Given an image, a prompt, and a generated response, systems must mark the hallucinated character spans, assign each span one of five categories (invention, mischaracterization, OCR problem, miscounting, other), and attach confidence values reflecting expected annotator agreement. It extends the SHROOM (SemEval-2024) and Mu-SHROOM (SemEval-2025) text tasks to image-conditioned outputs.
How well did the SKstars system perform?
On the official hidden English test set it scored Cor+Lbl 0.2902 (rank 15 of 29 teams), Cor 0.3642 (rank 18), and IoU 0.3151 (rank 18) — a mid-pack result. Its internal development IoU of about 0.51 on 50 examples did not transfer to the test set.
Which models does the SKstars system use and how are they combined?
It combines Qwen2.5-VL-72B-Instruct used zero-shot (4-bit quantized on three RTX A6000 GPUs) with Qwen2.5-VL-7B-Instruct fine-tuned via LoRA (rank 16, 47.5M trainable parameters, one epoch on 3,417 English task examples). The 72B model's spans form the final output; spans also proposed by the 7B model (overlap IoU ≥ 0.3) get confidence raised from 0.33 to 0.67, 7B-only spans are discarded, and confidences are then redistributed to match training frequencies.
Is the code and data available for this paper?
The paper links a public repository with code and predictions (github.com/aliathar1401/SK-Stars-shroom-visions-2026) and gives full hyperparameters and runtimes. However, the headline scores come from the organizers' hidden test set, and the 50-example internal diagnostic subset is not released, so exact test numbers cannot be independently re-derived without the shared-task data.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 is available only as a text-rendered flow description; the visual layout was inferred from the caption and body text.
- Leaderboard context beyond ranks (other teams' absolute scores, score distributions) is not in the paper, so 'mid-pack' is inferred from the reported ranks alone.
- The shared-task overview paper (Vázquez et al., 2026) and the SHEEP dataset paper are cited but not examined; task-metric definitions are taken from the paper's summary.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.