HALDETECT at ImageEval 2026 Shared Tasks: Answer-First Contrastive Grounding with QLoRA
Syed Mohaiminul Hoque, Md Sakhawat Hossain
cs.CV · cs.AI
Quick verdict
Answer-first contrastive grounding + QLoRA on Qwen2.5-VL-7B: CI 0.035, third place; rigorous honest stats, but the data-scaling claim is seed noise.
Summary
HALDETECT is the authors' system for the English hallucination-detection track (Task 1b) of ImageEval 2026, where a system sees an image with three statements and must mark the single visually grounded one. Each item is treated as one joint contrastive decision: the model emits an 'Answer: X' line before any explanation and organizes its rationale around a colour/texture, shape/form, and context checklist. The best submission fine-tunes Qwen2.5-VL-7B-Instruct with 4-bit QLoRA (rank 8, ~20M trainable parameters, frozen vision tower) and reaches Contrastive Instability 0.035 on the 1,000-item test set, third of eight teams. That 0.035 is, by the authors' own account, the post-hoc best of ten eligible submissions; fresh-seed retrains average about 0.040. Post-task paired McNemar tests with Holm correction confirm QLoRA significantly beats the best prompting system (fixes 35, breaks 8, CI 0.062 to 0.035), but the single-seed curve that reads as monotone data scaling vanishes under reseeding (fresh means flat at .0397-.0407), so it is seed variance, not a size effect. The 35 residual errors concern function/intent, material, and recognition distinctions, and adapter voting does not help. Limitations: the recipe was never tuned, the vision-frozen choice lacks an ablation, the checklist gain is specific to Qwen2.5-VL, and 0.035 is not the expected performance of a fixed configuration.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Framing Task 1b as one answer-first contrastive decision with an attribute checklist, plus QLoRA adaptation of the language layers of Qwen2.5-VL-7B, yields test CI 0.035 and third place of eight teams.
- The QLoRA gain over the best prompting-only system is statistically robust (35 fixes vs 8 breakages, exact McNemar p=4.19e-5, Holm p=.0016, bootstrap −0.027 [−0.040, −0.015]).
- The apparent monotone data-scaling curve across training sizes (0.049→0.035 at seed 42) does not survive reseeding — fresh-seed means are flat (.0397–.0407) — so it is seed variance, not a size effect.
- The 35 residual test errors are culturally plausible function/intent, material, and recognition distinctions (blind dual-annotated), and naive adapter voting does not improve on the single best adapter.
Strengths
Exceptional statistical honesty about its own results
Section 4 discloses that the headline 0.035 is the post-hoc best of ten eligible test submissions ('best of sixteen' runs, Section 4.1), reports the honest devtest-selected alternative (QLoRA-2,348 at 0.039), and Section 4.1/Table 5 shows the apparent data-scaling curve (seed-42: .049→.035) is seed variance (fresh means flat .0397–.0407).
Careful inferential statistics with multiple-comparison control
Paired item-bootstrap intervals (50k resamples), exact McNemar tests with Holm correction over all 45 system pairs (Section 5, Table 7), Wilson intervals for sparse country/category rates (Table 9), and a permutation test for the error-Jaccard gap (p=0.10). The paper explicitly separates single-seed significance from replicability.
Strong reproducibility artifacts
Cross-family and negative-result reporting
The answer-first effect is well-evidenced beyond aggregate deltas
Weaknesses
Headline number is a selection artifact
The data-scaling narrative in Tables 1–2 does not replicate
Key design decisions are un-ablated
The attribute-checklist component generalizes poorly within the evidence given
Error taxonomy is underpowered and rationale faithfulness untested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
HALDETECT at ImageEval 2026 Task 1b — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What did HALDETECT achieve at ImageEval 2026 Task 1b?
HALDETECT placed third of eight teams on the English hallucination-detection track, with a Contrastive Instability (lower is better) of 0.035 on the 1,000-item test set — 965 of 1,000 items fully correct. It combines answer-first contrastive prompting with QLoRA fine-tuning of Qwen2.5-VL-7B-Instruct.
Is the reported CI 0.035 the performance you should expect from this system?
Not exactly. The authors disclose it is the post-hoc best of ten eligible test submissions. When they retrain the same recipe with three fresh random seeds, results average about 0.040 regardless of training-set size, so ~0.040 is the more honest expectation; the strict devtest-selected submission scored 0.039.
Why does answer-first prompting help with hallucination detection?
Because the decision is committed before the model generates fluent reasoning, so self-generated text about a distractor cannot bias the answer token. The paper shows this is corrective, not merely disruptive: at 7B it repaired 29 of 500 devtest items and broke 8 (exact McNemar p≈7.5e-4), and the effect persists after excluding unparseable outputs.
What kinds of errors does the system still make?
Its 35 test errors are near-neighbour distinctions where all three statements sound culturally plausible: function or intent the image cannot show (e.g., film set vs real battlefield), observable material properties (granite vs quartz countertops), and object or scene recognition (train station vs airport terminal).
Is the code and data available?
Yes. Code, prompts, notebooks, prediction archives, the taxonomy rubric with blind worksheets, and analysis scripts are public on GitHub (SyedT1/IE2026-HalDetect). Test gold labels were released by the organizers after the shared task and used only for post-task analysis.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 1-10 and Figures 1-3 are embedded in the extraction as inline prose with some layout interleaving; all cited values were cross-checked against their surrounding labels, but complex table geometry (e.g., Table 5's per-seed grid) was reconstructed from the linearized text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.