GHOST-Q: Towards Studying Grounding Hallucinations Overlooked Under Same-score TradeOffs in Quantized VLMS
Saim Rehman, Muhammad Shafique
cs.CV · cs.AI · cs.LG · cs.MM
Quick verdict
Quantized VLMs keep headline scores but shift item-level grounding outcomes; paired statistics are rigorous, yet effects are small, scope bitsandbytes-only.
Summary
The paper presents GHOST-Q, a controlled cross-precision evaluation of quantization in three 8B vision-language model families (Qwen3-VL, InternVL3, Idefics3) under FP16, INT8 (LLM.int8()), and NF4, across 25,667 forced-choice items per configuration from MMStar, POPE, AMBER (existence/attribute/relation), and HallusionBench. Beyond aggregate accuracy, the authors pair FP16 and quantized predictions item-by-item, applying exact McNemar tests with Benjamini–Hochberg FDR correction, bootstrap confidence intervals, and matched-pair odds ratios over all 36 comparisons. Five of six quantized variants stay within ±2 percentage points on MMStar, yet 10 of 36 paired effects remain FDR-significant, nine on hallucination-sensitive conditions; effects are bidirectional (odds ratios 0.52–13.0). Same-device A100 profiling shows INT8/NF4 cut memory footprint 43–66% but do not reduce median latency on the tested Transformers/bitsandbytes stack, with NF4 dominating INT8. An open-ended AMBER audit finds 88.28% of responses hit a 256-token ceiling, with architecture- and precision-dependent censoring; a 1,024-token FP16 probe confirms the budget truncates naturally longer outputs. Limitations: only calibration-free bitsandbytes weight-only quantization is tested (no GPTQ/AWQ or the motivating MBQ/VLMQ), significant deltas are small (≤1.56 pp, one −2.07 pp case), the open-ended audit scores length behavior rather than hallucination content, and no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 18 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Preserving aggregate utility scores after post-training quantization does not preserve item-level grounding behavior: 10 of 36 FP16–quantized paired comparisons remain significant after BH-FDR correction, nine on hallucination-sensitive benchmarks.
- Precision-induced prediction flips are bidirectional — some quantized variants significantly improve grounding (Qwen3 INT8 AMBER-existence OR 13.0) while others degrade (Idefics3 NF4 POPE OR 0.56).
- On the same A100 with Transformers/bitsandbytes, INT8 and NF4 cut memory footprint 43–66% but do not reduce median end-to-end latency versus FP16; NF4 dominates INT8.
- A fixed 256-token generation budget censors 88.28% of open-ended AMBER responses at architecture- and precision-dependent rates, confounding naive generative hallucination comparisons.
Strengths
Rigorous paired statistical protocol
§3.2 goes beyond aggregate deltas: exact McNemar on discordant pairs, BH-FDR across all 36 comparisons, 10,000-resample bootstrap CIs, and matched-pair odds ratios with intervals — all reported per effect in Table 2.
Controlled design that isolates precision
§3.1 matches examples, preprocessing, prompts, and decoding across precisions; §4.2 fixes a 1-megapixel image budget for Qwen3-VL in all conditions, so precision is the only experimental factor.
Honest scoping and self-critique
Internally consistent numbers
Practical deployment signal
Weaknesses
Narrow quantization scope vs. motivation
Small absolute effects under high power
Stack-bound latency conclusion
Open-ended audit never scores hallucination content
No code/data release; ad hoc tolerance
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
GHOST-Q is a four-page empirical study asking whether post-training quantization (PTQ) preserves *which* items a VLM grounds correctly, not just headline accuracy. Three 8B families (Qwen3-VL, InternVL3, Idefics3) × three precisions (FP16, LLM.int8() INT8, NF4) are evaluated on 25,667 forced-choice items per configuration (MMStar 1,500; POPE 9,000; AMBER existence 4,924 / attribute 7,628 / relation 1,664; HallusionBench 951) — 231,003 predictions total. Four claims anchor the paper: (1) aggregate preservation hides item-level redistribution (10/36 FDR-significant paired effects, nine hallucination-sensitive); (2) memory savings do not imply latency gains on the tested stack; (3) fixed generation budgets censor open-ended outputs at architecture/precision-dependent rates; (4) quantized VLMs should be evaluated jointly on utility, grounding reliability, generation behavior, and realized efficiency. The motivating example (§1) is well chosen: Idefics3-NF4 moves MMStar by +0.40 pp yet shifts POPE with 257 harmful vs 145 beneficial flips (q = 3.0×10⁻⁷).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does quantization make vision-language models hallucinate more?
Not uniformly. GHOST-Q finds that while headline benchmark scores of three 8B VLMs barely change after INT8/NF4 quantization, item-level pairing shows errors are redistributed: some quantized variants fix far more errors than they introduce (odds ratio up to 13.0), others the reverse (down to 0.52). Ten of 36 comparisons stayed significant after multiple-testing correction, nine on hallucination-sensitive benchmarks — but all significant accuracy deltas were small (−2.07 to +1.56 percentage points).
Is a quantized model actually faster at inference?
Not in this paper's setup. On a single A100 GPU with the Transformers+bitsandbytes stack at batch size one, INT8 was 43.5–280.4% slower and NF4 2.2–69.6% slower than FP16, despite cutting model memory by 43–66%. NF4 beat INT8 on both memory and speed. The authors stress this reflects the tested software stack, not quantization methods in general — optimized serving stacks may behave differently.
What is generation-budget censoring in VLM evaluation?
When open-ended VLM evaluations cap generation at a fixed token budget, responses that would naturally run longer get truncated before completion. In GHOST-Q, 88.28% of 9,036 responses to 'Describe this image.' hit the 256-token cap, with rates varying by architecture (73–99.8%) and precision. A 1,024-token probe confirmed the cap cuts naturally longer outputs, so naive comparisons of generative hallucination scores under a fixed budget partly measure truncation, not grounding quality.
Which models and benchmarks did GHOST-Q use?
Qwen3-VL-8B, InternVL3-8B, and Idefics3-8B, each in FP16, INT8 (LLM.int8()), and NF4. Forced-choice evaluation used MMStar (1,500 items), POPE (9,000), AMBER existence/attribute/relation (4,924/7,628/1,664), and HallusionBench (951) — 231,003 predictions in total — plus the 1,004-image AMBER open-ended generation subset.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are available only as captions in the extracted text; their internal data (e.g., the Fig. 2 dot plot) could not be verified.
- The extraction appears complete for a 4-page venue paper, but supplementary details (exact prompts, seeds, scoring scripts) may exist outside the extracted text.
- Benchmark scoring implementations (e.g., exact-match rules for AMBER subsets) cannot be audited from the text alone.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.