MLLMs Hallucinate when Information Distribution Drifts in Synergy Heads
Meng'en Qin, Junye Chen, Jucheng Liu, Youlu Xing, Song Wang, Ruize Han
cs.CV · cs.CL · cs.LG
Quick verdict
HEAL 将幻觉归因于协同头中的信息漂移，并在推理时进行校准；收益广泛，但每个基准的调整和因果证据仍然薄弱。
Summary
本文提出了 HEAL，这是一个无需训练的推理时框架，将多模态 LLM 的幻觉归因于 'synergy' 注意力头内部视觉-语言信息平衡的漂移。它首先应用因果噪声干预（用分布匹配的高斯噪声替换头的输出）来过滤因果冗余的头，然后对四个掩码输入使用反事实双重差分（Difference-in-Differences），将每个剩余的头分解为视觉、语言和协同成分，并使用 Logit-MAD 阈值对头进行分类。核心发现是，幻觉 token 在协同头中表现出向语言偏移的比率（在 LLaVA-1.5 上为 0.28 视觉 : 0.62 语言，而正确 token 为 0.43 : 0.54），而特定模态的头数量几乎没有变化。HEAL 在每一步通过 β=α/αvis 和 γ=(1−α)/αlang 重新缩放视觉和语言值向量，每十步刷新一次头类型。定理 1 认为值空间校准等同于信息分布校准；定理 2 声称在 α 中存在单调的视觉对齐，尽管两者都依赖于可加性和局部线性近似假设。在七个 MLLM（LLaVA-1.5/NeXT, Qwen2/2.5/3-VL, InternVL-7B/3.5）和六个基准（POPE, CHAIR, MMHal-Bench, LLaVA-Bench, MME, BLINK-Twice）上，HEAL 报告了一致的收益——在 LLaVA-1.5 上，POPE F1 从 85.4 提升至 87.7，CHAIRS 从 51.0 降至 36.9——代价是吞吐量为 5.02 tokens/s，而基础模型为 17.25。主要局限性：平衡因子是针对每个基准和模型家族进行调优的，推理速度减慢了约 3.4 倍，定理建立在强有力的假设之上，且未声明代码发布。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MLLM hallucinations arise when the visual-language information distribution inside synergy heads drifts from its healthy equilibrium, not from the number or strength of modality-specific heads (Fig. 1, Sec. 4.5).
- Causal noise intervention plus counterfactual Difference-in-Differences robustly partitions attention heads into redundant, visual, language, and synergy types (Sec. 3.1-3.2, Tables 4, 8, 9).
- Dynamically rescaling synergy-head value vectors with an equilibrium factor α reduces hallucinations across seven MLLMs and six benchmarks without any retraining (Tables 1, 2, 15, 16).
- Value-space calibration is equivalent to information-distribution calibration (Theorem 1), and visual alignment increases monotonically with α (Theorem 2).
Strengths
Causally grounded head attribution instead of weight proxies
Sec. 3.1 replaces each head's output with distribution-matched Gaussian noise and scores contribution as 1 − Sim(y, ỹ(−i)); robustness to zero, uniform, Cauchy, and activation-swap replacements shows 95-97% agreement and stable POPE F1 (Table 9).
Candid treatment of the PID connection
Appendix B.3 explicitly derives Isyn ≈ S − R (Eq. 50-54), admitting 'synergy heads' carry net interaction, and shows what an explicit redundancy functional would add — avoiding the usual overclaiming of information-decomposition papers.
Broad, consistent empirical coverage
Unusually thorough robustness battery
Honest limitation discussion
Weaknesses
Per-benchmark, per-family hyperparameter tuning
Thin support for the headline causal claim
Theorems weaker than framed
Substantial and unevenly foregrounded inference cost
No code release and an unexplained normalization
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
范围与主张
HEAL 是一种无需训练的推理时方法，用于缓解 decoder-stack MLLM（LLaVA、Qwen-VL、InternVL 系列）中的目标级幻觉。论文提出了三个主张：(1) 当 'synergy' 注意力头内部的视觉-语言信息分布偏离健康平衡时，会产生幻觉，而不是由于特定模态头部的数量或强度（图 1，第 4.5 节）；(2) 因果噪声干预加上反事实双重差分（DiD）产生了一个稳健的四向头部分类（第 3.1-3.2 节）；(3) 使用平衡因子 α 动态重新缩放协同头值向量，可以在七个 MLLM 和六个基准测试中将生成引导回视觉证据（第 4 节）。范围特意设定得很窄：即推理时干预；作者承认早期视觉编码失败仍然无法解决（第 5 节，附录 D.1）。
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HEAL and how does it reduce MLLM hallucinations?
HEAL is a training-free, inference-time method. It classifies attention heads into redundant, visual, language, and synergy types using noise interventions and counterfactual difference-in-differences, then rescales the visual and language value vectors inside synergy heads at each decoding step toward a target visual-language balance (controlled by a factor α). The authors report lower hallucination rates on benchmarks like POPE and CHAIR across several multimodal models.
How much does HEAL improve hallucination metrics?
On LLaVA-1.5-7B, HEAL raises POPE F1 from 85.4 to 87.7 and cuts CHAIRS from 51.0 to 36.7, while improving MME to 669.76. Similar gains are reported for LLaVA-NeXT, Qwen2/2.5-VL, InternVL-7B, and newer Qwen3-VL-8B / InternVL3.5-8B models. The cost is slower decoding: about 5 tokens/s versus 17 for the unmodified model on RTX 4090 GPUs.
Does HEAL require retraining or extra data?
No. It is plug-and-play at inference: all quantities are computed from the model's own activations, and the only hyperparameters are the equilibrium factor α (recommended 0.4-0.6) and a head-type update interval (10 steps). However, α is currently chosen empirically per model family and task difficulty.
Is code available for HEAL?
The paper text provided does not mention any code release, repository, or artifact link, which is a practical obstacle given that the method requires nontrivial KV-cache-level interventions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded content (e.g., Fig. 1 axis labels, Fig. 2/7 scatter and bar visuals) is garbled by PDF extraction; assessments that would depend on figure visuals rest on captions and body text instead.
- The paper's own dashes for VHR and VISTA rows in Table 1 are present in the source (not an extraction loss); those baselines are only partially comparable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.