DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMs
Yingxuan Zhuang, Miao Pan, Wangjie Gan, Jingxiao Yang, Fan Wang, Weiming Liu, Cheng Tan, Xuhong Zhang, et al.
cs.AI · cs.LG
Quick verdict
DEEPO claims dual fixes to RL hallucination correction in MLLMs; evidence is unusually rigorous but marred by cross-table number inconsistencies and no code release.
Summary
The paper diagnoses two failure points in the reward-to-update chain of GRPO-style RL for MLLM hallucination: high semantic-entropy queries often yield all-wrong groups, collapsing the group-relative advantage to zero, and confident-but-wrong tokens receive vanishing score gradients. DEEPO responds with a dual-stage method: entropy-triggered expert prefixes (truncated, answer-free CoT prefixes from Video-R1 traces, NLI-audited against answer leakage) restore advantage variance, and advantage-sign-aware Rényi-2 gradient scaling (correction weighted by the inverse damped Fisher trace, reinforcement by the trace itself) routes correction toward confident errors. On Qwen2.5-VL-7B across six benchmarks it outperforms GRPO, DAPO, and PAPO (VideoMMMU 53.3 vs 44.7); the two-branch interaction is significantly positive only on VideoMMMU (+4.0, 95% CI [1.1, 6.9], four seeds) and additive elsewhere. Theory covers the vanishing-gradient identity and a conservative damped proximal step, honestly scoped; ablations include shuffled hints, self-generated hints (retaining roughly 71% of the VideoMMMU gain), fairness and trigger controls. Main limitations: no code release, internally inconsistent numbers across tables, and guarantees restricted to a local sample-wise surrogate.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- GRPO's correction of hallucinations fails at two points: all-wrong groups on high-semantic-entropy queries collapse the group-relative advantage to zero, and confident-but-wrong tokens have vanishing score gradients (Prop. 1).
- Semantic-entropy-triggered expert prefixes (truncated, answer-free, NLI-audited) restore advantage variance and inject grounded supervision on hard queries.
- Advantage-sign-aware Rényi-2 gradient scaling (correction weighted by (τt+ε₀)⁻¹ with cap 20, reinforcement by τt) delivers correction to confident errors where entropy-only rules provably cannot prioritize correction (Remark 4.2).
- On Qwen2.5-VL-7B across six benchmarks, DEEPO beats GRPO, DAPO, and PAPO (e.g., VideoMMMU 53.3 vs 44.7); the branch interaction is significantly positive only on VideoMMMU (+4.0, 95% CI [1.1, 6.9]) and additive elsewhere.
- The gains are not explained by extra compute or expert data, and a self-generated-hint variant retains ≈71% of the VideoMMMU gain.
Strengths
Diagnosis and mechanism are quantified, not just asserted
Table 4(a) shows all-wrong group fraction rising monotonically 12%→63% across semantic-entropy deciles; Table 4(b) shows hint activation converts 60% of triggered all-wrong groups to mixed, moving failed-rollout mean advantage 0→−0.212 and variance 0→0.600 — directly evidencing the claimed advantage collapse.
Theory is matched to the design and honestly scoped
Prop. 1 (E‖∇z log π‖² = 1−e^(−H₂)) is a clean standard identity proved in App. B; Thm. 4.1 explicitly declares itself sample-wise, sign-blind, and non-modeling of PPO/Adam; App. C records four formal caveats — rare candor in RL methods papers.
Unusually thorough causal-control suite
Statistical reporting above field norm
Leakage-conscious hint design with audits
Weaknesses
Internal numerical inconsistencies across tables
No multiple-comparison discipline around the headline significance claim
The 'delivery repair' mechanism is weakly selective in practice
Coverage boundary undercuts the headline narrative
No code or artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
DEEPO (arXiv 2609.28570) targets hallucination in RL fine-tuning of multimodal LLMs. The core claim: GRPO's correction chain fails at two points — (I) on high-semantic-entropy queries, unanimously wrong groups collapse the group-relative advantage to zero (Sec. 1, Table 4a: all-wrong fraction 12%→63% across Hs deciles); (II) confident-but-wrong tokens are gradient-invisible, since E‖∇z log π‖² = 1−e^(−H₂) (Prop. 1). The method pairs entropy-triggered expert prefixes (Sec. 4.1) with advantage-sign-aware Rényi gradient scaling (Sec. 4.2). Headline result: interaction of the two branches is significantly positive only on VideoMMMU (+4.0, 95% CI [1.1, 6.9], Table 2), additive elsewhere; DEEPO reaches 53.3 on VideoMMMU vs 44.7 for GRPO (Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DEEPO and how does it reduce hallucination in multimodal LLMs?
DEEPO (Dual-Entropy Enhanced Policy Optimization) is a GRPO-style RL fine-tuning method for multimodal LLMs. It targets two failure points where reward signals fail to fix hallucinations: (1) high-semantic-entropy queries where all sampled answers are wrong, collapsing the group-relative advantage to zero, and (2) confident-but-wrong tokens whose score gradients vanish. It triggers expert hints (truncated, answer-free reasoning prefixes) on high-entropy queries and applies advantage-sign-aware Rényi-entropy gradient scaling, improving over GRPO on six multimodal benchmarks (e.g., VideoMMMU 53.3 vs 44.7) while preserving accuracy.
Is DEEPO's improvement over GRPO statistically significant?
Partially. Using four-seed means over six benchmarks, the interaction between the method's two components is significantly positive only on VideoMMMU (+4.0, 95% CI [1.1, 6.9]); on the other five benchmarks the confidence intervals include zero, consistent with additive gains. The paper performs no multiple-comparison correction across these six tests, and several pairwise comparisons (POPE, HallusionBench) also include zero.
Are there inconsistencies in the DEEPO paper's reported numbers?
Yes, at least one apparent internal inconsistency: in Tables 3, 5, and 6 the DEEPO row's middle column reads 69.7, whereas Tables 1 and 2 report 57.6 on VideoHallucer for the same configuration — and 69.7 matches Table 1's HallusionBench column value. This suggests a column-labeling or value-transposition error in at least one set of tables; the authors should clarify which values correspond to VideoHallucer.
How is DEEPO trained and what compute does it need?
Qwen2.5-VL-7B fine-tuned with GRPO (group size 8, temperature 1.2, 2000 steps, KL coefficient 0.04) on a 20k subset of the Video-R1 dataset. Semantic entropy is computed by clustering rollout answers with DeBERTa-MNLI; expert prefixes are capped at half the thought segment and NLI-filtered. The full run takes about 3 days on 8×A100 80GB GPUs, with roughly 18% higher per-step cost than plain GRPO.
Is code available for DEEPO?
The extracted paper text does not mention any code or data release. Reproduction requires implementing GRPO with the two DEEPO branches on top of a public framework, using the public Video-R1 dataset and Qwen2.5-VL-7B; the detailed hyperparameter specification (App. G) makes replication feasible but not turnkey.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on extracted text: Figures 1–5 are unrecoverable from text, so visual evidence (e.g., Fig. 3 gradient-mass pilot, Fig. 5 stability curves) could not be independently assessed.
- Author list differs between metadata (8 authors) and the paper byline, which includes Jintao Chen (9 authors).
- Table column alignment in the extraction may be imperfect; the cross-table inconsistency flagged in riskFlags could partly stem from extraction, though the alignment is internally consistent (69.7 matching HallusionBench) across four separate tables.
- Appendix F's Hs–H1 correlation protocol is described, but Table 12 was extracted and analyzed; no text truncation otherwise evident.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.