SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models
Shiyu Li, Zi-Yuan Hu, Shijia Huang, Yanyang Li, Yiwu Zhong, Liwei Wang
cs.CV · cs.AI · cs.CL · cs.LG
Quick verdict
Training-free MLLM token pruning claiming 96.5% of LLaVA-1.5 performance at 89% reduction; credible, broad results but no variance tests and speed trails VisionZip.
Summary
SinkPruner is a training-free framework that accelerates multimodal LLM inference by pruning visual tokens in a coarse-to-fine cascade. A visual sanitizer exploits the observation that about 1% of vision-encoder tokens have abnormally high L2 norms, are spatially and feature-redundant, yet dominate CLS attention: it aggregates these outliers into a single sink token and selects remaining tokens by CLS-attention salience plus diversity. A text-guided pruner then ranks the purified tokens by aggregated text-to-vision attention at LLM layers 2, 6, and 15. On LLaVA-1.5-7B, keeping 64 of 576 tokens preserves 96.5% of full-model performance (91.2% with 32 tokens), ahead of VisionZip, HoloV, VisPruner, and SparseVLM; on dynamic-resolution Qwen2.5-VL it retains 98.6/96.3/91.8% at 66.7/77.8/88.9% pruning, and it leads DART and DivPrune on video benchmarks at 80% pruning. Plugged into VisionZip, the filter adds 0.9-2.6 points. Main limitations: no variance estimates or significance tests support the small margins; wall-clock speedup trails VisionZip because eager attention is needed at pruning layers; and only offline inference is evaluated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- High-norm outlier tokens (~1% of visual tokens, per the top-rho criterion with rho=1%) are spatially and feature-redundant (intra-set cosine 0.9817 vs 0.2246) yet dominate CLS attention, systematically misleading attention-based vision-centric pruners.
- Cascading a visual sanitizer before text-guided attention pruning achieves state-of-the-art retention: 96.5% on LLaVA-1.5-7B at 64/576 tokens and 91.8% on Qwen2.5-VL at 88.9% pruning.
- The sanitizer reduces LLM-decoder attention sink ratio from 14.23% to 3.85% and text-visual attention entropy from 6.36 to 4.85, enabling more reliable query-aware selection.
- The high-norm filter is a plug-and-play module that improves VisionZip by 0.9-2.6 relative-performance points on MMBench/POPE at matched token budgets.
Strengths
Measurable, well-grounded diagnosis of two failure modes
Sec. 3.2 quantifies the high-norm phenomenon: two-regime norm histogram (Fig. 2a), neighbor cosine similarity separation (Fig. 4a), and intra-set collapse 0.9817 vs 0.2246 (Fig. 4b); App. A extends the observation to DINO/DINOv2 (2.37% of DINOv2 tokens above norm 150), grounding the method in the Darcet et al. register literature rather than anecdote.
Unusually broad evaluation across architectures, scales, and modalities
12 image + 4 video benchmarks; fixed-grid LLaVA-1.5 (7B and 13B, Tables 1/11/14), high-resolution LLaVA-NeXT (~2,880 tokens, Table 15), and dynamic-resolution Qwen2.5-VL (Table 2) with a CLS-free salience substitution (App. D.3) validated empirically (91.8% retention at 88.9% pruning).
Transparent sensitivity analyses and disclosed approximations
Mechanism-level evidence, not just end-task scores
Real-inference efficiency measured with overhead disclosed
Weaknesses
No statistical reporting anywhere
Efficiency headline is framed against the unpruned ceiling, not the strongest baseline
Configuration degrees of freedom around per-budget schedules
Baseline coverage and ablation gaps
Core criterion bias on tiled high-resolution inputs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SinkPruner — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SinkPruner in one sentence?
A training-free framework that speeds up multimodal LLM inference by first filtering redundant high-norm outlier visual tokens (aggregating them into one sink token) and then pruning the remaining tokens with text-guided attention inside the LLM decoder.
How much does SinkPruner compress without hurting accuracy?
It removes 88.9-94.4% of visual tokens while retaining 96.5% (LLaVA-1.5-7B, 64 tokens) to 91.2% (32 tokens) of full performance; on Qwen2.5-VL it retains 98.6/96.3/91.8% at 66.7/77.8/88.9% pruning, and 98.0% on video benchmarks at 80% pruning.
What are attention sinks and high-norm outliers in this context?
High-norm outliers are the ~1% of vision-encoder tokens with abnormally large feature norms, usually in homogeneous background regions; they absorb disproportionate attention and get wrongly kept by attention-based pruning. Similar sink tokens inside the LLM distort text-guided token selection, which SinkPruner mitigates upstream.
Does SinkPruner require training or model changes?
No. It is inference-time only and works on LLaVA-1.5, LLaVA-NeXT, LLaVA-1.5-13B, and Qwen2.5-VL without retraining; its high-norm filter can also be plugged into other pruners such as VisionZip for reported gains of 0.9-2.6 points.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as captions in the extracted text; the heatmap/attention-map claims (Figs. 1, 2, 4, 6, 7, 8) cannot be visually inspected from this extraction.
- A leftover editorial note in Chinese is embedded in the Figure 3 caption region of the extraction; it is present in the source text, not an extraction artifact.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.