AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference
Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang
cs.CV · cs.AI
Quick verdict
Claims adaptive per-module visual-token skipping halves MLLM FLOPs at near-zero loss; well-ablated, strong tables, but no significance tests or code.
Summary
The paper targets vertical redundancy in multimodal LLM inference: visual tokens are processed by every self-attention and MLP module even when their contribution is negligible. Profiling per-module necessity via KL divergence between full and intervened teacher-forced distributions (Section 3.1), the authors show skippable computation varies across inputs and differs between attention and MLP. AdaVSkip equips each layer with two lightweight routers that independently gate visual-token participation in attention and MLP, using text-side representations; text tokens always compute. Routers are trained in two stages with the backbone frozen: Stage I uses necessity-derived targets (BCE) plus language-modeling loss with straight-through gradients; Stage II refines with GRPO using an answer-correctness reward and a skip-consistency reward anchoring keep ratios to Stage-I references. On LLaVA-1.5-7B, AdaVSkip retains 100.0% of vanilla average performance at 52.0% FLOPs (K=16) and 95.1% at 32.5% FLOPs (K=24), versus 85.8% for ShortV; on LLaVA-NeXT-7B it cuts FLOPs 53.2%, and combined with CDPruner reaches 91.2% FLOPs reduction at 97.2% relative performance. On Qwen3-VL-8B, combined with VisionZip it recovers 85.5% relative performance at 13.5% FLOPs. Measured speedups reach 2.09x prefill (Table 6). Limitations: no significance testing or variance reporting; average relative performance mixes accuracy and MME scores with unstated normalization; necessity profiling requires ground-truth answers and many forward passes; no code availability statement; appendix hyperparameters are absent from the provided text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vertical redundancy in MLLM visual-token computation is input-dependent and differs between self-attention and MLP (Section 3.1, Figure 2).
- Per-layer dual routers with two-stage training skip roughly half of visual-token computation with no average performance loss (Table 1: 100.0% average at 52.0% FLOPs on LLaVA-1.5-7B, K=16).
- Under aggressive skipping, AdaVSkip degrades far more gracefully than fixed-layer and early-exit baselines (Table 1, K=24: 95.1% vs ShortV 85.8% and VTW 66.4%).
- AdaVSkip composes with horizontal token compression: CDPruner+AdaVSkip on LLaVA-NeXT-7B cuts FLOPs by 91.2% while retaining 97.2% of average performance (Abstract, Table 2).
- The skip-consistency reward controls the computation budget during RL refinement without sacrificing accuracy (Table 4: 53.5%->52.0% FLOPs at equal 100.0% average).
Strengths
Novel module-wise, input-adaptive skipping design
First to decouple self-attention vs MLP skipping per layer per input, unlike ShortV (fixed layer subsets), VTW/DyVTE (prefix early exit). Motivated by a quantitative KL-based necessity analysis (Sec 3.1, Figs 2a-c) showing benchmark- and sample-level variation.
Component-wise ablations isolate each design choice
Table 4 progression: module-wise + necessity targets 98.2% -> +CE loss 98.5% -> +input-specific targets 99.3% -> +RL 100.0%; SCR cuts FLOPs 53.5%->52.0% at equal accuracy. Table 5 further separates input-dependent budgets (98.6% fixed-budget) from input-dependent selection (98.7% count-matched prior) vs 100.2% full adaptivity.
Large, consistent margins under aggressive skipping across three backbones
Real-system efficiency measured, not just FLOPs
Composition with horizontal compression is convincingly demonstrated
Weaknesses
No statistical rigor anywhere
Mixed-metric averaging with unstated normalization
Stage-I supervision requires ground-truth answers and heavy profiling
Training-cost fairness vs training-free baselines
Reproducibility signals are weak
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
AdaVSkip: Adaptive Visual Token Skipping Across Layers - Structured Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AdaVSkip and how does it speed up multimodal LLMs?
AdaVSkip adds two small routers to each transformer layer of a multimodal LLM. For each input, the routers decide whether the image tokens need to pass through that layer's self-attention and MLP computations or can skip them (their values pass through unchanged). Because image tokens dominate computation, skipping unneeded modules cuts FLOPs, prefill latency, and KV-cache size, while text tokens are always fully computed. It reports about half the original FLOPs with no average accuracy loss on 7-8B models.
How much faster and cheaper is AdaVSkip?
Standalone it retains roughly 47-52% of original FLOPs on the tested models (e.g., 53.2% FLOPs reduction on LLaVA-NeXT-7B with a measured 2.09x prefill speedup, 1.78x end-to-end speedup, and 52.4% smaller KV cache). Combined with token compression (CDPruner) it reaches a 91.2% FLOPs reduction on LLaVA-NeXT-7B while keeping 97.2% of the original average benchmark performance, and 6.10x prefill speedup.
Does AdaVSkip require retraining or modifying the model?
No. The backbone stays frozen; only the lightweight per-layer routers are trained, in two stages: supervised training against targets derived from module-wise necessity scores, then GRPO-based reinforcement learning with an answer-correctness reward plus a skip-consistency reward that keeps the skip ratio near the Stage-I reference.
How is AdaVSkip different from ShortV, VTW, or DyVTE?
VTW and DyVTE use early exit, restricting visual-token computation to a prefix of layers, and ShortV skips visual-token updates in a fixed layer subset shared by all inputs. AdaVSkip learns input-specific skipping instead: each layer's self-attention and MLP get independent decisions, so both how much and where visual computation is skipped varies per input. The paper reports clear gains especially at aggressive skipping budgets (95.1% vs 85.8% relative performance at 32.5% FLOPs on LLaVA-1.5-7B).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Abstract is truncated mid-sentence; final claim numbers were cross-checked against Section 4.1 and Tables 1-2 instead.
- Figures 1-4 are available only via captions; the t-SNE cluster separation (Fig 2c) and routing visualizations (Fig 4) could not be independently assessed.
- Appendix content (accelerated necessity profiling, hyperparameters lambda_route/lambda_lm/lambda_SCR, GRPO group size, tau_c values, additional settings) is not in the extracted text; several reproducibility judgments reflect that gap.
- No code-availability statement appears in the extracted text; its absence may be an extraction artifact rather than a fact about the paper.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.