StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
Zhenbin Wang, Lei Zhang, Lituan Wang, Wei Huang, Yan Wang, Zhenwei Zhang
cs.CV · cs.AI
Quick verdict
Training-free VLM token selector that tops all tested training-free baselines; principled and well specified, but margins are thin and no latency is reported.
Summary
StackTok is a training-free visual-token selector that compresses VLM visual sequences before LLM inference. Its core move is an asymmetric formulation: query relevance (facility-location coverage over query-vision affinities in the LLM embedding space) is the objective, while visual coverage (the same function over pre-projector vision features) serves as budget-calibrated support. A coverage-only greedy sequence yields a size-indexed reference curve; the normalized entropy of query-vision affinities calibrates the strictness of the support target; and a per-step gate switches between relevance- and coverage-oriented greedy additions according to the current subset's support deficit. For multi-crop high-resolution inputs, crop-local nominees compete for one shared budget by combined marginal gain, with optional coverage-floored swap refinement. Across five VLMs (LLaVA-1.5 and LLaVA-NeXT at 7B/13B, Qwen2.5-VL-7B), ten benchmarks, and three budgets, StackTok ranks first among training-free selectors in every tested setting, though margins over MMTok are often only 0.08-0.29 points. On LLaVA-NeXT-7B, 160 of 2,880 tokens retain 95.26% of full-token performance. Main limitations: no measured latency despite the acceleration claim, no variance or significance analysis for the thinnest margins, the fine-tuned VisionZip‡ surpassing it in several multi-crop cells, and theory that covers only the coverage reference rather than the emitted selection sequence.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Query relevance and visual coverage play asymmetric, state-dependent roles, and gating coverage additions by a size-indexed, entropy-calibrated target outperforms fixed relevance-coverage scalarization (Table A2: +0.06-0.14 points over best per-budget α).
- StackTok ranks first among training-free token selectors in every tested model-budget setting across five VLMs and ten benchmarks (Tables 1, 3, A1).
- On high-resolution LLaVA-NeXT-7B, StackTok retains 95.26% of full-token performance using 160 of 2,880 (5.6%) visual tokens.
- Dynamic shared-budget allocation across crops outperforms static per-crop quotas, with the gap widening from 0.42 to 0.86 points as the budget tightens (Table A3).
- Replacing the size-indexed reference with a final-budget target costs 0.29-0.61 points as budgets tighten, showing early prefixes need size-matched support targets (Table 2).
Strengths
Principled asymmetric formulation validated by targeted ablations
Reframing relevance as objective and coverage as budget-calibrated support is cleanly formalized (Eqs. 2-11), and Table 2/Table A2 show the gated policy beats the best per-budget fixed scalarization by 0.06-0.14 points and the one-signal variants by larger margins (0.45-2.46 points at 64 tokens), so the design is tested against its natural alternatives rather than only weak baselines.
Unusually transparent reporting
The authors disclose that Avg. is recomputed from rounded displayed values (Table 1 caption), expose per-benchmark regressions at 64 tokens (Fig. 3b: MME -0.67, VQAText/SEED -0.19 vs MMTok), explicitly label the |Δ|≤0.10 band as a visualization rule rather than a significance threshold, and state plainly that Theorem 1 does not cover the interleaved policy.
Broad, consistent empirical coverage
Complete algorithmic specification enabling reproduction
Multi-crop shared-budget allocation is a distinct, isolated contribution
Weaknesses
Thin margins with no variance or significance analysis
Acceleration claim lacks any runtime evidence
Fine-tuned baseline dominates half the multi-crop cells
Theory does not cover the actual algorithm
Component value unevenly demonstrated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
StackTok (arXiv 2609.16841, Sichuan University) addresses training-free visual-token selection for VLM inference: given a fixed retention budget K, choose k visual tokens to pass to the LLM so accuracy stays close to full-token inference. The central claim is a reframing — query relevance is the objective, visual coverage is a budget-calibrated support requirement — and that a reference-gated interleaved greedy selector built on this framing beats relevance-only (FastV, SparseVLM), coverage-only (VisionZip, DivPrune), and fixed-scalarization hybrid (MMTok) methods. Supporting claims: first place among training-free selectors in all tested model-budget cells across five VLMs and ten benchmarks; 95.26% of full-token performance on LLaVA-NeXT-7B with 5.6% of tokens; and measurable contributions from each mechanism (size-indexed reference, entropy calibration, dynamic crop allocation).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is StackTok and does it require training?
StackTok is a training-free visual token selection method for vision-language models: before the language model runs, it keeps only a small, adaptively chosen subset of the image's visual tokens. It needs no fine-tuning or added parameters — it scores tokens using the model's own embeddings plus a greedy, reference-gated selection procedure.
How much accuracy does StackTok lose when compressing visual tokens?
Per the paper, StackTok keeps 98.99%, 98.03%, and 96.66% of full-token performance on LLaVA-1.5-7B while retaining 192, 128, and 64 of 576 tokens. On high-resolution LLaVA-NeXT-7B it keeps 95.26% of performance with only 160 of 2,880 tokens (5.6%). Losses grow on some individual benchmarks like MME and TextVQA at the tightest budget.
How much faster is VLM inference with StackTok?
The paper reports only token-retention ratios, not measured latency or throughput. Cutting 2,880 tokens to 160 should substantially reduce prefill compute, but the selection itself costs roughly O(n²) affinity computations, and no wall-clock speedup numbers are provided, so actual acceleration is not quantified.
Which baselines and benchmarks does StackTok compare against?
Training-free token reduction methods: FastV, SparseVLM, VisionZip (plus its fine-tuned variant VisionZip‡ as an excluded reference), DivPrune, and MMTok. Benchmarks include GQA, MMBench, MME, POPE, ScienceQA-IMG, VQAv2, TextVQA, MMMU, SEED-Bench, and OCRBench, using the lmms-eval framework.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as captions and inline numbers; the plotted curves in Figure 3 could not be visually inspected.
- The extracted text contains duplicated passages (the Method section and figure captions appear twice), indicating extraction noise; some cross-references (Table 2 vs Table 3) are inconsistent in the source.
- Appendix content beyond the sections shown (e.g., full Qwen2.5-VL implementation details, limitations section) may be incomplete.
- Baseline run provenance (re-run vs cited numbers) cannot be verified from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.