Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs
cs.CV · cs.CL
Quick verdict
Claims frame selection, not spatial resolution, is the dominant lever in long-video MLLMs; a rigorous, unusually honest controlled study, harness-scoped.
Summary
This controlled study asks how a fixed visual-token budget should be spent in long-video multimodal LLMs: on which frames are kept, on each frame's spatial resolution, or on more frames. Holding the scorer (LongCLIP over a 1 fps pool), prompt boundary, frame budget, and answerer fixed, the author varies one decision at a time across six training-free selectors, three benchmarks (LongVideoBench, Video-MME, LVBench), and Qwen3-VL-8B, InternVL3, and GPT-5-mini answerers. Selection is the largest lever: eight frames chosen by Orthogonal Matching Pursuit (OMP, 1993) beat sixteen uniform frames by 6.9 points on hour-long LongVideoBench (p=.0011); OMP beats uniform sampling by 5.7-11.8 points and stays within one point of LDDR's stage-1 selector. Halving spatial resolution at fixed timestamps costs at most 0.44 points, bounded by a post-hoc ±3-point TOST equivalence margin; reinvesting the savings in sixteen compressed frames returns a further 1.6-3.0 points (token parity audited on LongVideoBench). The paper also reports a 0.07-3.74-point disagreement between two matched harnesses and an implementation bug in which the AKS baseline silently duplicated top-k while aggregate accuracy barely moved. Main limitations: equivalence margins chosen post hoc, no multiplicity correction, FOCUS tested only as a schedule replay, LDDR only at stage 1, the selection stage's compute cost unmeasured, and effects confined to contrastive scorers and specific answerer-benchmark pairs.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Selection is the largest allocation lever: eight OMP-selected frames beat sixteen uniform frames by 6.9 points on hour-long LongVideoBench (p=.0011), and OMP beats uniform sampling by 5.7–11.8 points across three benchmarks.
- Unmodified 1993-era Orthogonal Matching Pursuit matches or comes within one point of every purpose-built training-free selector tested, including LDDR's stage-1 Linear-DPP selector.
- Halving per-frame spatial resolution at fixed timestamps costs at most 0.44 points, bounded by a post-hoc ±3-point TOST equivalence interval on pooled long LongVideoBench videos.
- Reinvesting the saved tokens in twice as many compressed frames returns a further 1.6–3.0 points at audited equal-or-lower token cost on LongVideoBench (ratios 0.984–0.996).
- Two matched harnesses running the same published rules disagree by 0.07–3.74 points, so cross-paper selector comparisons at this scale are unreliable and must be done inside one controlled harness.
Strengths
Genuinely controlled paired design
§3.1 fixes scorer, prompt boundary, frame budget, and answerer, and defines the experimental unit as one question answered under two policies; every selector reads the same cached LongCLIP stem-score vector over the 1 fps pool (§3.2), directly attacking the confounding the paper diagnoses in cross-published comparisons.
Statistical discipline unusual for the area
Exact two-sided McNemar tests for all paired accuracy contrasts; TOST with 90% CIs for equivalence (±3 cleared at p=.0022, ±2 honestly reported as failing at p=.054); discordant counts and power limits stated (e.g., scorer-swap CI [−3.6, +5.6]); §5.2 explicitly rejects reading significant-beside-non-significant as interaction.
Exceptional transparency about its own errors
Robustness and mechanism, not just a leaderboard
The harness-disagreement result is itself a contribution
Weaknesses
Headline equivalence claim rests on a post-hoc margin
No multiplicity correction across dozens of contrasts
Selector comparison is partial
Selection-stage cost is unmeasured
Uneven generality and an unexplained anomaly
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: under a fixed visual-token budget for a long-video MLLM, how much accuracy does each allocation decision buy — which timestamps are kept (selection), how many pixels each kept frame gets (compression), and whether the savings buy more frames (reinvestment)? The author, an independent researcher, positions this against a literature (AKS, FOCUS, MDP3, LDDR, Q-Frame, DAFS, and others) in which these components move together, so published deltas confound the subset rule with scorer, prompt boundary, resolution policy, and answerer. Five claims emerge: (1) selection is the largest lever — OMP-8 beats uniform-16 by 6.9 points on hour-long LongVideoBench (p=.0011); (2) unmodified 1993 OMP stays within one point of every purpose-built selector tested; (3) halving spatial resolution costs at most 0.44 points, equivalenced within a post-hoc ±3-point TOST margin on pooled long LVB; (4) reinvestment returns 1.6–3.0 points at audited token parity on LVB; (5) two matched harnesses disagree by 0.07–3.74 points (Table 7), so cross-paper comparisons at this scale are fragile.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
In a controlled evaluation harness, choosing which frames to keep is the biggest accuracy lever for long-video multimodal LLMs: eight query-selected frames can beat sixteen uniformly spaced ones (by 6.9 points on hour-long LongVideoBench), halving each frame's resolution costs almost nothing (≤0.44 points), and reinvesting the saved tokens in more frames adds another 1.6–3.0 points.
Why does the paper use Orthogonal Matching Pursuit (OMP), a 1993 algorithm?
As a deliberately untuned, off-the-shelf baseline: it has no video-specific machinery or hyperparameters fitted to the benchmarks, so its performance measures how much modern purpose-built selectors gain from their subset rules rather than from their scorers and pipelines. OMP landed within one point of the best tested selector (LDDR's stage-1).
Is code and data available for this paper?
Yes. The release at github.com/codeprakhar25/omp-keyframe-sampling includes evaluation code, selector implementations, analysis scripts, selected frame indices for every rule and budget, per-item predictions, the Table 7 builder, and pre-registration documents; benchmark videos and API credentials are excluded.
What are the main caveats to the paper's claims?
The equivalence margin for 'compression is nearly free' was chosen after seeing the data; tests are uncorrected for multiple comparisons; FOCUS was tested only as a schedule replay and LDDR only at stage 1; the frame-selection stage's own compute cost is unmeasured; and effects are scoped to contrastive scorers (LongCLIP/SigLIP) and specific answerer–benchmark pairs — one model (InternVL3-8B) showed almost no selection gain on Video-MME's medium/long videos.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded text (axis and annotation labels in Figures 3, 5, and 6) is partially garbled by PDF extraction; all numbers cited in this review come from body text, tables, and figure captions, which extracted cleanly. The abstract snippet in the metadata is cut mid-sentence, but the embedded full text contains the complete sentence.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.