Evaluation of MLLM-Agnostic Plug-and-Play Keyframe Selection Methods for Long Video Understanding
Dilip Sarkar, Md. Safayet Islam, Liang Liang
cs.CV · cs.AI
Quick verdict
First head-to-head of five PaP keyframe selectors; authors' own QAaF wins 13/15, but an unpublished core method and encoder/budget confounds weaken it.
Summary
This paper presents the first unified head-to-head evaluation of five training-free, plug-and-play (PaP) keyframe selection methods for long-video understanding: Q-Frame, AKS, FOCUS, MaxInfo, and QAaF. The authors run all five with three MLLMs (Qwen2-VL-7B, LLaVA-Video-7B, InternVL3.5-8B) on three benchmarks (MLVU, Video-MME, LongVideoBench), fixing the frame budget at k=16 (Q-Frame keeps its native 4+8+32 multi-resolution setting), using BLIP for AKS/FOCUS and CLIP-family encoders for the others. The headline result is that QAaF — which scores frames by cosine similarity against both the query and concatenated query-answer texts — ranks best in 13 of 15 aggregate settings and second in the other two, with FOCUS second overall; QAaF also improves every MLLM over uniform sampling (e.g., +8.8 MLVU points for Qwen2-VL). Limitations are substantial: QAaF is the authors' own method and its technical report is not yet publicly available; AKS/FOCUS use a different vision encoder than the other methods and Q-Frame uses a different frame budget, confounding the comparison; no significance tests or variance estimates accompany margins as small as 61.93 vs 61.81; and the InternVL3.5 MLVU cells for QaF/QAaF show anomalous category patterns (ER around 40, AC around 76) suggesting a pipeline error. No code release is stated, and no runtime costs are reported despite efficiency being the core motivation.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Under a unified protocol (k=16 frames, three MLLMs, three benchmarks), QAaF is the best of five PaP keyframe-selection methods in 13 of 15 aggregate evaluation settings and second-best in the remaining two (Table 2).
- FOCUS ranks second overall among the five methods, with one best and seven second-best aggregate results (Sec. 4.2).
- Incorporating candidate answers into frame-query alignment (QAaF) generalizes across MLLM architectures and is particularly effective with subtitles and on long videos.
- QAaF improves all three MLLMs over uniform 16-frame sampling, e.g., MLVU averages rise by 8.8, 7.8, and 5.11 points for Qwen2-VL, LLaVA-Video, and InternVL3.5 respectively (Sec. 4.2.1).
- No single method dominates all task categories; Q-Frame in particular is highly model-dependent (Sec. 4.2.1).
Strengths
Fills a real gap: first same-protocol head-to-head of the five PaP selectors
Sec. 1/Table 1 show the five methods were previously evaluated on disjoint benchmark/MLLM combinations; Sec. 4.1 and Tables 3-5 put Q-Frame, AKS, FOCUS, MaxInfo, and QAaF on the same three benchmarks (MLVU, Video-MME, LongVideoBench) and three MLLMs (Qwen2-VL-7B, LLaVA-Video-7B, InternVL3.5-8B), 546 result entries.
Unusually explicit evaluation protocol
Sec. 4.1 states the fixed k=16 frame budget, Q-Frame's native 4+8+32 multi-resolution input, BLIP for AKS/FOCUS versus CLIP-family for the rest, identical MLVU category sets, and both subtitle settings for Video-MME/LongVideoBench, plus the hardware (4x H100 NVL, 1.5 TB RAM).
Headline claim is internally consistent and verifiable from the tables
Honest task-level nuance
Clean problem formalization
Weaknesses
The winner is the authors' own, unpublished method
Encoder and frame-budget confounds in the comparison
No statistical treatment of small margins
Anomalous InternVL3.5 MLVU cells for QaF/QAaF
No efficiency measurements and no code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This arXiv preprint (2609.13250) positions itself as the first head-to-head evaluation of the five training-free, MLLM-agnostic plug-and-play (PaP) keyframe-selection methods reported in the past year: Q-Frame (ICCV 2025), AKS (CVPR 2025), FOCUS (ICLR 2026), MaxInfo (WACV 2026), and QAaF — the authors' own technical report [9], not yet publicly available. Sec. 2 formalizes frame selection as maximizing answer confidence over k-subsets (Eq. 1), states a frame-level cooperative assumption (Assumption 1, borrowed from n-person cooperative game theory), and derives a greedy top-k selection rule (Lemma 1). Sec. 4 evaluates all methods on MLVU, Video-MME, and LongVideoBench with Qwen2-VL-7B, LLaVA-Video-7B, and InternVL3.5-8B. The central claim (Sec. 4.2, Table 2): QAaF is best in 13 of 15 aggregate settings (per MLLM: MLVU plus both subtitle conditions of Video-MME and LongVideoBench) and second in the other two, with FOCUS second overall (one best, seven second-best). I re-derived both tallies from Tables 3-5 and they check out exactly; the two non-wins are LLaVA-Video on Video-MME without subtitles (AKS 61.93 vs QAaF 61.81) and InternVL3.5 on Video-MME without subtitles (FOCUS 61.56 vs QAaF 60.89). The internal bookkeeping is consistent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which keyframe selection method won in this comparison?
The paper reports QAaF — the authors' own query-and-answer-aligned frame selection method — as best in 13 of 15 aggregate settings across MLVU, Video-MME, and LongVideoBench with three MLLMs, with FOCUS second overall. An important caveat: QAaF's underlying technical report was not yet publicly available when the paper was written, so the winning method could not be independently inspected.
What benchmarks and models does the paper evaluate?
Three long-video QA benchmarks (MLVU, Video-MME, LongVideoBench) crossed with three open multimodal LLMs: Qwen2-VL-7B, LLaVA-Video-7B, and InternVL3.5-8B. Every method selects 16 frames, except Q-Frame, which uses its native multi-resolution input of 4+8+32 = 44 frames. Both with-subtitle and without-subtitle settings are reported for Video-MME and LongVideoBench.
Is code available for this evaluation?
The paper announces no code or data release of its own. It states that evaluations were based on the five methods' own released evaluation code with necessary adaptations, and the QAaF method paper is described only as a technical report 'to be uploaded to arXiv soon'.
Does keyframe selection actually help long video question answering?
In this study, query-aware selection helped: QAaF improved all three MLLMs over uniform 16-frame sampling — for example, MLVU average accuracy rose by 8.8 points for Qwen2-VL (60.4 to 69.2) and 7.8 points for LLaVA-Video (63.5 to 71.3) — with the largest gains on long videos and when subtitles are available. The query-agnostic MaxInfo and the model-sensitive Q-Frame were noticeably less consistent.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics (Figs. 1-3) are not assessable beyond their captions and embedded axis labels; the qualitative examples cannot be visually verified from the text extraction.
- All quantitative claims above were verified internally against the extracted Tables 2-5 and Sec. 4 text; no external verification against the five original method papers or benchmark documentation was performed (network access disabled).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.