SVMemAgent: A Streaming Video Memory Agent for Query-Agnostic Online Frame Selection
Dohwan Ko, Ji Soo Lee, Pierce Chuang, Debojeet Chatterjee, Ashish Shenoy, Yichao Lu, Seungwhan Moon, Xin Luna Dong, et al.
cs.CV
Quick verdict
GRPO-trained streaming memory agent for query-agnostic frame selection; small but consistent gains over online baselines; offline AKS still leads on average.
Summary
The paper tackles keyframe selection for long-video QA in a strict streaming setting: frames arrive at 1 FPS, total duration is unknown, and the user query is unavailable during selection. It introduces SVMem, a K-frame memory (K=16), updated by SVMemAgent, a 196M-parameter dual-branch policy (DINOv3-Base semantic branch plus timestamp-only temporal branch) that at each timestep decides to discard the incoming frame or replace a memory slot. Training combines cold-start on heuristic pseudo-labels (DINOv3-Huge teacher) with GRPO using a task reward — the frozen VideoLLM's average log-probability of gold answers over four queries per video — plus Diversity-Aware Advantage Discounting (DAAD), which down-weights rollout groups whose rewards barely vary despite diverse memories. On VideoMME, LongVideoBench, EgoSchema, and RVS-Ego with InternVL3.5-8B and Qwen3-VL-8B, SVMemAgent beats strict-online baselines by 0.9-2.3 average points, and DAAD adds 0.5-1.1. It runs at 16.2 ms/frame (61.7 FPS). Main limitations: margins are small with no variance or significance reporting; the query-aware offline method AKS still outperforms it on both backbones; several key generalization analyses (budget scaling, query arrival times, novelty) appear only in figures.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SVMemAgent maintains a K-frame streaming memory under strict online constraints (unknown duration, no query access) and outperforms all evaluated online baselines across four benchmarks and two VideoLLMs (Table 2).
- GRPO with task-driven rewards teaches a query-agnostic selection prior, and DAAD suppresses rollout groups whose rewards are insensitive to memory content (language-biased queries, static scenes).
- Despite weaker information access, SVMemAgent is competitive with offline methods, though the strongest query-aware baseline (AKS) still outperforms it on both backbones (Table 3).
- The trained policy emergently prefers text-bearing frames (their share of memory rises from 33.8% to 40.8%), which the authors link to downstream VideoQA benefit.
- The policy is efficient (16.2 ms/frame, 61.7 FPS) and robust to memory budget, query arrival time, query novelty, and reward-model choice.
Strengths
Clear problem formalization with an honest baseline taxonomy
Table 1 classifies every baseline by whether it assumes known video duration and/or query access; the strict-online protocol (1 FPS, unknown length, query-agnostic, fixed memory budget) is stated precisely in Sec. 3.1 and Sec. 4, making the comparison axes and fairness conditions legible.
RL applied to an external memory policy rather than the VideoLLM, with evidence of transfer
Unlike Video-R1/Visual-RFT (Sec. 2), GRPO trains a 196M-parameter selector. Trained with an InternVL3.5-2B reward, it still improves Qwen3-VL-8B results (Table 2), and swapping the reward model to Qwen3-VL-2B shifts averages by only 0.1-0.3 points (Table 6), suggesting a backbone-agnostic prior.
DAAD targets a real failure mode and is supported by concrete diagnostics
Streaming-viable cost plus unusually broad ablation coverage
Weaknesses
Small margins with no uncertainty quantification
The 'competitive with offline methods' claim is weaker than Table 3 shows
Reward is a proxy that the paper itself shows can be insensitive
The semantic branch contributes little beyond the temporal branch
Reproducibility gaps and figure-only evidence for key claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SVMemAgent and what problem does it solve?
SVMemAgent is a lightweight (about 196M-parameter) policy that maintains a small memory of up to 16 frames while a video streams in at 1 frame per second. Frame by frame, and without knowing the user's question or the video's total length, it decides whether to discard the incoming frame or use it to replace a stored frame. When a question finally arrives, the stored frames are handed to a video-language model to answer. This targets real-time settings such as wearables and embodied AI, where the full video is never available in advance.
How is SVMemAgent trained?
In two stages. First, cold-start supervised training on heuristic pseudo-labels: a DINOv3-Huge teacher scores frames for relevance and novelty to produce discard/replace targets, while a temporal module targets uniform time coverage. Second, reinforcement learning with GRPO: candidate memory states are scored by how well a frozen video-language model can produce the gold answers to four questions per video (average token log-probability). A technique called DAAD down-weights training groups where different memories yield nearly identical rewards, which typically means the question can be answered from language bias alone.
How well does SVMemAgent perform compared to other methods?
On four benchmarks (VideoMME, LongVideoBench, EgoSchema, RVS-Ego) with two 8B video-language models, it beats prior online frame-selection methods (FIFO, Reservoir, TimeChat-Online, Flash-VStream, MA-LMM) by roughly 1-2 average points. Against offline methods that see the full video and the question, results are mixed: it beats several, but the query-aware AKS method still scores higher on average with both backbones. It processes each frame in about 16 ms (about 62 frames per second), fast enough for real-time streaming.
Is code or data available for SVMemAgent?
The arXiv v1 text contains no code or dataset release statement. Reported implementation details include the DINOv3-Base encoder, K=16 memory size, GRPO group size 32, and training on Video-R1 and LongVILA subsets (videos at least 64 seconds), but pseudo-label thresholds and some hyperparameters are not specified, so independent reproduction would currently require contacting the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are not analyzable from the extracted text (captions only); budget-scaling, query-arrival-time, and query-novelty results could not be verified numerically.
- Possible appendix/supplementary material (full hyperparameters, per-benchmark DAAD ablation) is not included in the provided text and may address some noted gaps.
- Review covers arXiv v1 (2609.18540); reported numbers were not independently verified against the PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.