CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
Dingli Liang, Yiqiao Xie, Yukai Huang, Zhaokai Wang, Weitong Cai, Guangwen Feng, Jifei Song, Zhensong Zhang, et al.
cs.AI · cs.CV
Quick verdict
A credible benchmark: captions beat direct VideoQA on long egocentric videos for most of 12 VLMs, but controlled gains are Qwen-only and partly null.
Summary
The paper introduces CapMem, a human-annotated benchmark asking whether VLM-generated captions can serve as episodic memory for long egocentric video QA. It defines Episodic Memory Video Caption QA over 75 public videos (33.7 hours, 16 scenarios) with 1,000 four-option multiple-choice questions written by 12 annotators and filtered via a four-model text-only ensemble, manual visual-solvability checks, and a CleanQA blind gate. Evaluating 12 VLMs, full-coverage caption QA (30s/60s windows) beats direct VideoQA on >20-minute videos for 10/12 (30s) and 8/12 (60s) models. A frame-aligned control on six Qwen models retains mean long-video gains of +3.22 (30s) and +2.55 (60s) points, with only the 30s confidence interval excluding zero; a caption-guided retrieve-and-verify harness adds up to +5.3 points for open-weight models. Caption memory amortizes visual processing (7.1x fewer frame inputs than per-question VideoQA). Main limitations: controls restricted to one model family, 60s and full-set CIs include zero, the harness override policy was tuned on the evaluation split without uncertainty estimates, the visual baseline is uniform frame sampling, and Ego4D contributes 742/1000 questions.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA on videos longer than 20 minutes for 10/12 and 8/12 of the evaluated models, respectively.
- After matching visual frames, caption memory retains mean long-video accuracy gains of 3.22 (30s) and 2.55 (60s) points across six Qwen models, with statistical significance only in the 30s setting.
- A caption-guided retrieve-and-verify harness improves accuracy by up to 5.3 points for open-weight models by using captions as a temporal index and sparse frames for verification.
- Caption memory amortizes visual processing across queries, reducing frame inputs 7.1x relative to per-question Direct VideoQA (101,814 vs. 725,973 frame inputs).
- The caption advantage is duration-dependent: direct VideoQA remains stronger on videos of 20 minutes or less (60s aligned gap -5.29, CI [-9.17,-1.29]).
Strengths
Novel, well-posed evaluation target
First dedicated benchmark (Secs. 1-3) testing caption-as-episodic-memory against direct VideoQA, with a formally defined task and a CleanQA blind gate that empirically confirms shortcut removal (all 12 models score 25.6-30.9%, near the 25% chance level, Table 3).
Multi-layer shortcut and quality control in benchmark construction
Sec. 4.3 / App. E / Table 9: 2,109 candidate QA pairs -> 1,276 after two-reviewer consensus -> 924 after a four-model (GPT-5.2, Gemini 3, Qwen3.5, InternVL) text-only ensemble filter -> 1,000 after reviewers restored 76 false positives and verified visual solvability; more rigorous than the LLM-generated QA benchmarks the paper criticizes.
Confound-aware experimental design
Quantified practical value, not hand-waving
Unusually detailed reproducibility documentation
Weaknesses
Statistical support for headline gains is thinner than the abstract implies
Controlled evidence limited to one model family
Weak instantiation of the visual baseline
Harness evaluation is post-hoc and unaudited for uncertainty
Benchmark validation gaps and source skew
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CapMem benchmark?
CapMem is a human-annotated benchmark of 1,000 four-option multiple-choice questions over 75 public egocentric videos (33.7 hours, 16 scenarios) that tests whether captions generated by a vision-language model can act as the model's episodic memory: the model first captions the whole video in 30- or 60-second timestamped windows, then answers questions using only that text instead of video frames.
Do captions work better than feeding video frames directly to the model?
It depends on video length. On videos longer than 20 minutes, caption-based QA beat direct video QA for 10 of 12 tested models with 30-second windows; after equalizing the number of frames both settings see, mean accuracy across six Qwen models was still 3.22 points higher with captions (statistically significant) and 2.55 points with 60-second windows (not significant). On videos of 20 minutes or less, direct video input was better.
Why use captions instead of raw video frames for long-video memory?
Captions are computed once per video and reused for every question, so repeated visual processing is amortized: the paper counts about 7.1x fewer frame inputs than answering each question directly from frames, and text avoids frame-budget limits and long-context degradation such as the lost-in-the-middle effect.
Is the CapMem dataset and code publicly available?
Yes, with caveats: the questions, answers, evidence timestamps, labels, prompts, evaluation scripts, and retrieve-and-verify scripts are released (annotations under CC BY-NC 4.0, code under Apache 2.0), but the source videos are not redistributed; users must obtain them from the seven source datasets (e.g., Ego4D, EPIC-KITCHENS) under their own licenses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were available only as numbers embedded in the extracted text; visual figure content (e.g., Figure 5 evidence-location profiles) could not be independently verified.
- The supplied metadata omits co-author Hang Zhang, who appears as a corresponding author in the paper text.
- Gemini 3 Flash VideoQA was run on a 786-question subset, and GPT-5.2/Gemini CaptionQA output limits were reconstructed post-hoc from truncation boundaries, so closed-source rows cannot be exactly replicated.
- Release artifacts (annotation files, scripts) were not inspected; availability claims rest on Appendix G and licensing tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.