Navigating Sparse Evidence: Agentic Visual RAG via Explicit Context Selection and Consolidation
Yucheng Shen, Lingyong Yan, Lingyong Yan, Jiulong Wu, Shuaiqiang Wang, Shuaiqiang Wang, Jianmin WU, Jianmin WU, et al.
cs.AI
Quick verdict
SCORE makes visual-evidence selection and consolidation an explicit, trainable stage of agentic VRAG and reports SOTA on three benchmarks, but the gains rest on an LLM-judge metric with no variance estimates and partly non-comparable baselines.
Summary
SCORE targets visual retrieval-augmented generation over visually rich documents, where answer-bearing evidence is sparse - confined to a small region of one page or scattered across several - and where prior agentic methods answer from raw exploration trajectories or compressed textual memory. The core move is to make evidence organization explicit. During exploration the agent emits an observation, a binary relevance judgment, and one of three actions (search, bbox zoom, consolidate) at each turn; only relevance-filtered observations plus source pointers enter a persistent textual ledger, while a sliding window of W=2 raw visual turns bounds image-token growth. At termination, consolidate reloads the referenced original page images, denoises and reorders them, and generates the answer inside the same rollout, with indexed claim-to-image links. Training is two-stage: cold-start SFT on 2,500 teacher trajectories from Qwen3.5-122B-A10B retained only when the answer is judged correct and every gold page appears in the final ledger, then GRPO on 1,600 queries with a reward combining gold-page coverage, compactness, and answer correctness, where compactness and answer credit are both gated on full coverage. With Qwen2.5-VL-7B, SCORE reports 77.16 on SlideVQA, 75.31 on ViDoSeek and 31.52 on MMLongBench versus VISOR's 72.37, 74.87 and 28.45; the 3B backbone shows the same ordering. Ablations attribute gains to both modules and show cold-start SFT supplies most of the improvement (55.17 to 75.62, RL to 77.16). Limitations: accuracy is a binary LLM-judge score with no confidence intervals or significance tests, baselines mix quoted and reproduced numbers under different retrievers, the ViDoSeek margin is under half a point, and no code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Agentic VRAG is bottlenecked not by evidence discovery but by evidence preservation and organization before answer generation.
- Retaining only relevance-filtered observations with source pointers in a textual ledger, then reloading and reordering the original images at a consolidate step, decouples final reasoning from exploratory noise while keeping strict visual grounding.
- Cold-start distillation from coverage-filtered teacher trajectories plus an evidence-aware GRPO reward that gates compactness and answer credit on full gold-page coverage yields state-of-the-art accuracy on SlideVQA, ViDoSeek and MMLongBench at both 3B and 7B scales.
- Explicit organization beats retrieving more pages: SCORE reaches the highest accuracy while keeping the fewest non-gold pages (0.16 per query) and the fewest retrieved pages (1.76).
- Relevance judgment and consolidation are complementary, and training shifts the denoising burden from the consolidation exit to the collection-time entrance.
Strengths
Explicit, testable mechanism for the coverage-noise trade-off
Table 3 quantifies exactly what the design is supposed to fix: SCORE reaches 77.16 accuracy while retaining only 0.16 non-gold pages per query, versus 1.29 for VISOR and 8.81 for ViDoRAG, and it does so with the lowest average retrieval count (1.76 pages vs 2.34 and 10 fixed). The mechanism claim is thus backed by an intermediate metric, not only end accuracy.
A well-gated reward that blocks an obvious reward hack
Eq. (4) ties both the compactness term (lambda=0.2) and the answer reward to I[coverage=1] and applies a beta=1 penalty otherwise, so the agent cannot raise precision by discarding evidence nor earn answer credit with an incomplete ledger. The paper reports the resulting behavior: post-hoc re-fetch raises coverage only 78.4 to 78.5 and drops accuracy 77.16 to 76.79 (Table 10), i.e. the policy already errs toward retention.
Ablations are retrained, not merely disabled at inference
Honest reporting of a negative result and a backbone failure mode
Efficiency is measured, not asserted
Weaknesses
Every headline number is a binary LLM-judge score with no dispersion
Baselines are not measured under one protocol
Training and strongest evaluation are the same distribution
Efficiency table disagrees with the main table and is a small subsample
No code, data or teacher-trajectory release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SCORE enters a crowded corner of multimodal retrieval: agentic Visual Retrieval-Augmented Generation (VRAG), where a vision-language model iteratively retrieves rendered document pages and reasons over them. The paper's diagnosis is sharp and largely convincing. It names two failure modes - uneven evidence distribution (relevant content confined to a small region of one page, or scattered across many) and implicit evidence organization (answers generated from raw exploration trajectories or compressed textual memory) - and argues the bottleneck is not discovery but preservation and organization of evidence *before* answer generation. That reframing is the paper's intellectual contribution, and it is sharper than the usual "add RL to an agent loop" framing.
The concrete claims are: (1) a unified agent loop with a persistent textual evidence ledger plus a sliding window of W=2 raw visual turns; (2) a consolidate action that reloads the original page images referenced by the ledger, re-selects and orders them, and answers inside the same rollout with indexed claim-to-image links; (3) a two-stage training recipe - cold-start SFT on coverage-filtered teacher trajectories, then GRPO with an evidence-aware reward; and (4) state-of-the-art accuracy on SlideVQA, ViDoSeek and MMLongBench at both 3B and 7B scales. Claims (1), (2) and (4) are supported; claim (3) is supported but unevenly weighted, as discussed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SCORE in visual retrieval-augmented generation?
SCORE is an agent framework for answering questions about visually rich documents (slides, reports, scanned PDFs). It searches a pool of page images, zooms into bounding boxes when detail is illegible, keeps only the observations it judges relevant in a running textual ledger, and at the end reloads the original images, reorders them into a logical evidence chain, and answers from them with each claim indexed to its supporting image.
How much does SCORE improve over previous visual RAG methods?
With a Qwen2.5-VL-7B backbone the authors report 77.16% accuracy on SlideVQA, 75.31% on ViDoSeek and 31.52% on MMLongBench, compared with 72.37%, 74.87% and 28.45% for the prior best method VISOR. Gains are largest on multi-hop SlideVQA questions (62.26 vs 53.62) and smallest on ViDoSeek, where every question is answerable from a single page.
Is SCORE's accuracy advantage statistically significant?
The paper does not establish this. All scores are means of binary judgments by an LLM judge (Qwen-max-latest), and no confidence intervals, standard deviations, seed variation or significance tests are reported. The ViDoSeek margin is only 0.44 points over 1,142 questions, which is well within what judge noise alone could produce.
Is the SCORE code publicly available?
No. The paper does not announce a code, data or model release, and reproducing it would require a 122B teacher model for trajectory distillation, the SlideVQA training split, and roughly 8x NVIDIA A800 80GB GPUs for the GRPO stage. Independent verification is therefore not currently feasible.
What are the main limitations of SCORE?
Three stand out. Its accuracy rests entirely on an LLM judge with no variance reporting. Its gains are concentrated on SlideVQA, the same dataset its training data comes from, and shrink to 0.44 points on ViDoSeek. And under prompting alone it actively hurts the smallest backbone tested: on Qwen2.5-VL-3B, full SCORE scores 28.58 versus 42.26 for a variant without selection and consolidation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 are not analyzable from the extracted text; Figure 3 (Pareto front) and Figures 8-9 (case studies) are read only through their captions and the accompanying narrative.
- The extracted text is complete through the appendix, so no section is missing, but numeric claims could only be verified against the tables as rendered in text; table column alignment for Table 1 was reconstructed from context.
- Several referenced items (Wang et al. 2026 VRAG-RL, Shen et al. 2026 VISOR, Qwen Team 2026) are contemporaneous or future-dated; their claimed results are taken as reported and cannot be independently checked without network access.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.