NoteVQA: Benchmarking VLMs on Real-Life Questions from Human Communities
Haonan Jiang, Guojian Zhan, Jiancong Xie, Shijun Wan, Dongiia Zhao, Cheng Chen, Yahui Liu, Chuan Mu
cs.AI
Quick verdict
Claims everyday VQA stumps frontier VLMs (top 52.8%); well-built 252-item benchmark, but self-judging by Qwen backbone and small N temper conclusions.
Summary
NoteVQA is a 252-item benchmark of real visual questions posted by users on Xiaohongshu, a Chinese image-sharing platform, spanning 12 topical categories and 7 intents such as diagnosis, authenticity checking, and spoilage. From 5.24M raw posts, a four-stage pipeline (type cleaning, rubric filtering, refinement with a ten-run difficulty probe, human auditing) selects items, each paired with a short expert-distilled reference and a human-audited interleaved reference combining text with 378 supporting images. The paper contributes AgenticInterleave, a ReAct agent that generates retrieval-supported interleaved answers, and IVR-12, a 12-dimension rubric scoring content, presentation, and image quality, applied identically to references and model outputs. Nine frontier VLMs reach at most 52.8% short-answer accuracy (Gemini-3.1-Pro); agentic retrieval lifts Qwen3.5-397B-A17B by only 2.0 points (35.7% to 37.7%), with failures attributed chiefly to confirmation loops (111 of 157 incorrect runs). AgenticInterleave scores 3.52 under IVR-12 versus 4.65 for the human references, with content quality the largest gap. An image ablation shows embedded images raise reader accuracy on image-critical probes from 20.7% to 97.0% while text-answerable controls stay near 98%. Limitations: the same Qwen model serves as agent backbone, judge, and grader; 252 items give small per-category cells; no inter-annotator agreement or uncertainty estimates for the main accuracy numbers are reported.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier VLMs reach at most 52.8% short-answer accuracy (Gemini-3.1-Pro) on 252 real-user visual questions from Xiaohongshu, spanning 12 categories and 7 intents.
- Agentic retrieval improves Qwen3.5-397B-A17B by only 2.0 percentage points (35.7% to 37.7%) on NoteVQA, far below the 14-43 point gains observed on 10 comparison benchmarks.
- AgenticInterleave's interleaved answers score 3.52 under IVR-12 versus 4.65 for human-audited references, with content quality (accuracy 2.68, source fidelity 2.07 gaps) the largest shortfall.
- Embedded images in interleaved references carry information absent from the text: reader accuracy on image-critical probes rises from 20.7% to 97.0% when images are shown, while text-answerable controls stay at 97.9-98.4%.
- Confirmation/anchoring loops are the dominant agentic failure mode, accounting for 111 of 157 incorrect runs (70.7%), with tool-related failures at 86.6%.
Strengths
Genuine user-need grounding
Questions come from real Xiaohongshu posts where users seek community help, with a documented 5.24M-to-252 funnel (Section 3.1, Figure 3); this contrasts with capability-first benchmarks and is supported by the Figure 2 comparison showing only +2.0pp agentic gain here versus 14-43pp on 10 other benchmarks.
Dual reference formats with a symmetric rubric
Each item has a short reference (mean 107.5 words) and a human-audited interleaved reference (219.4 words, 378 images), and IVR-12 (Appendix E) grades gold and model outputs under identical prompts, including an anti-gaming rule that scores zero-image answers as 1 on all image dimensions.
Honest negative results and failure diagnostics
Well-controlled image ablation
Unusually complete documentation
Weaknesses
Five-way judge/model circularity
No uncertainty on headline accuracy comparisons
Subjective ground truth, unquantified annotation quality
Small, imbalanced cells
English translation of Chinese-context queries
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
NoteVQA (arXiv 2609.15695v2) is a benchmark paper from the self-identified AllSpark Team targeting the gap between capability-first multimodal QA benchmarks and the everyday visual questions real users ask. It curates 252 questions from Xiaohongshu posts (12 topical categories, 7 intents such as diagnosis, authenticity checking, and spoilage), each with two references: an expert-distilled short answer (mean 107.5 words) and a human-audited interleaved answer (mean 219.4 words, 378 embedded images total). The paper claims: (1) frontier VLMs top out at 52.8% short-answer accuracy (Gemini-3.1-Pro, Table 2); (2) agentic retrieval adds only +2.0pp to Qwen3.5-397B-A17B (35.7% to 37.7%), versus 14-43pp gains on 10 comparison benchmarks (Section 2, Figure 2); (3) its AgenticInterleave generator scores 3.52 under the new IVR-12 rubric versus 4.65 for human references, with content quality the dominant gap (Table 3); and (4) embedded images carry information absent from text, with image-critical probe accuracy rising from 20.7% to 97.0% when shown (Table 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is NoteVQA and where do its questions come from?
NoteVQA is a 252-question benchmark built from real posts on Xiaohongshu, a Chinese image-sharing platform, where users photograph objects and ask the community for help. Questions cover 12 everyday categories (pets, plants, health, jewelry, food, electronics, etc.) and 7 intents such as diagnosis, authenticity checking, and spoilage assessment. Each question is paired with a short expert answer and a human-checked illustrated answer.
Why do frontier VLMs score only around 50% on NoteVQA?
The questions target niche, real-world visual judgments - telling similar species apart, spotting counterfeits, judging spoilage - grounded in Chinese community knowledge. The best model (Gemini-3.1-Pro) reaches 52.8% short-answer accuracy. Adding a web-search agent to Qwen3.5-397B-A17B helps only about 2 points, largely because retrieval rarely surfaces useful evidence (a 0.12% useful-hit rate on 1623 web searches) and agents often lock onto an early wrong guess (confirmation loops caused 111 of 157 failures).
What is IVR-12?
IVR-12 is a 12-dimension rubric the authors introduce for grading illustrated (interleaved text-and-image) answers. It scores five content dimensions, three presentation dimensions, and four image dimensions on a 1-5 scale, combines them with 0.5/0.2/0.3 weights, and sets the score to zero if any of three pass/fail gates (timeliness, safety redlines, refusal correctness) fails. The same rubric is applied to both human references and model outputs.
Is the NoteVQA dataset and code publicly available?
The paper points to a GitHub repository (github.com/AllSpark-Research/notevqa) and states that the benchmark, evaluation prompts, model outputs, and code for the generation framework and data pipeline will be released. As with any preprint release promise, actual availability should be verified directly in the repository.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on the extracted full text including appendices; Figures 1-5 and the trajectory figures are only partially recoverable from text, and the per-benchmark reasoning-vs-agentic bar values in Figure 2 cannot be reliably attributed to specific benchmarks.
- The GitHub repository's actual release status could not be verified (no network access during review); codeAvailable reflects the paper's stated repository and release promise.
- The paper contains an unresolved cross-reference ('Appendix ??') in Section 3.3 and minor typos (e.g., 'Answerablity' in Figure 3), suggesting the judge prompts referenced exist in Appendix E regardless.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.