InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
Maeve Hutchinson, Syed Mahbubul Huq, Mohammad Albinhassan, Radu Jianu, Aidan Slingsby, Pranava Madhyastha
cs.CL · cs.CV · cs.HC
Quick verdict
First agentic claim-verification benchmark over interactive visualizations; well-designed and credible, but label noise and absent error bars temper results.
Summary
InSight is a benchmark for agentic claim verification over interactive visualizations: given a natural-language claim and a partially observable, web-based Vega-Lite environment, an agent must interact (scroll, click, hover, drag) within a bounded action budget to label the claim True, False, or Not Enough Information. The 21,349 claims derive from 297 human-authored analytical notebooks via a four-stage pipeline: LLM span extraction with cross-run agreement, atomic claim decomposition, controlled mutation (antonym and in-lexicon substitution for False; out-of-lexicon for NEI) validated by NLI thresholds, and expert validation (13 annotators, 475 annotations; raw agreement 81.3%; Dawid–Skene posterior mean accuracy 66.2%, MAP 77.6%). A proposed Interaction Efficiency Score multiplies accuracy by the fraction of state-changing actions, assigning zero to correct answers given without interaction. On a balanced 500-claim subset, GPT-5.5 leads at 57.2% accuracy (IES 26.98%), with clear capacity scaling in the Qwen and Gemma families; Gemini 3.5 Flash improves non-monotonically with budget (44.2%, 41.6%, 50.0% at Tmax 1/10/25). Falsification proves hardest, and small models skip tooltip-revealing hovers. Limitations include mutation-derived negatives validated only textually, a re-balanced evaluation subset diverging from the 13.4% NEI dataset prevalence, no confidence intervals on model comparisons, and human validation covering just 294 claims.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- InSight is a benchmark of 21,349 claims grounded in 297 human-authored interactive Vega-Lite notebooks, enabling agentic True/False/NEI verification in partially observable web environments.
- State-of-the-art models find interactive claim verification unsolved: the best configuration, GPT-5.5, reaches only 57.2% accuracy on a balanced 500-claim subset (random: 33.3%).
- The proposed Interaction Efficiency Score exposes correct-but-ungrounded behavior, e.g., Qwen 3.5 0.8B answers without any interaction and scores 0.00% IES.
- Falsification is disproportionately difficult relative to verification, extending a known finding from textual fact-checking to the agentic multimodal setting.
- Interaction traces differentiate model strategies: small models rarely hover, forfeiting access to tooltip-hidden evidence that constitutes the most prevalent interaction mechanism.
Strengths
Ecologically valid source data
Claims come from 297 expert-authored analytical notebooks with their interactive Vega-Lite environments preserved (§3.2, A.1), a deliberate contrast with synthetic templates (FigureQA/DVQA) and static crowdsourced images (ChartQA, ChartCheck).
Partial observability is engineered, not incidental
§3.3/§4.1: the viewport is deliberately smaller than the page and tooltips hide precise values, so by construction no claim is resolvable from the initial screenshot — the design explicitly penalizes passive perception.
IES metric with honest scope statement
Transparent, auditable construction pipeline
Statistically careful human validation
Weaknesses
Label validity rests on textual NLI, never the environment
Evaluation subset re-balanced away from dataset distribution
No uncertainty quantification for model results
Thin human-validation coverage relative to corpus
Shortcut behavior not fully excluded
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
InSight (arXiv:2609.01383) is a benchmark paper introducing agentic claim verification over interactive visualizations. Each instance pairs a natural-language claim with a partially observable, web-based Vega-Lite environment rendered in headless Chromium via Playwright; the agent sees only RGB screenshots (no DOM or accessibility tree, §4.1) and must act within a budget (default 10 actions) before labeling the claim True, False, or Not Enough Information (NEI). The corpus: 21,349 claims from 297 human-authored analytical notebooks (41.6% True, 45.0% False, 13.4% NEI; mean 18 tokens; 4.5 views and 3.7 specs per notebook on average). Central claims: (i) interactive verification is unsolved — GPT-5.5 tops out at 57.2% on a balanced 500-claim subset vs. 33.3% chance; (ii) the proposed Interaction Efficiency Score (IES) exposes ungrounded correctness; (iii) falsification is disproportionately hard; (iv) interaction traces function as traceable reasoning proxies.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the InSight benchmark?
InSight is a dataset and evaluation framework testing whether AI agents can verify natural-language claims about data by interacting with web-based visualizations (scrolling, clicking, hovering) rather than viewing static chart images. It contains 21,349 claims over 297 human-authored Vega-Lite analytical notebooks, each labeled True, False, or Not Enough Information (NEI).
How well do AI models perform on InSight?
The best model, GPT-5.5, reaches 57.2% accuracy on a balanced 500-claim subset with a 10-action budget — well above the 33.3% random baseline but far from reliable. Smaller models often answer from the first screenshot without interacting at all (e.g., Qwen 3.5 0.8B averages zero actions per episode).
What is the Interaction Efficiency Score (IES)?
IES multiplies answer accuracy by the fraction of the agent's actions that changed what was visible on screen. Correct answers given without any interaction score zero, by design: since no claim in InSight can be resolved from the initial view, such answers reflect guessing or parametric knowledge rather than visual evidence.
Where can I access InSight?
The authors state the benchmark is released at github.com/maevehutch/insight. Environments are standard Vega-Lite visualizations rendered via Playwright, and the paper reports that evaluation used a stratified 500-claim subset of the full 21,349-claim corpus.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-9 (interaction traces, dataset composition histograms) are image content; only their captions were extracted, so the visual trace analyses could not be independently inspected.
- Table 1 was extracted as flattened text; row-column alignment was reconstructed and cross-checked against in-text citations, but the flattened layout leaves minor residual alignment risk.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.