PPTBench: Can Coding Agents Reconstruct the Visual World through Structured, Editable Slides
Xiaoqiu Wang, Yizhe Chi, Wenyi Li, Deyao Hong, Zhihan Shan, Mingju Gao, Kaisen Yang, Youjie Zheng, et al.
cs.CL · cs.AI
Quick verdict
PPTBench: 500 arXiv diagram-to-editable-slide tasks with a validated agentic judge; best agent 67.8/100; rigorous but harness-model confound persists.
Summary
The paper introduces PPTBench, a benchmark measuring whether coding agents can reconstruct the visual world: each of 500 tasks supplies a scientific flow diagram from a real arXiv paper, and the agent must rebuild it as a single PPTX slide of native, editable objects rather than embedded images. A four-stage Agentic Judge evaluates each submission: a deterministic artifact-validity gate, a semantic gate on nodes, connector endpoints and arrow directions, a render/text gate, and a detail stage whose structured findings a fixed program converts into points; the gates are decided by majority vote of three independent GPT-5.6 Luna High runs. Validation on 200 blinded human-labeled cases yields 86.5% agreement with the human majority (Cohen's κ 0.696 vs 0.728 among humans). Across 31 model-harness-effort configurations, the best (Kimi K3, high effort) scores 67.80/100 and the median 19.47. Nearly all runs produce valid files (97.92%), but 70.43% fail the semantic gate, and text defects such as unintended wrapping carry 51.6% of post-gate deductions. Extra reasoning mainly buys gate passes, not finer drawing, and self-inspection correlates with score (r=0.881). Main limitations: model-harness coupling, a judge from the same family as the evaluated GPT-5.6 systems, and weak reproducibility of itemized detail findings.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current coding agents write the PPTX format fluently but read diagrams poorly: 97.92% of 15,500 model-task pairs pass the deterministic artifact gate while 70.43% fail the semantic gate.
- A four-stage agentic judge that only gates and enumerates findings (never assigning scores) matches blinded human majority decisions at Cohen's κ = 0.696, just below the 0.728 human-human ceiling.
- Additional reasoning effort raises hard-gate pass rates by up to 46.8 points but leaves conditional detail quality nearly unchanged (Luna: 85.28 → 85.23), and maximum effort is not a safe default (two of five swept families worse or no better at max).
- Agents' mean score rises with how often they inspect their own renders (r = 0.881) while edit count carries almost no signal (r = 0.255), and text-fitting failures — led by unexpected_wrap at 5,002.0 penalty points — carry 51.6% of post-gate deductions because they are invisible in the OOXML document layer.
Strengths
Staged gating with judge that never scores
Stage 0 is deterministic; Stages 1-2 are hard gates by majority of three GPT-5.6 Luna High runs; Stage 3 only enumerates findings that frozen arithmetic (severity bases 3.0/1.5/0.5/0, ratio multipliers, caps 30/40/30, Eq. 4-7) converts to points. A reversed arrow zeroes the score (Figure 8a, 9), which pixel or LPIPS metrics would pass.
Exemplary judge validation
Section 3.4: 200 blinded cases, 86.5% agreement with human majority (Wilson CI 81.1-90.6%), Cohen's κ 0.696 vs human Fleiss' κ 0.728 ceiling, gate F1 0.899, strict-direction error quantified (15 over-gates vs 12 missed, Table 3), and three-round-majority repeatability of 96.0% (κ 0.908, 8/200).
Honest ranking-resolution reporting
Large paired evaluation with clean attribution
Actionable behavioral finding with causal caveats
Weaknesses
Model-harness coupling confounds the leaderboard
Judge-family self-exposure
Judge development and validation live inside the frozen test set
Detail layer weakly attested
Training-data contamination not probed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PPTBench (arXiv:2609.29718) frames visual coding as reconstructing a fixed visual target as structured, editable code. Its 500 frozen tasks each supply one scientific flow diagram from a real arXiv paper; the agent must emit a single PPTX page of native, editable objects, and the contract mechanically rejects any rasterized copy of the reference (Section 2.1). The central claim is an attribution result: agents write the slide format fluently but read the picture poorly — 97.92% of 15,500 model-task pairs pass the deterministic artifact gate while 70.43% fail the semantic gate (Figure 7a). Secondary claims: a judge that never assigns scores reproduces human decisions near the human-human ceiling; reasoning effort buys gate passes, not finer drawing; self-inspection of renders, not edit count, tracks quality.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PPTBench?
PPTBench is a benchmark of 500 tasks in which a coding agent is shown one scientific flow diagram from a real arXiv paper and must rebuild it as a single PowerPoint (PPTX) slide made entirely of native, editable objects — shapes, text frames, connectors — rather than an embedded picture, so the structure the agent inferred is recorded in the file it submits.
What are the key results of PPTBench?
Across 31 model-harness-effort configurations, the best system (Kimi K3 at high effort) scores 67.80 out of 100 and the median configuration 19.47. Nearly all runs (97.92%) produce a valid editable file, but 70.43% fail the semantic gate (wrong nodes, connections or arrow directions), and text defects such as unintended wrapping account for 51.6% of the remaining deductions.
How is PPTBench scored?
A four-stage judge: Stage 0 deterministically checks the file is a valid single-page PPTX of native objects without a rasterized copy of the reference; Stage 1 checks process semantics; Stage 2 checks legibility after rendering; Stage 3 enumerates localized visual defects that fixed program logic converts into points. The hard gates are decided by majority vote of three independent GPT-5.6 Luna High runs, and failing a gate zeroes the score.
Is PPTBench's code and data available?
The paper states the benchmark is released with a downloader-only materializer that reproduces the reference images and approved resources locally, including full provenance and pinned rendering specifications; source-paper pixels are not redistributed. The release status of the judge implementation and prompts is not clearly stated in the text.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Text embedded inside Figures 1-2 is partially garbled in the PDF extraction, so figure-only content (e.g. the Figure 1 chart values) could not be verified; all numbers cited in this review come from body text and tables, which extracted cleanly and completely through the appendices.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.