ReFigBench: Benchmarking Scientific Figure Reconstruction as Editable PowerPoint Artifacts
Liyang Fan, Liyang Fan, Chi Wei, Yitai Li, Yitai Li, Xinping Bi, Xinping Bi, Guhong Chen, et al.
cs.CL · cs.CV
Quick verdict
1,000-figure benchmark for editable PPTX reconstruction; a careful paired study shows harness and workflow jointly decide agent quality; broadly convincing, with judge-family caveats.
Summary
ReFigBench tests whether multimodal coding agents can turn a raster scientific overview figure into a single-slide, natively editable PowerPoint artifact that preserves text, topology, layout, and document structure. Its 1,000 figures come from ORBIT, a caption-gated, provenance-complete arXiv retrieval procedure over cs.AI/CL/CV/LG. Ten configurations - four model families (GPT-5.5, Claude Opus 4.6, MiMo-v2.5, MiniMax-M3) under a direct python-pptx workflow versus a specialized PPTX-skill workflow, with GPT-5.5 additionally run in both Claude Code and Codex - produce 10,000 artifacts. Scoring combines a deterministic artifact gate and raster-paste cap, a five-axis 100-point rubric (text 20, structure 30, layout 15, editability 25, visual 10) applied by a pinned GPT-5.4 judge across three runs and by Qwen3.6-27B as a second judge family, and 4,270 blinded human pairwise comparisons fit with Bradley-Terry Elo. The best configuration scores 77.4/100 (GPT-5.5 + Codex specialized); perception remains the largest shortfall; the specialized workflow's value is harness-dependent (+3.16 in Codex, -2.21 in Claude Code for the same model, with the harness alone shifting +1.44 under an identical direct prompt); and it erases native connectors in every configuration even though humans still prefer its renderings in 4/5 pairs. Limitations: the judge shares a family with the strongest system, the harness contrast covers one model, and each task is generated only once.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Scientific figure reconstruction as a single-slide editable PPTX task exposes failure sources (perception vs planning vs harness) that screenshot- or count-based proxies conflate.
- Perception is the dominant bottleneck: text and visual-detail axes show the largest shortfalls even for the best configuration, and iterative intermediate rendering recovers only part of the gap.
- The return on specialized-workflow effort is jointly set by model and harness: the same GPT-5.5 gains +3.16 points in Codex but loses 2.21 in Claude Code, and the harness alone shifts GPT-5.5 by +1.44 under an identical direct prompt.
- The specialized workflow erases native connectors in all ten configurations (median 0), so the workflow layer, not the model, decides whether the artifact remains an editable document.
- The automated ranking is robust to judge family (Qwen Spearman 0.988) and rubric weight perturbation (Spearman >=0.95), correlates with blinded human Elo at r=0.958, and diverges from humans only where fidelity and editability decouple.
Strengths
Task contract closes a real evaluation gap
Requiring one native PPTX plus a deterministic raster-paste cap (c(P)=50 when one picture covers >=85% canvas with few native objects, Section 3) defeats both screenshot-similarity and object-count shortcuts; the cap fires on only 2 of 10,000 artifacts, so it deters rather than distorts.
Paired design cleanly isolates factors
Workflow effects hold (model, harness) fixed per task; the harness contrast holds (model, workflow) fixed with a word-identical direct prompt, yielding +1.44 [+0.66,+2.22] for GPT-5.5 in Claude Code vs Codex (Appendix G) - rare internal validity among agent benchmarks.
Unusually thorough evaluator validation
Provenance and cost transparency
Actionable negative results
Weaknesses
Judge-family confound with the strongest system
Headline scalar conclusion is weight-dependent and partially contradicted by humans
No agent-side repetition
Human study under-specified
Harness conclusions generalize from one model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ReFigBench?
ReFigBench is a benchmark of 1,000 real overview figures taken from arXiv papers that tests whether AI coding agents can turn a figure image into a single-slide, natively editable PowerPoint file that faithfully preserves the text, module structure, arrows, and layout. It uniquely scores both visual fidelity and whether the output remains an editable document, and it studies how much the surrounding agent tooling (the 'harness', e.g., Claude Code vs Codex) affects results.
Which AI models were tested and how did they score?
Four model families were tested: GPT-5.5, Claude Opus 4.6, MiniMax-M3, and MiMo-v2.5, each reconstructing 1,000 figures under two workflows. GPT-5.5 configurations took the top four ranks; its best average was 77.4 out of 100 (specialized PPTX workflow inside Codex), versus about 69 for Claude Opus 4.6, 66 for MiniMax-M3, and 61-63 for MiMo-v2.5. Even the best agent fell well short of the 100-point rubric ceiling, mainly on perception-heavy axes.
What is the main takeaway about agent harnesses?
Reported agent performance is a joint product of the model and its harness, not the model alone. The same GPT-5.5 gained +3.2 points from a specialized PPTX workflow inside Codex but lost 2.2 points inside Claude Code, and with an identical prompt and workflow the harness alone shifted its score by 1.4 points. The paper argues benchmarks should always disclose the harness alongside the model.
Is the ReFigBench dataset and code publicly available?
The authors commit to releasing, upon publication, the benchmark code, Dockerfiles, run scripts, and prompts (MIT license) plus figure metadata, annotations, and provenance records (CC BY 4.0). Images whose source licenses forbid redistribution will be withheld and replaced by provenance records and retrieval scripts. Reproducing the standard automated scoring requires access to a GPT-5.4-class API judge.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review is based on extracted text; Figures 1-7 and the Figure 6 case-study image could not be visually inspected, so claims about the annotated failure example rest on the text alone.
- The judge and human-eval prompts appear only in abbreviated form (Listings 1-2); exact full instruction wording is unverified.
- Code/data availability is a stated release plan 'upon publication'; actual availability was not verified.
- The metadata abstract was truncated mid-sentence, but the complete abstract appears in the extracted body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.