EdiTikZ: Scientific Figure Editing from Revision Trajectories
Christian Greisinger, Zhixue Zhao, Steffen Eger
cs.AI · cs.CL · cs.CV
Quick verdict
Strong resource+method: 391K natural TikZ edits, 9B model beats GPT-5.6-Sol overall yet ties on edit application; half of inferred instructions carry errors.
Summary
The paper introduces DaEdiTikZ, a dataset of 391K plausible TikZ figure-edit pairs mined from arXiv version histories, GitHub repositories, and TeX StackExchange threads, from which 781K directed edit instructions are inferred with Qwen3.6-27B conditioned on rendered figures and code. Because both endpoint figures are human-authored, only the instruction is synthesized. A human-refined benchmark, DaEdiTikZ-Bench (690 instances), supports evaluation. The authors fine-tune Qwen3.5 4B/9B models (EdiTikZ) with joint reconstruction+editing SFT on 1.51M instances, then apply RL (GDPO) combining a rendered-similarity reward from a frozen vision encoder with a reference-free, VLM-judged instruction-following reward. EdiTikZ-9B-RL reaches 0.726 Avg on the benchmark, above GPT-5.6-Sol (0.673) and Gemini-3.1-Pro (0.693); human evaluation (9 annotators, 4,320 ratings) yields 17.43 vs Gemini 17.72 and GPT-5.6-Sol 16.75. Notably, on edit application itself the proprietary models are nominally ahead (automatic EA 0.735 vs 0.734; human EA 5.89 vs 5.80); the aggregate win comes from source preservation and visual quality. Gains persist on OOD SPIQA/CharXiv within the 2K-token training horizon. Limitations: 50% of inferred instructions contain at least one error, single-run RL without variance reporting, GPT-generated synthetic instructions for OOD evaluation, and code promised but not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Naturally occurring revision trajectories (arXiv versions, GitHub, TeX SE) can be mined into 391K plausible TikZ edit pairs with 781K VLM-inferred directed instructions, claimed as the first large-scale revision-derived scientific figure editing dataset.
- Joint reconstruction+editing SFT followed by GDPO RL with complementary rendered-similarity and reference-free instruction-following rewards yields compact models; EdiTikZ-9B-RL achieves the best average score (0.726) among all tested systems on DaEdiTikZ-Bench.
- In human evaluation (9 annotators, 4,320 ratings), EdiTikZ-9B-RL (17.43) surpasses GPT-5.6-Sol (16.75) and is on par with Gemini-3.1-Pro (17.72).
- Reconstruction supervision improves editing, but editing supervision does not improve reconstruction (Tables 4 and 15).
- Post-training gains transfer to OOD SPIQA/CharXiv figures, with RL benefits growing with generation length and remaining effective through and somewhat beyond the 2K-token training horizon.
Strengths
Supervision where only the instruction is synthetic
Both endpoints of every edit pair are human-authored (arXiv versions, GitHub, TeX SE, §3), so the VLM synthesizes only the missing instruction. Filtering is empirically calibrated: implausibility was manually audited at 50 pairs per 0.01-wide similarity interval (Tables 2 and 11), retaining ≥0.95 intervals (2–12% implausible), plus bidirectional VLM acceptance; a 125-pair audit (κ=0.82) finds 98% plausible.
Editing-specific RL design targeting a real failure mode
The paper explicitly identifies that target-similarity rewards are gameable by preserving the source, so RIF verifies each atomic edit against the source render, reference-free, and the SFT vision encoder is frozen to limit reward hacking (§4). Table 5 supports the design: RIF-only 0.679 > RSSim-only 0.670 Avg; both rewards with GDPO 0.726 > with GRPO 0.688.
Above-average evaluation rigor
Candid negative results and asymmetric transfer finding
Weaknesses
The headline 'editing' win does not hold on the editing criterion itself
Central thesis (natural revisions beat synthetic supervision) is asserted, not tested
No variance or significance for RL results
Multiple judge/generator couplings
Cross-family evaluation asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a resource-plus-method paper on instruction-conditioned editing of TikZ scientific figures. It claims three artifacts and one headline: (1) DaEdiTikZ — billed as the first large-scale revision-derived figure-editing dataset — with 390,516 edit pairs and 781,032 directed instructions mined from arXiv version histories, GitHub repositories, and TeX SE threads (§3); (2) DaEdiTikZ-Bench, 690 human-refined editing instances; (3) EdiTikZ-4B/9B, Qwen3.5-based models trained with joint reconstruction+editing SFT (1.51M instances) and GDPO RL. The headline: EdiTikZ-9B-RL scores 0.726 Avg, above all tested baselines including GPT-5.6-Sol (0.673) and Gemini-3.1-Pro (0.693) (Table 3), and in human evaluation (9 annotators, 4,320 ratings) surpasses GPT-5.6-Sol (17.43 vs 16.75) while sitting on par with Gemini-3.1-Pro (17.72).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the DaEdiTikZ dataset and how was it built?
DaEdiTikZ is a dataset of about 391K pairs of TikZ scientific figures that plausibly differ by a real edit, mined from arXiv paper version histories, GitHub repositories, and TeX StackExchange threads. Figures from the same source context are paired by visual similarity (only pairs above 0.95 cosine similarity with low implausibility are kept), and a vision-language model (Qwen3.6-27B) writes the natural-language edit instruction by comparing both rendered images and code. Each pair yields two directed instances; both directions must be validated, giving 781K editing trajectories. A manual audit found 98% of retained transformations plausible, though about half of instructions contain at least one error.
Does EdiTikZ beat GPT-5.6 and Gemini at scientific figure editing?
On the aggregate automatic score, yes: EdiTikZ-9B-RL scores 0.726 average vs GPT-5.6-Sol 0.673 and Gemini-3.1-Pro 0.693 on the 690-instance benchmark. In human evaluation (9 annotators, 4,320 ratings) it scores 17.43 vs 16.75 (GPT-5.6-Sol) and 17.72 (Gemini-3.1-Pro). However, on the edit-application criterion specifically it is essentially tied with GPT-5.6-Sol (0.734 vs 0.735 automatic; 5.80 vs 5.89 human) and behind Gemini's automatic 0.761; its advantage comes from preserving unchanged figure content and visual quality. Under out-of-distribution figures it stays competitive mainly within its 2K-token training length.
Are the EdiTikZ models and dataset publicly available?
According to the paper (v2, September 2026), the models (4B and 9B, SFT and RL versions) and the datasets (DaEdiTikZ and DaEdiTikZ-Bench) are available on Hugging Face. Code is stated to be released on GitHub but had not been released at the time of writing. Reproducing the headline EA/SP/VQ metrics also requires access to a GPT-5.5 judge API.
What are the main limitations of this work?
About 50% of the inferred training instructions contain at least one error (omissions, hallucinations, or misinterpretations), partially mitigated by the RL reward. Performance degrades on long, complex out-of-distribution figures beyond the 2K-token training horizon. The out-of-distribution evaluation itself relies on synthetic instructions generated by GPT-5.6-Sol and reference-free judging. Finally, there is no direct experimental comparison against training on synthetic edit supervision, so the benefit of natural revision mining is argued rather than measured.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The supplied text is a sampled excerpt with explicit '[...sampling gap...]' omissions (e.g., parts of the Section 3 quality analysis, appendix A.5 OOD details, and some references); all judgments cite only content visible in the excerpt.
- Figures 3, 6, 7 and 15–21 and Table 1 are available only as extracted numbers and captions; visual claims such as CI overlap in Figure 6 could not be verified from rendered plots.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.