SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation
Marco Cipriano, Leonardo Zini, Alexandra Schild, Valentin Teutschbein, Afsana Mimi, Marcella Cornia, Lorenzo Baraldi, Gerard de Melo
cs.AI · cs.CV
Quick verdict
Claims a human-aligned text-to-SVG evaluator; strong dataset and results (ρ 74.85), but no CIs/significance tests and unchecked judge-captioner family overlap.
Summary
This paper argues that CLIPScore and off-the-shelf VLM judges barely register the errors SVG generators make (wrong colors, counts, spatial relations). It introduces SVG-Score: three CLIP backbones adapted to vector graphics on 3.2M recaptioned SVG-text pairs, then aligned to human preferences via a pairwise LoRA objective, plus a Qwen3-VL-8B judge trained with SFT and GRPO under ordinal and intra-caption ranking rewards. Supervision comes from a new human-rated Semantic Alignment dataset: 12,957 ratings (1-5) of 8,671 SVGs against 1,858 captions from OmniSVG, collected by five annotators, with caption- and SVG-disjoint splits and a reliability check (quadratic weighted κ=0.7985). On the held-out test set, the best adapted CLIP reaches ρ=63.18 versus 55.23 for HPSv2, and the judge improves zero-shot Qwen3-VL-8B from ρ=67.76 to 74.85, beating VectorGym (65.12) and all tested commercial judges. Ablations honestly show SFT alone hurts ranking and GRPO alone hurts MAE; only the combination plus ranking reward reaches the top. Benchmarking 16 generators on 1,616 independent captions, commercial systems dominate, with Claude Sonnet 5 first under six of seven evaluators. Weaknesses: no significance tests or CIs (the Hard stratum has just 116 prompts), temperature-0.7 judge decoding, and unexamined Qwen-family overlap between judge backbone and benchmark captioners.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Controlled perturbations (Sec. 3, Fig. 3) show CLIPScore and off-the-shelf VLM judges barely react to the color, count, and spatial errors SVG generators actually make.
- Preference-aligned SVG-adapted CLIP scorers reach ρ=63.18 (ViT-H/14, Table 2), beating HPSv2 (55.23) at identical parameter count.
- The GRPO-tuned Qwen3-VL-8B judge reaches ρ=74.85 on the held-out human test set (Table 3), beating zero-shot Qwen3-VL-8B (67.76), VectorGym (65.12), and all tested commercial judges.
- Ablations (Table 4) show SFT and GRPO are individually insufficient (SFT alone drops ρ to 66.40; GRPO alone 69.32); only their combination plus the ranking reward reaches 74.85.
- Benchmarking 16 generators on 1,616 independent captions (Sec. 6.3, Table 5), commercial systems lead, with Claude Sonnet 5 first under six of seven evaluators.
Strengths
Sharp diagnostic motivation
Sec. 3 quantifies evaluator blindness with controlled caption perturbations (color swap, spatial swap, count change) versus coarse image proxies (colored circles, solid background) on 200 OmniSVG SVGs split B&W (n=50) / color (n=150); Fig. 3 shows vanilla CLIP ViT-B/32 barely reacts to caption corruptions yet drops sharply on image collapse.
Carefully constructed human dataset
12,957 ratings over 8,671 SVGs and 1,858 captions (Sec. 4.1); the split partitions the caption-SVG bipartite graph into connected components so no caption or SVG crosses the train/test boundary, verified by SHA-256 hash checks (Supp. A); reliability: quadratic weighted Cohen's κ=0.7985, 83% of double-annotated examples within one point.
Consistent, decomposed gains
Honest ablation
Broad generator benchmark
Weaknesses
No statistical inference
Single-annotation density uncorrected
Unexamined Qwen-family overlap
Score-scale mapping unclear
Sampled decoding for a judge
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SVG-Score: Human-Aligned Evaluation of Text-to-SVG Generation — Structured Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SVG-Score and what problem does it solve?
SVG-Score is an evaluation framework for text-to-SVG generation. Prior practice scores generated SVGs with CLIPScore or off-the-shelf vision-language judges, both of which barely react to the errors SVG generators actually make (wrong colors, counts, spatial relations). SVG-Score trains two evaluators on a new dataset of 12,957 human ratings (1-5) of caption-SVG pairs: CLIP models adapted to vector graphics for fast large-scale scoring, and a fine-tuned Qwen3-VL-8B judge for expressive, rationale-producing assessments.
How well does SVG-Score match human judgments compared with existing metrics?
On the paper's held-out human-rated test set, its best adapted CLIP scorer reaches Spearman ρ=63.18 versus 55.23 for HPSv2, a leading natural-image preference scorer. Its fine-tuned VLM judge reaches ρ=74.85, above zero-shot Qwen3-VL-8B (67.76), VectorGym (65.12), Claude Haiku 4.5 (67.12), and GPT-5.4-mini (66.96). Reported results carry no confidence intervals, so the robustness of small margins is unverified.
Which text-to-SVG generators produce the best results according to SVG-Score?
Benchmarking 16 systems on 1,616 independent captions, commercial models lead: Claude Sonnet 5 ranks first under six of the paper's seven evaluators, followed by GPT-5 mini and Gemini 3.0 Flash. Among open-source methods, HiVG leads under the adapted CLIP scorers and the VLM judge; several open generators frequently fail to render (up to 33.17% for IntroSVG).
Is SVG-Score a reference-based or reference-free metric?
Reference-free: it scores only the caption and the rendered generated SVG, with no ground-truth SVG needed. The related metric SVGauge is reference-based (it compares a generation against an original SVG) and is excluded from comparisons on exactly that basis, so no reference-based baseline is tested.
Is the dataset, code, and model weights available?
The paper states that the dataset, evaluator models, benchmark prompts, and source code will be publicly released via its project page, and the supplementary material documents hyperparameters, reward functions, prompts, rendering settings, and the leakage-control protocol. Release status should be verified, as the review is based on the paper text alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based solely on the extracted text, including the full supplementary material; figures (Fig. 3-6, 8-16) are described only through captions and surrounding prose, so no visual verification of plots or qualitative examples was possible.
- Two in-text references to annotation-reliability details in 'Sec. A.1' appear inconsistent with the supplementary's section labels (reliability discussion appears in Supp. A.2), but the numbers themselves (κ=0.7985, 83% within-one-point, 100 examples) are present and unambiguous.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.