Designer-RSI: Evolving Procedural Memory from User Traffic for Agentic Graphic Design
Hongyang Du, Lan Yan, Christian Flores, Asim Kadav
cs.AI · cs.CV
Quick verdict
Skill bank evolved from user traffic lifts a frozen design agent (GenEval2 72.7→99.3%, Sonnet-4); credible but one-judge evaluation, no method baselines.
Summary
The paper introduces EVOLVE, a continual adaptation framework in which a frozen frontier LLM drives a professional graphic-design agent (230+ tools spanning Photoshop/Illustrator/InDesign equivalents) while an external library of natural-language SKILL.md playbooks evolves from real user traffic. Two mechanisms act on the bank: widening distills recurring uncovered subtasks into new skills (minted only after three recurrences), and deepening rewrites failure-prone skills by contrasting failed with successful executions. A matched replay gate admits a change only if, under frozen contexts and order-randomized pairwise judging, it wins at least one replayed prompt and loses none. Over five rounds on 1,406 briefs (1,869 graded trajectories, no weight updates or human labels), the bank grows from 76 documentation-derived skills to 139. Reported gains: GenEval2 execution success on Claude-Sonnet-4 rises from 72.7% to 99.3% (+11.99 quality points); GPT-5.4-judged win rates versus the no-skill agent reach 67.6% (Opus-4.6) and 61.8% (Sonnet-4) across four design benchmarks; an ablation on 200 held-out briefs shows neither mechanism alone separates from the cold-start bank (46.4–49.4% win rates) while the combination reaches 58.5% (p=0.025). Limitations: all quality judgments come from LLM judges, no comparison against competing skill-evolution methods, internal benchmark prompts are withheld, the gate is only locally safe (round 4 regresses at some thresholds), and Opus-4.6 GenEval2 quality drops.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An external, evolved library of natural-language skills substantially improves a frozen frontier agent at professional graphic design with no weight updates and no human labels (GenEval2 execution success 72.7%→99.3% on Claude-Sonnet-4; overall win rates 67.6% and 61.8% across four design benchmarks on Opus-4.6 and Sonnet-4).
- Widening (minting skills for recurring uncovered subtasks) and deepening (rewriting failure-prone skills from success-failure contrasts) are individually insufficient but jointly superadditive (49.4%/48.6% alone vs 58.5% combined, p=0.025, on 200 held-out briefs).
- A matched replay gate — frozen contexts, order-randomized pairwise judging, no absolute scores — admits only changes that repair failures without regressing observed successes.
- Five evolution rounds over 1,406 real user briefs and 1,869 graded trajectories grow the bank from 76 documentation-derived skills to 139, with 100 rewrite and 67 mint proposals rejected by the gate.
Strengths
Real deployment scale with unusually honest gate accounting
Five evolution rounds on 1,406 briefs / 1,869 graded trajectories (§4.1); the paper reports not only committed changes but gate rejections (100/231 rewrite proposals, 67/136 mint candidates) and a non-monotonic held-out trajectory where round 4 falls below the no-skill baseline at τ≥0.3 (90% vs 94%, Fig 5).
Principled admission control under unverifiable feedback
The replay gate (§3.3, App F) freezes upstream contexts with distinct retrieved assets so the skill condition is the only manipulated variable, judges pairwise with presentation-order randomization, and never uses absolute scores — a concrete answer to judge drift (App F) and asset confounding.
Clean ablation isolating the two mechanisms
Candid negative results and limitations
Cost reported as first-class outcome
Weaknesses
Judge–reward circularity with no human evaluation
No competing-method baselines
Thin statistical reporting
Reproduction effectively blocked by proprietary stack
Configuration-specific regressions left unexplained
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EVOLVE (titled Designer-RSI) is a system/method paper from Adobe Research with Brown University claiming that a frozen frontier LLM can be continually improved for professional graphic design purely by evolving an external library of natural-language skills from real user traffic. Central claims: (1) the evolved bank lifts GenEval2 execution success from 72.7% to 99.3% on Claude-Sonnet-4 with +11.99 quality points (Table 1); (2) blind pairwise win rates vs the no-skill agent reach 67.6% (Opus-4.6) and 61.8% (Sonnet-4) averaged over four design benchmarks (Table 2, GPT-5.4 judge); (3) widening and deepening are individually inert but jointly superadditive — 49.4%/48.6% alone vs 58.5% combined, p=0.025, on 200 held-out briefs (Table 3); (4) all of this runs with no weight updates and no human labels over 1,406 briefs and 1,869 graded trajectories in five rounds, growing the bank from 76 to 139 skills.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Designer-RSI (EVOLVE) paper about?
Adobe researchers keep a frontier LLM frozen as a graphic-design agent that operates 230+ design tools (Photoshop, Illustrator, and InDesign equivalents). Instead of fine-tuning the model, they evolve an external library of written 'skills' — step-by-step natural-language playbooks — from real user requests, minting new skills for recurring uncovered tasks and rewriting skills that keep failing.
How much does the evolved skill bank improve the agent?
On Claude-Sonnet-4, GenEval2 execution success rises from 72.7% to 99.3% with +11.99 quality points. Against the same agent without skills, the evolved system wins 67.6% (Opus-4.6) and 61.8% (Sonnet-4) of blind pairwise comparisons averaged over four graphic-design benchmarks. Gains are not uniform: Opus-4.6 loses a little GenEval2 quality, and the smaller Qwen backbone regresses on some benchmarks.
Does the method require fine-tuning or human labels?
No. Five evolution rounds over 1,406 briefs and 1,869 automatically graded trajectories use no weight updates and no human reward labels. Changes ship only after a replay gate shows, on frozen contexts with order-randomized pairwise judging, that a candidate fixes failures without regressing any observed success.
Is the code, data, or benchmark public?
No. The tool environment is Adobe-proprietary, skills evolve on private user traffic, and the 200-brief internal benchmark's prompt texts are withheld for policy reasons; the paper shows only rendered outputs for 193 held-out prompts. The replay-gate and retrieval logic are described in enough detail to reimplement in isolation, but end-to-end replication is not possible from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure images (Figs 1–11) could not be inspected; assessments rely on captions and on the in-text tables (Fig 5b retention values and Fig 6 latency numbers are present as text).
- Qualitative panels (Figs 7–9) and the 193-prompt galleries (Figs 10–11) are judged only from their written descriptions, not from the images themselves.
- The identities and prompts of the evaluation judges (App D) and the exact number of replay contexts and tie band (App F) are not fully specified in the text, limiting auditability of judge-based claims.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.