Does AI Save Time on Product Design? A Randomized Controlled Experiment of AI Prompt-to-Design Workflows
Remy Stewart, Olabode Anise, Andrew Hogan, Augustus Griffin
cs.HC · cs.AI
Quick verdict
A genuinely credible RCT with good statistics, but completer-only time analysis, garbled tables, and no COI statement (all authors are Figma employees) temper the 20% claim.
Summary
Figma employees report a moderated, between-subjects randomized controlled trial testing whether Figma Make, a prompt-to-design AI tool, saves time on design work. Fifty product designers and fifty product managers, recruited via MeasuringU panels and randomized within role (25 per condition), attempted three fixed-order UI tasks — dark-mode conversion, a settings-menu addition, and a comment-flyout prototype — with or without Figma Make. Among completers, access cut average per-task time by ~20% and cumulative time by ~27% (AME 15:08); completion probability rose 19.9 points on the hardest task (38.1 points for PMs). Gains concentrated among PMs: -22:46 (-35%), plus higher ease and SUS scores; designers showed only a marginal -17% cumulative effect, driven by the complex task. The authors candidly flag selection in the completer-only time analysis, tool reliability failures, and a narrow sample, but never explicitly disclose the vendor conflict of interest. Analysis used GEE models with cluster-robust errors, log-transformed times, and survey-based tests.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Among participants who completed the study tasks, access to Figma Make reduced average per-task completion time by ~20% (cumulative time -15:08, ~27%) after demographic and AI-experience adjustment (§4.2, Tables 3–4).
- Figma Make raised the probability of completing the hardest task (Task 3) by 19.9 percentage points overall and 38.1 points among PMs (§4.2, Table 2).
- PMs drive the headline effect: -22:46 cumulative (-35%), +1.42 ease and +16.6 SUS, while designers show a marginal -17% cumulative time effect driven by Task 3 (§4.3, Tables 3–5).
- Treatment participants rated tasks easier (+0.76 on 1–7) and Figma more usable (+10.0 SUS), with self-reported design quality also higher in the treatment arm (§4.4, Tables 5–6).
Strengths
Credible RCT architecture with role-stratified randomization
§3.1: 50 designers + 50 PMs randomized within role (25/condition), identical tasks, starting frames and reference images across arms; treatment restricted to Figma Make only (no other Figma AI) to isolate the effect; internal pilots preceded the study.
Appropriate correlated-data statistics
§3.2: GEE clustered per participant with cluster-robust sandwich SEs, log-transformed times, average marginal effects with SEs, covariate adjustment (Appendix D Table 8), and robustness checks (chi-squared plus Mann–Whitney, §4.4) with post-hoc correction noted.
Honest limitation §5.2 on selection
Outcome breadth beyond time
Weaknesses
Undisclosed conflict of interest
Completer-only time analysis invites informative-censorion inflation
Internal inconsistencies and template residue
Time outcome is entangled with unverifiable quality
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Four Figma employees report (§1) a moderated between-subjects RCT testing whether **Figma Make** — the vendor's prompt-to-design AI — saves time on everyday design work (RQ1), whether effects differ between designers and non-designers (RQ2), and downstream effects on ease, usability and quality (RQ3). Headline claims: ~20% average per-task time reduction among completers (§4.2: AME 10:53, p=0.0014), a 27% cumulative reduction (15:08, p<0.0001), a +19.9pp completion lift on the most complex task (+38.1pp for PMs, Table 2), gains concentrated in PMs (§4.3: -22:46, -35%; +1.42 ease; +16.6 SUS), and a marginal designer-level cumulative effect of -17% (p=0.049) driven by Task 3.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does Figma Make actually save designers time?
In this RCT of 100 experienced Figma users, access to Figma Make cut average per-task completion time by about 20% among those who finished the tasks; for product designers specifically, only the hardest task showed a significant per-task gain, and the cumulative designer-level effect (-17%) was marginal (p=0.049).
Who benefited most from Figma Make in the study?
Product managers. Treatment-arm PMs saved 22 minutes 46 seconds cumulatively (a 35% reduction), raised their completion of the hardest task by 38.1 percentage points, and reported higher task ease (+1.42 points) and SUS usability (+16.6 points); designers' ease and SUS gains were not statistically significant.
Was the study independent from Figma?
No. All four authors are Figma employees, and the tool tested was Figma's own Figma Make; moderation was outsourced to MeasuringU, but the paper contains no conflict-of-interest statement, which is a notable transparency gap.
What are the study's main limitations?
Time savings were analyzed only among participants who completed tasks, while the tool also changed who completed (potential selection bias); tasks were standardized rather than open-ended; completion was verified by human moderators without reported reliability checks; quality was self-reported; and stochastic Figma Make failures likely made the measured effect conservative.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–2 (workflow and time distributions) are not analyzable from extracted text; Appendix A task images are absent, so visual task fidelity cannot be assessed.
- Table 5's column alignment appears garbled in extraction (SEs may attach to the wrong column); significance-marker judgments for that table are flagged with reduced certainty.
- This is an arXiv preprint with placeholder DOI and unedited copyright boilerplate; numbers may change in the peer-reviewed version.
- Moderator completion criteria, scripts and widget telemetry are not in the text, limiting verification of measurement procedures.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.