CADWorld: Computer-Use Benchmark for Long-Horizon Computer-Aided Design
Zihan Dong, Yuanzhe Liu, Zhiyuan Ma, Qishi Zhan, Dehan Kong, Guohao Li, Kaixin Li
cs.AI
Quick verdict
CADWorld offers a 200-task FreeCAD GUI benchmark with executable artifact checks; best agent 17.5% vs 87% expert; valuable yet internally inconsistent.
Summary
CADWorld is a benchmark for evaluating computer-use agents on long-horizon mechanical-CAD workflows in FreeCAD. It contains 200 human-authored tasks spanning 11 workflow categories (sketch, part, assembly, CAM, FEM, measurement, mesh, technical drawing, and others) and 183 knowledge points, authored over roughly three months (~1,300 person-hours). Agents operate through screenshots and an allowlisted pyautogui action space (including stateful keyboard-mouse chords) inside a Docker/QEMU Ubuntu VM, capped at 100 steps per task; success is decided by task-specific executable evaluators over the saved .FCStd artifact and auxiliary outputs, requiring all checks to pass with no partial credit. Checks cover geometry, dimensions, constraints, document structure, CAM stock removal, and FEM outputs. Across seven agents on all 200 tasks, the strongest, GPT5.4 with a native computer-use harness, reaches 17.5% success versus 87.0% for a single expert reference pass; Opus4.8 scores 16.0%, the best open-weight model (Kimi K2.6) 7.5%, and all models score 0% on CAM, FEM, and TechDraw. A deterministic F1-F11 failure taxonomy shows weaker agents failing before producing a valid artifact while stronger ones fail on document structure, geometry, and construction process. Main limitations: contradictory headline numbers (25.0% on a 60-task subset in §1 vs 17.5% on 200 tasks in §4), a terminal-only ablation whose text and Figure 5 disagree, tiny FEM/TechDraw samples, and single runs without variance.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CADWorld is the first benchmark evaluating long-horizon mechanical-CAD workflows through the native FreeCAD GUI, with task-specific executable checks on saved native .FCStd artifacts rather than code or rendered appearance.
- The strongest of seven evaluated agents (GPT5.4 with native computer-use tooling) achieves 17.5% success on all 200 tasks versus 87.0% for a single expert reference pass.
- Agent failure migrates downstream with capability: weaker agents fail before producing any valid artifact, while stronger agents fail on document structure, geometry, and construction process.
- Every evaluated model scores 0% on all CAM, FEM, and TechDraw tasks.
- Removing the computer-use harness collapses GPT5.4 from 17.5% success to zero finished tasks (as stated in §4.2).
Strengths
Executable artifact-grounded evaluation with no LLM judge
Success is decided by task-specific programmatic checks on the saved .FCStd and auxiliary outputs: sketch entities/constraints/dimensions, model metadata (types, labels, volumes), assembly joint objects, CAM stock-to-target removal with undercut/overcut ratios, and FEM analysis objects plus exported CSV quantities (§3). Binary all-checks-pass scoring makes results deterministic and reproducible, unlike visual-similarity or LLM-judged pipelines.
First dedicated long-horizon GUI benchmark for mechanical CAD
200 human-authored tasks across 11 categories and 183 knowledge points (~3 months, ~1,300 person-hours), covering underrepresented operations (fillets/chamfers, assemblies, CAM, FEM, TechDraw). Table 1 shows prior CAD benchmarks (CADPrompt, CADBench, HistCAD, etc.) evaluate code/geometry, not an agent executing the native GUI workflow with editable construction history.
Failure taxonomy yields a genuine finding
Unusually complete harness documentation
Ablations probe the interface premise itself
Weaknesses
Internal contradictions in headline numbers
Terminal-only ablation contradicts its own text
No statistical uncertainty anywhere
Severely imbalanced category coverage backing strong claims
Cross-model comparison confounded by interface fit
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CADWorld (arXiv 2609.16251) is a benchmark paper targeting computer-use agents (CUAs) on long-horizon mechanical-CAD workflows in FreeCAD. It contributes: (1) 200 human-authored tasks across 11 workflow categories (Part 77, Sketch 63, Assembly 25, CAM 15, FEM 3, TechDraw 2, plus five 3-task utility categories) and 183 knowledge points, authored over ~3 months / ~1,300 person-hours; (2) artifact-grounded evaluation - task-specific executable checks on the saved .FCStd and auxiliary outputs, all-checks-pass binary scoring with no LLM judge; (3) an evaluation of seven agents. Headline: best agent GPT5.4 (CUA) at 17.5% success on all 200 tasks versus an 87.0% single-expert reference; a deterministic F1-F11 taxonomy shows failure migrating downstream with capability. The core claims are plausible given the presented runs, but the paper repeatedly undermines its own numbers.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CADWorld?
CADWorld is a benchmark of 200 human-authored FreeCAD tasks for computer-use agents. An agent sees screenshots and issues GUI actions (mouse, keyboard) inside a virtual machine; success is judged by executable, task-specific checks on the saved .FCStd CAD file and auxiliary outputs, covering geometry, dimensions, constraints, document structure, CAM material removal, and FEM simulation results.
How well do AI agents perform on CADWorld?
On the full 200 tasks, the best agent (GPT5.4 with a native computer-use interface) succeeds on 17.5% of tasks; Opus4.8 reaches 16.0% and the best open-weight model (Kimi K2.6) 7.5%. A single expert reference pass achieves 87.0%. All evaluated models score 0% on the CAM, FEM, and technical-drawing categories.
Why does CADWorld use FreeCAD instead of SolidWorks or Fusion 360?
FreeCAD is open-source and locally deployable, so the benchmark's virtual machine can be redistributed without license obstacles. It saves native, editable .FCStd projects with construction history, letting evaluators check not just final geometry but the parametric structure, constraints, and downstream manufacturing/simulation state (Appendix F, Table 8).
Is the CADWorld benchmark or code publicly available?
The paper points to a project page (cad-world.github.io) and documents the VM image, runner settings, prompts, and model adapters in detail. However, the extracted text does not state an explicit license or the release scope of task files and evaluators, so availability should be confirmed on the project page.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 4 and 5 were extracted as run-on numeric grids; only prose-stated values (e.g., 22.1% Part, 14.0% terminal-only GPT-5.4) were cited, and per-cell failure-reason assignments were not fully verifiable.
- Screenshot-bearing appendix tables (Table 11 failure cases, Table 12 task examples) could not be inspected from text.
- The internal inconsistencies flagged here are quoted from the provided extraction itself; a corrected preprint revision could resolve them.
- The project page and released artifacts could not be inspected (no network access); codeAvailable reflects the paper's statement only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.