RealCADBench: Benchmarking Parametric CAD Modeling from Industrial Design Intents
Guanlin Li, Zhichao Huang, Yichen Long, Hongsen Liu, Linxin Cai, Huimu Yu, Zongzhen Li, Wei Wang, et al.
cs.CV · cs.CL
Quick verdict
RealCADBench adds a four-metric, industrial-intent CAD benchmark whose metrics genuinely diverge; model-proposed references and absent uncertainty need rebuttal.
Summary
RealCADBench is a benchmark for intent-to-program parametric CAD modeling from real industrial design intents: 12,632 tasks across 19 factory-automation categories spanning text, 2D engineering drawings, real product pictures, and rendered images, for both part and assembly modeling, though results are reported only on a 1,770-task slice. Systems generate FreeCAD Python that a shared runtime executes to export a 3D model, scored via executability, Solid and Surface IoU after alignment, and a rubric-based visual-semantic Judge (Kimi K2.6) that never sees the reference model. No model leads all four metrics: Gemini 3.1 Pro Preview attains the best regime-balanced composite (0.5507), GPT-5.5 leads executability (0.9311) and Judge (73.79), and Kimi K3 leads both IoUs while executing only 34.22% of tasks. On the 25-task assembly slice, Codex with GPT-5.5 improves executability and both IoUs yet lowers the Judge by 6.98 percentage points. Key limitations: quality scores are conditional on successful execution (a survivorship effect the paper flags but does not resolve), most references are model-generated rather than vendor CAD, no statistical uncertainty is reported, and the benchmark itself is not yet released, so results cannot be independently verified. Protocol detail and failure-mode analysis are valuable; reference-fidelity and robustness concerns warrant further work.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Executability, geometric IoUs, and visual-semantic identity are separable outcomes on real industrial CAD tasks, and no evaluated frontier model leads all four metrics.
- Difficulty does not follow a single order across the four input regimes; rendered-image inputs are easiest on every component (e.g., Judge 75.80) while the other three regimes reorder across metrics.
- Agents are complete evaluated methods, not wrappers: on RCB-Assm25, Codex+GPT-5.5 improves executability (+0.1600) and both IoUs over standalone GPT-5.5 but lowers the Judge by 6.98 pp.
- Recurring failure modes on realistic industrial intents are missing fine structures, loss of part identity, and incorrect assembly placement.
Strengths
Four-metric design with demonstrated, quantified metric separation
Executability, Solid/Surface IoU, and a reference-blind Judge are reported separately; on RCB-Assm25 the Judge correlates only r=0.103 (Solid) and r=0.237 (Surface) with IoUs across eight systems (Figure 4), empirically justifying the multi-metric claim rather than asserting it.
Genuine industrial scale with auditable curation
12,632 tasks over 19 factory-automation categories (Figure 6); five CAD practitioners verify references at ~17% acceptance (score ≥8), with near-duplicate/SKU grouping to reduce split overlap (Sec. 3.2, Appendix A.1).
The most novel metric is also the best-validated
Honest, self-aware reporting
The Codex agent result concretely substantiates the thesis
Weaknesses
Conditional quality metrics create survivorship bias that is acknowledged but not resolved
References for most Part tasks are model-proposed reconstructions, not vendor CAD
No statistical uncertainty anywhere
Benchmark, code, and prompts not yet released
Gemini family entanglement in the text regime
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
RealCADBench — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RealCADBench?
A benchmark that tests whether AI systems can turn real industrial design intents — text descriptions, 2D engineering drawings, product photos, and rendered images — into parametric CAD models. Systems write FreeCAD Python code, a shared runtime executes it to produce a 3D model, and the model is scored on executability, geometric overlap (Solid/Surface IoU), and a rubric-based visual-semantic judge. It contains 12,632 tasks across 19 factory-automation categories, covering both single parts and multi-part assemblies.
Which AI model is best at parametric CAD modeling according to RealCADBench?
No single model wins on every metric. Gemini 3.1 Pro Preview achieves the best regime-balanced composite score (0.5507), GPT-5.5 leads on executability (93.11% of tasks) and the visual-semantic judge score (73.79), while Kimi K3 leads both geometric overlap metrics but successfully executes only 34.22% of tasks. This metric separation is the paper's central finding.
How does the visual-semantic Judge metric work?
It is a rubric-based evaluation using a frozen Kimi K2.6 model that compares rendered views of the generated 3D model against the original input evidence (photos, drawings, or text-backed pictures), scoring product identity, salient features, and assembly relations on a 0-100 scale. Crucially, it never sees the ground-truth reference model, making it independent of the geometric IoU metrics. Its consistency was checked with repeated runs and alternative judge models on an external benchmark.
Is RealCADBench publicly available?
Not yet. The paper reports results only on a 1,770-task evaluation slice of the full 12,632-task benchmark and states that the full benchmark, evaluation code, and frozen prompts will be publicly released in the future, subject to access conditions and source-license constraints from the commercial data sources it was built from.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are recovered only via their embedded captions and axis text; per-regime scatter points and image-based evidence in Figures 1-8 are not visually assessable from the extracted text.
- Per-model counts of tasks entering the conditional quality metrics are not in the text (the paper states the released aggregation spec records them), so survivorship impact could not be quantified from this text.
- The running header spells 'MODELIMNG' (metadata title says MODELING) — a presentational flaw noted from the extraction.
- No reference-fidelity experiment against vendor CAD appears anywhere in the provided text; 'absent' refers to the entire text, not truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.