Design Docs Are All You Need: An AI-native Machine-Learning Performance Tool
Samuel Kushnir, Kimia Noorbakhsh, Kavya Sreedhar, Liqun Cheng, Ming Liu, Parthasarathy Ranganathan, Mohammad Alizadeh, Fred Kjolstad, et al.
cs.PL · cs.AI
Quick verdict
SMART regenerates an ML perf-model library from design docs via agents; promising, rigorously motivated, but zero quantitative validation is presented.
Summary
SMART is a symbolic performance-modeling library for ML systems whose repository consists almost entirely of natural-language design docs arranged as a DAG (50 docs, ~9,000 lines of prose); coding sub-agents regenerate the implementation from the docs alone on every version change, with human input restricted to doc edits. The authors argue this drives 'incremental generation debt' (Eq. 1) to zero and mitigates context-window myopia via one self-contained doc per sub-agent, processed in topological order by an orchestrator. Two ingredients are credited for reliability: worked-example-style docs with exact 'reconciliation anchor' presets enforced by generated tests, and a minimal recursive operator IR with SymPy symbolic costs, a fast analytical roll-up mode for large sweeps, and a slow modulo-scheduling mode. The library is TPU-shaped (mxu_op, VMEM tiles, ICI collectives), includes a flash-attention builder DSL example, and covers dense, MoE, latent-attention, and robotics/VLA model families. The paper reports full regeneration takes 1.5–3 hours and ~$100 via Claude Code, and claims regenerated code reproduces hand-audited reference models — including DeepSeek-V3 serving on a TPU pod slice — to round-off precision. The main limitation is that no quantitative evidence is presented: no results tables, no regeneration success/repair rates, no DAG-discovery error analysis, no ablations of worked examples or the per-doc agent scheme, and no artifact release.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 15 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Making natural-language design docs the sole durable artifact and regenerating code from them on every version change drives incremental-generation technical debt (Eq. 1) to zero by construction.
- Worked-example-style docs, with reconciliation-anchor tests stating expected outputs exactly, are the key ingredient making independent agentic code generations consistent.
- A minimal recursive operator IR with SymPy symbolic costs, resource reservation tables, and dual fast-analytical/slow-modulo-scheduling roll-up modes suffices for rigorous ML performance co-design.
- Regenerated implementations reproduce hand-audited reference models — including DeepSeek-V3 serving on a TPU pod slice — to round-off precision.
- Full-library regeneration is routine-cheap: 1.5–3 hours and roughly $100 per rebuild using Claude Code.
Strengths
Clear, timely position with formal framing
Eq. 1 (Sec. 1) formalizes incremental-patching debt as D(t+1) = G(S(t+1), C(t)) − G(S(t+1)), and 'context-window myopia' (failure mode 2) is directly attacked by making docs — not fragmented code — the artifact handed to agents.
Well-engineered minimal IR
Sec. 3's recursive Op (symbolic SymPy costs, resource reservation tables, InnerLoop/Graph/Leaf params) cleanly separates algorithm-side loop nests from system-priced TPU-shaped leaves; dual roll-up modes (fast analytical roofline-style for 'sweeps over thousands of design points', slow modulo-scheduling per Rau 1994) with symbolic propagation and numeric binding only at the edge is a sound and practical design.
Actionable workflow engineering
Concrete DSL demonstration
Rare economic datapoint
Weaknesses
No quantitative evaluation
Validation circularity risk
Anecdotal cost/latency figures
Unverified DAG edge discovery
No ablations or baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a short (five-page) systems/position paper (arXiv 2609.05364, cs.PL/cs.AI; Google DeepMind, MIT, Stanford, Google) describing SMART, a symbolic performance-modeling library for ML systems whose main branch 'contains almost no code.' The durable artifact is a DAG of ~50 natural-language design docs (~9,000 lines of prose, Sec. 4). On every version change, read-only agents infer the docs' dependency edges and an orchestrator walks the DAG in topological order, assigning one coding sub-agent per self-contained doc to regenerate the whole implementation from the docs alone; humans only edit docs. The motivation is formalized as 'incremental generation debt' D(t+1) = G(S(t+1), C(t)) − G(S(t+1)) (Eq. 1, Sec. 1) plus 'context-window myopia.' Two reliability ingredients are proposed (Secs. 2–3): docs built around step-by-step worked examples with 'reconciliation anchors' (presets whose expected outputs are stated exactly and enforced by generated tests), and a minimal recursive Op IR with SymPy symbolic cost expressions, resource reservation tables, and two roll-up modes — a fast analytical one (roofline-style, closed-form, for 'sweeps over thousands of design points') and a slow modulo-scheduling one (Rau 1994).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SMART and what does 'design docs are all you need' mean?
SMART is a symbolic performance-modeling library for ML systems whose repository holds almost no code. Its main branch is a directed acyclic graph of about 50 natural-language design documents; on every version change, AI coding sub-agents regenerate the entire implementation from the docs alone, and humans make changes only by editing documents.
How much does it cost to regenerate the SMART library?
The paper reports a full clean-slate regeneration takes 1.5–3 hours and roughly 100 USD using Claude Code, which the authors estimate at about 20% of a weekly high-tier (Claude Max) usage budget.
Did the agent-generated code actually match the hand-written reference models?
The authors state that regenerated implementations reproduce hand-audited reference models — including DeepSeek-V3 serving on a TPU pod slice — to round-off precision. However, the paper provides no tables, protocols, tolerance definitions, or error statistics supporting this claim.
Is the SMART code or its design docs publicly available?
No. The paper contains no code, data, or artifact availability statement — neither the design-doc corpus, the orchestration prompts, nor the reference models are described as released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 appears only as a caption in the extracted text (figure graphics are not extracted); the regeneration workflow was assessed from the caption and body prose. No truncation: the text ends naturally after the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.