TicTacBench: Benchmarking Timing Closure Capabilities of Coding Agents
Bowei Wang, Zhigang Fang, Zhijie Yang, Renzhi Chen, Shanshan Li, Lei Wang
cs.AI
Quick verdict
TicTacBench adds 30 post-PnR RTL timing-closure tasks; across 300+ runs of 8 LLM agents the best closes 53.3% — a credible open benchmark, but rankings rest on single untested runs.
Summary
The paper introduces TicTacBench, a benchmark that evaluates whether LLM-driven coding agents can solve hardware timing closure, judged by post-place-and-route (post-PnR) static timing analysis. It contains 30 tasks built from RTLLM 2.0 and RTL-OPT designs, each providing suboptimal Verilog, realistic SDC constraints, baseline timing reports, and functional-equivalence artifacts; evaluation runs a fully open-source EDA flow (Yosys, SymbiYosys, LibreLane/OpenROAD/OpenSTA), and agents see only a budget-limited pre-PnR STA tool. Across 300+ runs of eight frontier LLM agents, the best (GPT-5.4-based) closes 53.3% of tasks; all eight agents buy timing gains at the cost of area-delay product degradation (7.18% on average), while 23% of tasks are closed by none. Semantic analysis of 240 trajectories yields six failure categories in three capability groups, and encoding them as standard operating procedures (TicTacSkill) raises the closure rate of three agents by 3.4-13.4 percentage points. Main limitations: the 30-task scale, a scope limited to single-clock, single-module setup violations, one run per task with no significance testing, and trajectory analysis performed by another LLM without human validation.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Post-PnR STA, not pre-PnR WNS, is the only reliable judge of timing closure: treating pre-PnR closure as a predictor yields 27.6% false positives and flips agent rankings (Fig. 1).
- On TicTacBench's 30 tasks, even the best agent (GPT-5.4-based) closes only 53.3% of tasks, and all 8 tested agents buy timing gains with ADP degradation (7.18% average).
- Trajectory semantic analysis over 240 runs yields 6 failure categories in 3 capability groups, with lower-closure agents failing in the optimization loop and higher-closure agents failing in RTL design choices.
- Encoding the failure categories as standard operating procedures (TicTacSkill) improves closure rate by 3.4-13.4 percentage points on the 3 ablated agents (Table IV).
Strengths
Addresses a genuine, well-motivated evaluation gap
Section II and Fig. 1 show pre-PnR signals mispredict post-PnR closure (27.6% false positives) and flip winner rankings (setup WNS agreement only 48.3%); no benchmark in Table I treats post-PnR closure as the scored target.
Reproducible open-source evaluation infrastructure
Section III-B integrates Yosys, SymbiYosys equivalence checking, and LibreLane/OpenROAD/OpenSTA post-PnR evaluation, with interface/function/latency guards before scoring; an anonymized code/data release is linked in the abstract.
Metrics capture both success and physical cost
Large, transparent measurement campaign
Actionable failure taxonomy and a constructive baseline
Weaknesses
Small and narrow task suite limits generalization
No statistical rigor behind model rankings
LLM-analyzed trajectories without human validation
Table II contains implausible and internally inconsistent numbers
TicTacSkill evaluation is thin and headline ambiguous
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
TicTacBench (arXiv 2609.23363) targets a real gap: existing RTL benchmarks (VerilogEval v2, RTLLM v2, RTLBench, RTL-Repo, RTLRewriter, RTL-OPT, CVDP — Table I) score function and synthesis-stage QoR but not whether an agent's RTL actually closes timing in physical implementation. The authors formalize the ATC task as (r, e, c, v, q, m) in Section II-B, deliberately restricting to setup violations repairable at RTL (hold left out of scope). Fig. 1 motivates the post-PnR ground truth empirically: pre-PnR closure mispredicts post-PnR with 27.6% false positives, and pre-PnR setup-WNS winner agreement is only 48.3%. Central claims: (1) the best of 8 frontier LLM agents closes 53.3% of 30 tasks; (2) all agents trade area for timing (ADP degraded 7.18% on average); (3) six failure categories explain the failures; (4) TicTacSkill SOP guidance lifts closure ~9%.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is TicTacBench?
TicTacBench is a benchmark that measures whether AI coding agents (LLM-driven) can fix digital hardware designs that fail their timing constraints. Unlike earlier RTL benchmarks that judge designs before physical implementation, it uses post-place-and-route static timing analysis as ground truth, across 30 Verilog tasks with suboptimal starting designs, realistic SDC constraints, and functional-equivalence checks.
How well do LLM coding agents perform on timing closure?
In the paper's 300+ runs across eight frontier LLMs, the best agent (GPT-5.4-based) closed 53.3% of the 30 tasks; the rest closed 23.3%-43.3%. All eight agents improved timing at the cost of area: average area-delay product degradation ranged from -2.5% (GPT) to -11.9% (DeepSeek V4 Pro), and 23% of tasks were closed by no agent.
Is TicTacBench's code and toolchain open source?
Yes. Evaluation uses fully open-source EDA tools: Yosys for synthesis, SymbiYosys for equivalence checking, and LibreLane with OpenROAD and OpenSTA for the post-PnR flow. An anonymized code-and-data repository is linked in the paper's abstract.
What is TicTacSkill and how much does it help?
TicTacSkill is a set of standard operating procedures derived from the paper's failure analysis (critical-path localization, datapath/pipeline design checks, evidence-driven search, and pre/post-PnR margin calibration). In ablations on three of the eight agents it raised closure rate by 3.4 percentage points (GPT-5.4), 13.4 (MiniMax), and 10.0 (DeepSeek V4 Flash); the abstract summarizes the improvement as about 9%.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table II in the extracted text is whitespace-compacted ('cells are compacted to control density'); power-unit outliers and the Dom./Dmd. row alignment may be extraction artifacts, but the text/table inconsistency is stated in the paper's own prose.
- Figure 3/4/5 data are available only as embedded axis labels; exact per-model distributions and the '+9%' computation cannot be verified from text alone.
- The abstract mentions a cross-model 'learning curve' plot whose details are not in the extracted text; its content could not be reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.