Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies
Hang Xiao, Chuhong Xu, Chuhong Xu, Kainan Zhou, Kainan Zhou, Gangzhen Qian, Gangzhen Qian, Lu Yi
cs.CR · cs.AI · cs.CE · cs.IR · cs.LG
Quick verdict
Preregistered LLM RTL-verification audit whose AI runs yielded no valid analysis; valuable fail-closed benchmark+baseline, no model-performance evidence.
Summary
The paper introduces SecTB-RTL, a preregistered benchmark for auditing whether LLM-generated RTL verification plans form a valid security-measurement instrument. It comprises 31 RTL tasks in 16 (CWE, root-mechanism) strata, each carrying four authored, formally qualified security-regression mutants (124 total), a restricted JSON stimulus-and-oracle DSL, and a fail-closed chain spanning provider, schema, production semantics, rendering, dual-simulator execution, golden validation, and mutant detection. A deterministic, outcome-blind stimulus baseline killed 36, 75, and 78 of 124 mutants at 32, 64, and 128 simulation steps. The planned confirmatory AI experiment produced no analyzable endpoint: run C1-R2 failed when the provider rejected the frozen response schema on eight attempts, and the separately frozen C1-R3 completed 1,860 calls — 1,857 provider-accepted, but only nine passing the production semantic validator, because a reset-state-machine rule enforced by the executor was absent from the generation-facing contract and probe. The authors classify both runs as instrument-validation incidents and report no prompt-effect estimate, refusing complete-case rescue. Main limitations: the central treatment question remains unanswered; the key lesson arguably amounts to engineering hygiene rather than new science; mutants are authored proxies, tasks are purposively sampled from HardSecBench, and preregistration was author-sealed rather than independently reviewed.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Provider or schema acceptance does not establish execution validity: in C1-R3, 1,857 of 1,860 provider-accepted responses yielded only nine that passed the production semantic validator.
- A deterministic, outcome-blind non-AI baseline kills 36/75/78 of 124 mutants at 32/64/128 simulation steps, establishing benchmark feasibility and a bounded attainable-yield reference.
- Fail-closed governance — frozen hashes, separate assurance gates, and restricted amendments — prevented infrastructure failure from being misreported as model behavior, hence no prompt-effect estimate is reported.
- SecTB-RTL provides 31 tasks in 16 (CWE, mechanism) strata with 124 seven-gate-qualified mutants as a reusable security-evidence benchmark.
Strengths
Formally grounded mutant qualification
Sec. IV-B requires all 124 mutants to pass seven gates — compile, exact interface identity, synthesis, legal witness through top-level ports, formal non-equivalence to golden, witness replay under the trusted harness, and pairwise formal distinctness — making detection targets executable, reachable, observable, and distinct before any model output exists.
Exemplary fail-closed governance
Secs. V–VI fix denominators, one attempt per cell, no retries or redraw, and outcome-blind audits; after R3's calibration failure the authors preserve the full 1,860-call incident and explicitly refuse complete-case rescue or a post hoc effect estimate — a rare anti-HARKing stance.
Deterministic baseline calibrates the benchmark
Fine-grained incident forensics
Weaknesses
No empirical AI result
Headline lesson is largely self-inflicted engineering hygiene
Incomplete outcome reporting even for the incident
Self-attested provenance
Unknown headroom above the baseline ceiling
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SecTB-RTL (Secs. I–IV) is a preregistered framework for one narrow question: do LLM-generated RTL verification plans — persisted JSON stimulus-and-oracle artifacts — survive a trusted production path and detect authored security regressions? The frame derives from HardSecBench (rev e2084be7): 31 tasks in 16 (CWE, root-mechanism) strata (T21 excluded pre-contact; 15 paired strata plus the CWE-1241 singleton), each with four mutants passing seven qualification gates. Four prompt arms (F0 functional objective; FC equal-attention functional control; SA abstract security context; SE explicit security requirements) cross three model aliases and five repetitions into 465 blocks / 1,860 single-attempt calls. The claims are deliberately instrumental: (1) provider/schema acceptance does not establish execution validity (1,857 vs. 9 of 1,860); (2) a deterministic baseline (36/75/78 kills at 32/64/128 steps) calibrates attainable yield; (3) fail-closed governance prevented an unanalyzable run from becoming an effect claim. The treatment contrast SE−FC — the experiment's actual purpose — is never estimated.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SecTB-RTL?
SecTB-RTL is a preregistered benchmark of 31 RTL hardware-security tasks with 124 formally qualified, hand-authored security-regression mutants, built to test whether AI-generated verification plans can detect security regressions in synthesizable RTL.
Did the paper show that LLMs are bad at writing hardware verification plans?
No. Both confirmatory AI runs failed for instrumentation reasons before any valid measurement: one run was rejected at the provider's schema validation, and in the other only 9 of 1,860 calls passed the paper's own production validator because a reset rule was missing from the generation-facing contract. The authors explicitly report no model-performance estimate.
What does the 9 out of 1,860 result mean?
In run C1-R3, 1,857 of 1,860 model responses were accepted by the provider, but only nine satisfied the production semantic validator. The 1,848-call gap was traced to a reset-state-machine rule enforced at execution but absent from the prompt-side schema and probe — an instrument mismatch, not evidence about model capability.
Is the SecTB-RTL benchmark and code available?
The paper states that the benchmark, frozen protocols, generation and execution code, qualification evidence, and incident manifests are held in a public repository with cryptographic tags; the extracted text does not include the repository URL, so availability should be confirmed directly.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Numeric values in Tables I–III were extracted linearly from two-column text; layout-dependent pairings (e.g., baseline yields vs. budgets) were cross-checked against prose, but any figures were not analyzable.
- No repository URL appears in the extracted text, so artifact availability and the citation record (e.g., HardSecBench arXiv:2601.13864, dated 2026) could not be independently checked.
- The paper appears textually complete (Sections I–XII present), but supplementary material such as the DSL specification and incident manifests was not available for review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.