Large Language Models as Falsifiers for Cyber-Physical Systems
Ali ArjomandBigdeli, Jiawei Zhou, Stanley Bak
eess.SY · cs.AI · cs.LO · cs.SE
Quick verdict
Uses LLM prompting with semantic cues for CPS falsification; ranks first on 14/21 ARCH-COMP specs, but contamination and cost concerns remain.
Summary
The paper introduces LLM-Falsifier, which uses a large language model as the optimizer in STL robustness-guided falsification of cyber-physical systems. Built on the Ψ-TaLiRo toolbox, the loop prompts an LLM for the next input sample, appends each sample's robustness, output trajectory, and critical-time witness (the time realizing the robustness value, formalized recursively in Eqs. 9–15), and stops when robustness turns negative. Four prompt variants (MP1–MP4) progressively add the STL specification, signal names, output signals, and critical times. On 21 ARCH-COMP 2025 falsification benchmarks (Instance 1, budget capped at 100 simulations vs. the protocol's 1500), gpt-5-mini with high reasoning ranks first against the best competing tools on 14 of 21 specifications (by falsification rate, then mean simulations), falsifying six specifications on the very first simulation in every run. The ablation attributes large gains to semantic context (AT1: 25.2→1.2 simulations from MP1 to MP4), and open-source gpt-oss-20b is competitive. Main limitations: possible training-data contamination on long-published public benchmarks is not addressed; statistics are limited to means over 10 runs without confidence intervals or significance tests; wall-clock cost is high (81.8 s/iteration for the headline model); and the method fails entirely on 4 of 21 specifications, including two where uniform random sampling succeeds.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLM-Falsifier is the first robustness-guided LLM falsifier for CPS, minimizing STL robustness via iterative prompting built on Ψ-TaLiRo.
- Semantic prompt content — natural-language signal names, output trajectories, and critical-time witnesses — substantially improves sample efficiency (AT1: 25.2→1.2 mean simulations from MP1 to MP4).
- On ARCH-COMP 2025 Instance 1, gpt-5-mini with high reasoning ranks first against the best competing tools on 14 of 21 specifications.
- Six specifications are falsified on the first simulation in every run, which purely numerical optimizers cannot achieve.
- Stronger models falsify better, but the open-source gpt-oss-20b remains competitive (e.g., CC5: 8.9 vs 17.1 simulations).
Strengths
Novel, well-integrated method
First robustness-guided LLM falsifier, cleanly embedded in Ψ-TaLiRo with RTAMT monitors; the loop differs from OPRO in justified ways (Section 4.1): one sample per iteration, exactly one simulation per candidate, last-ten chronological history, no external selection.
Formally grounded critical-time witness
Eqs. (9)–(15) recursively define the witness time with correct inner-subformula semantics; Figure 4 gives a concrete example where the outer G operator's arg inf (t=2) misleads while the inner F's t*=3 correctly localizes the robustness.
Convincing ablation
Broad, standard evaluation
Honest limitations discussion
Weaknesses
Training-data contamination unaddressed
Thin statistical reporting
Cost asymmetry obscures practical standing
Complete failures on 4 of 21 specs, unexplained
Explainability from a different model
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper connects two research lines: STL robustness-guided falsification of cyber-physical systems (Breach, S-TaLiRo, ARCH-COMP) and LLM-as-optimizer prompting (OPRO). LLM-Falsifier treats a large language model as the derivative-free optimizer of Eq. (8): given a meta-prompt, the LLM proposes one input sample per iteration, an RTAMT monitor returns the robustness, and the search stops at the first negative value. Three claims carry the paper: (1) it is the first robustness-guided LLM falsifier for CPS; (2) semantic prompt content — natural-language signal names, output trajectories, and a formally defined critical-time witness (Eqs. 9–15) — substantially improves sample efficiency; (3) on ARCH-COMP 2025 Instance 1, gpt-5-mini (high reasoning) ranks first against the best competing tools on 14 of 21 specifications (falsification rate first, then mean simulations S), falsifying six specifications on the first simulation of every run.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is LLM-Falsifier?
A method that uses a large language model (such as GPT-5-mini) as the search engine for falsifying Signal Temporal Logic specifications of cyber-physical system simulators. The LLM proposes input signals, an STL monitor scores each with a robustness value, and the search stops as soon as a specification violation (negative robustness) is found.
Does LLM-Falsifier beat classical falsification tools?
On the ARCH-COMP 2025 falsification benchmarks it ranks first on 14 of 21 specifications by falsification rate and mean number of simulations. The win is in simulation count, not speed or cost: each step needs an LLM call (up to about 82 seconds for the strongest model), so wall-clock time can be worse than classical tools.
Is the code available?
Yes. The implementation and experiment scripts are public at github.com/aliabigdeli/llm-falsifier, built on the open-source Ψ-TaLiRo toolbox with RTAMT robustness monitors. Note that headline results depend on a proprietary OpenAI model and Simulink models.
What are the paper's main weaknesses?
Possible training-data contamination (the benchmarks are long-public and some counterexamples are common knowledge), statistics limited to means over 10 runs without significance tests, high per-iteration latency, and complete failure on 4 of 21 specifications — on two of which uniform random sampling succeeds.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 only partially analyzable from text extraction; Figure 3's prompt is elided in its history section.
- Baseline numbers in Table 1 are taken from the paper's rendering of ARCH-COMP 2025 and were not verified against the primary report.
- Minor equation-formatting artifacts in extraction; definitions read consistently overall.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.