Beyond Static Guarantees: Measuring the Static-Pass Dynamic-Fail Gap in Security-Sensitive and LLM-Generated Python Code
Jessica Pourleyli, Maitreyee Das Urmi, Glaucia Melo
cs.CR · cs.AI · cs.SE
Quick verdict
Reports ~1 in 7 statically clean samples exploitable via an LLM exploit pipeline; plausible as pipeline yield but tempered by same-model self-confirmation.
Summary
The paper introduces the Static-Pass Dynamic-Fail (SPDF) phenomenon: Python samples that produce no findings under a composite Bandit–Semgrep static gate yet prove exploitable at runtime. A three-stage agentic pipeline—static scanning, a gpt-5.4-nano CWE-detection agent grounded in MITRE/NVD retrieval, and a Docker-sandboxed exploit-verification agent—was applied to 1,355 Python samples from SecurityEval, RedCode, and CyberNative. Of 654 statically clean files, the detector proposed 394 candidate file–CWE pairs across 235 files; dynamic verification confirmed 120 and partially confirmed 20, yielding a conservative SPDF rate of 12.69% (83/654 files) and an inclusive rate of 14.53% (95/654)—explicitly framed as a pipeline-yield lower bound, not prevalence. Confirmation varied by dataset (RedCode 33.7%, CyberNative 28.6%, SecurityEval 5.4%; χ² p=0.0019) and CWE class (p<0.0001); CWE-338 and CWE-330 were confirmed in every candidate, and several confirmed categories (CWE-338, CWE-916, CWE-73, CWE-770, CWE-862) were entirely absent from static output. The pipeline cost ~96.3M tokens and ~28.7 hours. Main limitations: detector and verifier share one model family (self-confirmation risk), no human validation of verdicts, RedCode dominates the aggregate, and results are Python-benchmark-specific.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Among 654 Python samples passing a composite Bandit–Semgrep gate, the pipeline dynamically confirmed exploitable vulnerabilities in 83 files (12.69%), or 95 files (14.53%) counting partial confirmations — a lower-bound pipeline yield, not prevalence.
- Several CWE categories (CWE-338, CWE-916, CWE-73, CWE-770, CWE-862) were entirely absent from Bandit/Semgrep output across the corpus yet confirmed exploitable at runtime.
- Exploit confirmation depends strongly on dataset source (χ²(2)=12.55, p=0.0019) and CWE category (permutation χ²(22)=138.06, p<0.0001), with adversarial corpora far above the curated generation benchmark.
- Agentic dynamic exploit verification is practical but expensive (~96.3M tokens, ~28.7 hours for 394 candidates), suiting selective deployment behind static filtering.
Strengths
Careful measurement semantics
Sections III-F, IV-A, V-A and the conclusion repeatedly and correctly frame SPDFR (12.69% conservative, 14.53% inclusive of 654 clean files) as a pipeline yield and lower bound, not a prevalence estimate — rare discipline in this genre.
Verification criteria that respect evidence limits
Table II's four-verdict taxonomy and the vulnerability-specific success criteria (e.g., runtime exceptions alone insufficient) correctly treat NOT TRIGGERED as absence of evidence, and require ≥3/5 CONFIRMED runs plus MITRE/NVD grounding before harness generation.
Whole-category static blind spots
Honest cost accounting
Appropriate statistical choices
Weaknesses
Detector and verifier share one model family
Zero human validation of verdicts
Aggregate dominated by one adversarial dataset
Unexplained execution-count inconsistency
Temperature-0 detection justified by stochasticity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: among Python samples producing zero findings under a composite Bandit–Semgrep gate, how often can runtime exploitation be demonstrated? It operationalizes this as the Static-Pass Dynamic-Fail (SPDF) phenomenon over 1,355 samples from SecurityEval (121), RedCode (810), and CyberNative (424). Headline claims: 12.69% conservative (83/654) and 14.53% inclusive (95/654) pipeline yield; strong dependence on dataset (p=0.0019) and CWE class (p<0.0001); whole CWE categories (CWE-338, CWE-916, CWE-73, CWE-770, CWE-862) absent from static output yet dynamically confirmed; and ~96.3M tokens / ~28.7 hours of agentic cost. The paper is unusually disciplined about scope: Sections III-F, IV-A, and V-A plus the conclusion all repeat that SPDFR is a pipeline yield — a model-dependent lower bound — not a prevalence estimate, and the corpus caveat (benchmark/vulnerable code, not assistant-in-the-loop development) is stated up front.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Static-Pass Dynamic-Fail (SPDF) rate reported in this paper?
Among 654 Python files that passed Bandit+Semgrep with zero findings, the pipeline dynamically confirmed exploitable vulnerabilities in 83 files (12.69%); including partial confirmations, 95 files (14.53%). The authors stress this is the yield of their detection-and-verification pipeline — a lower bound — not an estimate of how many vulnerabilities the clean files actually contain.
Which vulnerability types evaded static analysis but were still exploited?
CWE-338 (weak PRNG, 15 confirmed), CWE-916 (weak password hashing, 9), CWE-73 (external path control, 6), CWE-770 (unbounded allocation, 6), and CWE-862 (missing authorization, 6) produced no Bandit or Semgrep findings anywhere in the corpus yet were confirmed at runtime. CWE-400 (resource exhaustion) had the most confirmations overall (27) but did appear in static output elsewhere.
What LLM powered the pipeline and what did it cost?
OpenAI's gpt-5.4-nano for both stages: detection at temperature 0 (24.6M tokens, ~10.5 hours) and exploit verification at temperature 0.3 (71.7M tokens, ~18.2 hours), totaling ~96.3M tokens and ~28.7 hours for 394 candidates. Per dynamic verification, the mean was 36,388 tokens and 33.19 seconds.
Is the code and data from this study publicly available?
Yes. The paper cites a Zenodo replication package (DOI 10.5281/zenodo.20761891) containing the pipeline code, system prompts, tool specifications, generated exploit harnesses, and exploit evidence; the source datasets (SecurityEval, RedCode, CyberNative) are independently public.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's stacked-bar values were extracted as flattened digit runs; per-CWE candidate/confirmed/partial counts cited in this review were cross-checked against Table III and Section IV-B text, which are cleanly extracted.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.