The History Is the Detector: Executing CVE Patch History, End-to-End
Qiushi Wu, Kevin Eykholt, Youngja Park, Xiaokui Shu, Dhilung Kirat, Douglas Lee Schales, Ian Molloy
cs.CR · cs.AI · cs.SE
Quick verdict
Claims CVE history becomes executable detectors; strong staged evidence, but headline 644 findings rest on partial validation and a 23-case patch eval.
Summary
BUGSTONE-E2E (IBM Research) converts CVE patch history into executable vulnerability detectors and validates findings end-to-end. The pipeline mines 19,325 high-severity CVEs (2022–2026), verifies 2,710 fixing commits, and synthesizes 1,033 rules across 56 CWE families packaged into 172 agent skills. Detection is a cost-ordered funnel: Tree-sitter enumerates anchor call sites, deterministic filters remove 60.6% of candidates, and per-candidate LLM agents (default gpt-5-mini) return evidence-backed verdicts; runtime validation then builds PoCs, and generated patches pass two-sided differential testing. On 14 real projects, Phase B yields 2,933 findings; 644 gain runtime evidence, 224 a live PoC, and 62 reach the CONFIRMED_EXPLOITABLE tier, though 808 findings remain untested backlog. Against three agentic harnesses on rolled-back wolfSSL (64 localized post-snapshot CVEs), BUGSTONE-E2E with gpt-5-mini reaches 52% recall versus 34% (Codex), 23% (Anthropic), and 5% (VVAH). Ten identical Pillow runs quantify stochasticity: 97.6% of run-to-run losses are BUGGY-to-false-positive flips with strong within-batch correlation (φ=0.71). Phase D is a prototype validated on only 23 findings (11 in the sole independent third-party codebase), using worker-transcribed outputs. Main limitations: compute-limited dynamic validation, single-target harness comparison without significance testing, and unclear contamination control between mined rules and evaluation CVEs.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CVE fixing commits can be automatically mined into reusable, executable detection rules at scale: 1,033 rules across 56 CWE families, packaged into 172 skills, from 2,710 verified commits drawn from 19,325 high-severity CVEs (2022–2026).
- A cost-ordered funnel — deterministic Tree-sitter anchors, synthesized filters (60.6% candidate reduction), and per-candidate gpt-5-mini verdicts — makes LLM detection scale, with the small model (~$161/scan) covering 78% of GPT-5.5's finding set (~$900) and ten reruns reaching 97–99% coverage.
- On a rolled-back wolfSSL 5.8.4 tree, BUGSTONE-E2E reaches 52% recall over 64 post-snapshot CVEs versus 34% (Codex harness), 23% (Anthropic harness), and 5% (VVAH).
- Self-constructed runtime validation yields 644 runtime-evidenced findings across 14 projects, with 224 live PoCs and 62 CONFIRMED_EXPLOITABLE.
- Two-sided differential testing (exploit fails under patch, succeeds after revert) can validate self-discovered patches, demonstrated on a 23-case prototype with a median of 8 changed lines.
Strengths
Evidence-tiered validation with explicit epistemic separation
Table 5 uses strictly nested tiers (runtime 644 → live PoC 224 → CONFIRMED_EXPLOITABLE 62), and §6 reports the 808-finding Phase C backlog (704 in FreeBSD) as backlog, keeping 'not exploitable' (1,099 rejected) distinct from 'could not be tested' (382 static-only).
Rare reproducibility rigor for an LLM pipeline
§5.5 runs ten identical Pillow scans on the same 8,115 candidates: 1,061 of 1,087 run-to-run missing occurrences (97.6%) are BUGGY→FALSE_POSITIVE flips, concentrated in 153 batches (φ=0.71 within-batch, −0.001 across) — a clean demonstration that batches behave as single decisions, motivating small Phase B batches.
Honest ground-truth expansion in the harness comparison
Cost-ordered funnel empirically justified
Fix recovery and rule-direction discipline
Weaknesses
Potential rule-evaluation contamination in the flagship comparison
Headline metric is the loosest bar and compute-limited
No statistical testing anywhere
Patch evaluation is tiny and partly self-referential
No code/artifact release statement and no disclosure outcomes
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
BUGSTONE-E2E (IBM Research, arXiv 2609.05335) claims a general pipeline that converts CVE patch history into executable detection knowledge and carries findings through detection, runtime validation, and verified remediation. The artifact chain (Table 2): 19,325 high-severity CVEs (2022–2026) surveyed → 5,902 resolved to open-source projects → 2,710 verified fixing commits → 2,662 validated cases → 1,757 pre-consolidation rules across 266 CWEs → 1,033 production rules across 56 CWE families → 172 deployable skills. Deployment over 14 projects (Table 5) yields 2,933 Phase B findings, 644 with runtime evidence, 224 with live PoCs, and 62 at the CONFIRMED_EXPLOITABLE tier. The abstract headlines 644; the strictest bar is 62 — about 2.1% of Phase B findings.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is BUGSTONE-E2E?
An IBM Research framework that automatically converts CVE fixing commits into executable vulnerability-detection rules (1,033 rules across 56 CWE families, packaged into 172 agent skills) and applies them through a staged pipeline: static call-site enumeration with Tree-sitter, deterministic filtering, per-candidate LLM agent verification, runtime proof-of-concept validation, and patch generation checked by two-sided differential testing.
How well does BUGSTONE-E2E perform?
On 14 real projects (FreeBSD, PyTorch, OpenSSL, WordPress, etc.) it reported 2,933 findings, of which 644 gained runtime evidence, 224 had a live proof-of-concept, and 62 were confirmed exploitable. On a rolled-back wolfSSL 5.8.4, it matched 52% of 64 later-fixed CVEs, versus 34% for OpenAI's Codex security harness, 23% for Anthropic's reference harness, and 5% for Visa's VVAH.
Does BUGSTONE-E2E need an expensive LLM?
No. The default verification model is gpt-5-mini at roughly $161 per Pillow scan versus about $900 for GPT-5.5; deterministic filters remove 60.6% of candidates before any model call, and running the small model ten times recovers 97–99% of the stronger models' finding sets at lower total cost.
What are the paper's main limitations?
Dynamic validation is compute-limited (808 findings never runtime-tested; only 62 confirmed exploitable), the harness comparison covers a single C/TLS target without significance tests, patch generation is a 23-case prototype verified from worker-transcribed outputs, rules cannot express stateful vulnerabilities like nonce reuse, and contamination control between the mined rule base (2022–2026 CVEs) and the evaluation CVEs is not described.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with four marked '[...sampling gap...]' regions: part of §2.2/§3 (after the LLMxCPG passage), the Phase C/D details of §4.3, §5.5 run-table rows 3–10 and the Figure 3 discussion start, and the §7.1 tail through the §8 opening; judgments touching those regions rely on surrounding text only.
- The abstract in the metadata is cut off mid-sentence ('Using 19,325 high'), but the paper body restates all headline numbers, so no result is unknown.
- Figures 1–3 are available only via captions and in-text numbers; their visual content could not be verified.
- No code or artifact-availability statement appears in the provided text; one may exist inside the unseen sampling gaps.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.