VEX-Bench: Benchmarking LLM Agents for Assessing Exploitability of Software Supply Chain Vulnerabilities
Jiahao Shi, Edward Tsien, Yifeng Di, Hongjiao Zhang, Yuan Tang, Ronit Dey, Ilona Shishov, Gal Netanel, et al.
cs.CR · cs.LG · cs.SE
Quick verdict
Claims first expert-labeled benchmark for LLM agents judging supply-chain CVE exploitability; solid but small (75 cases), fine-grained labels ambiguous.
Summary
VEX-Bench is presented as the first benchmark evaluating whether LLM agents can assess the exploitability of known software supply chain vulnerabilities, a defensive triage task distinct from the zero-day attack settings of prior benchmarks such as CyBench and CyberGym. It contains 75 real-world cases (67 CVEs across 35 repositories) mined from GitHub dependency-update pull requests, covering Python, Java, and Go, labeled by five expert annotators using a calibration-then-scaling protocol (Fleiss' Kappa 0.667 during calibration). Given a repository snapshot and a CVE, agents must output a binary affected/not-affected status, a justification from four VEX-adapted categories, and grounded reasoning, under Docker isolation with anti-leakage probes. Nine models across three harnesses were each run three times. Claude Opus 4.6 reaches 81.6% status F1 and GPT-5.5 79.9%; for fine-grained justification only GPT-5.5 exceeds 70% macro-F1 (73.5%), while the OSV-Scanner baseline achieves just 40.5 F1. Key limitations: small scale with rare justification classes (requires_environment has two cases), single-label precedence rules that the authors themselves show admit factually valid alternatives, a model-harness confound from vendor pairing, and no statistical significance testing.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- VEX-Bench is the first benchmark evaluating LLM agents on exploitability assessment of known software supply chain vulnerabilities, as opposed to zero-day detection/exploitation settings.
- Frontier agents reach approximately 80% F1 on binary affected/not-affected classification (Claude Opus 4.6 81.6%, GPT-5.5 79.9%), but only GPT-5.5 exceeds 70% macro-F1 on fine-grained justification (73.5%).
- Traditional SCA tools, including OSV-Scanner's reachability mode, achieve at most ~43 F1 and cannot reliably distinguish affected projects from false dependency alerts.
- Fine-grained justification is systematically harder than binary status, with accuracy drops of 11.1-15.5 percentage points across all nine model-harness configurations.
Strengths
Genuine gap with clean task framing
Section 2 convincingly distinguishes dependency-level exploitability (known CVE, cross-repository reasoning, VEX-style output) from zero-day offense benchmarks (CyBench, CyberGym, SEC-bench, BountyBench); Figure 1 and Section 3.1 define inputs/outputs precisely, mirroring industry VEX practice (CISA 2022).
Above-average anti-hacking and leakage discipline
Section 3.4: Docker isolation, stripped Git history, out-of-label predictions scored FAILED rather than remapped (Appendix B.1), and a guided-completion memorization probe over 19 source PRs (Appendix C) finding only one trivial exact ten-token match caused by title repetition.
Expert labeling with a documented protocol
Honest error analysis
SCA baselines quantify the practical gap
Reproducibility scaffolding is detailed
Weaknesses
No trivial/majority baseline despite 70.7% imbalance
Model-harness confound in headline rankings
Statistically fragile fine-grained metrics
Post-calibration annotation reliability unmeasured
Small scale limits generalization claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VEX-Bench (arXiv 2609.08040) targets a gap the authors argue is real: no existing benchmark evaluates LLM agents on deciding whether a *known* upstream dependency vulnerability actually affects a downstream project — the daily triage behind VEX reports and the false-alert problem of Dependabot-style SCA matching (Section 1, citing Pashchenko et al. 2022 and Pu et al. 2026). The paper makes three claims: (1) the first benchmark for this task, with 75 expert-labeled cases over Python/Java/Go; (2) frontier agents reach ~80% F1 on binary status but fall short on fine-grained justification (only GPT-5.5 above 70% macro-F1, at 73.5%); (3) SCA tools, including reachability modes, cannot close the gap (Table 3: OSV-Scanner F1 40.5). The claims are mostly scoped honestly, and the related-work contrast with CyBench, CyberGym, SEC-bench, BountyBench, and CVE-bench (Section 2.2) is accurate as far as the text shows. The 'first benchmark' priority claim is plausible but rests on a security-venue-focused sweep; the SCA/VEX triage literature (e.g., Vuln4Real-style labeled datasets) deserves a deeper check before accepting priority outright.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VEX-Bench?
A benchmark of 75 real-world cases testing whether LLM coding agents can decide if a known dependency vulnerability (a CVE) actually affects a downstream software project. Agents receive a repository snapshot and the CVE, and must output an affected/not-affected status, a fine-grained justification label adapted from the industry VEX standard, and evidence-based reasoning.
How well do LLM agents perform on VEX-Bench?
The best systems reach about 80% F1 on the binary affected/not-affected decision (Claude Opus 4.6: 81.6%; GPT-5.5: 79.9%). Fine-grained justification is harder: only GPT-5.5 exceeds 70% macro-F1 (73.5%). Traditional dependency scanners perform far worse — OSV-Scanner reaches about 40 F1.
How is VEX-Bench different from CyBench or CyberGym?
Those benchmarks test agents discovering or exploiting previously unknown (zero-day) vulnerabilities inside a single codebase. VEX-Bench tests the defensive triage question of whether an already-known upstream CVE impacts a specific downstream repository, requiring cross-repository reasoning rather than exploitation.
Can the models cheat by looking up the answers online?
The authors argue no: the ground-truth labels were newly created by their expert annotators and are not published anywhere. They also ran a memorization probe on 19 source pull requests and found essentially no evidence that the evaluated models had memorized content that would reveal the labels.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 are available only through captions and in-prose summaries; per-configuration data points in Figures 3, 5, and 6 cannot be independently verified from the extraction. The text itself appears complete through Appendix E (no truncation observed).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.