Vulnerability Localization Benchmark: Measuring Agentic Security Analysis at Repository Scale
Aman Priyanshu, Supriti Vijay, Kimia Majd, Xuhong He, Fraser Burch, Takahiro Matsumoto, Jianliang He, Baturay Saglam, et al.
cs.CR · cs.AI
Quick verdict
A well-built 500-task benchmark of repository-scale vulnerability localization: the best system reaches only 0.229 File F1 and 38.4% of tasks are unsolved, but the Phase B restraint metric is partly confounded.
Summary
VLoc Bench measures vulnerability localization, a capability existing security benchmarks leave implicit: given only a CWE description and read-only terminal access to an unfamiliar repository, name the implementation files that instantiate that weakness. The 500 tasks are built from GitHub Security Advisories, each pairing the commit immediately before a security fix with the patch commit, across 290 repositories, six package ecosystems (Go 215, Maven 104, npm 88, pip 52, Rust 40, Composer 1) and 147 CWE categories. Ground truth is the set of non-test implementation files modified by the patch, scored with file-level F1. Phase A localizes on the vulnerable snapshot; Phase B re-presents the patched snapshot under the identical interface and scores whether the system correctly reports no vulnerability (true negative rate). Advisory text, CVE identifiers, file hints and severity are withheld. The paper evaluates 27 models and four static-analysis tools under one agent interface: 15 terminal commands, 20 turns, temperature 0.3, three independent runs. The strongest system, GPT-5.5 with xhigh reasoning, reaches 0.229 File F1 (precision 0.310, recall 0.221); the specialized Antares-3B reaches 0.223 with 3B parameters; 38.4% of tasks get zero correct localization from every model. Repository structure predicts difficulty far better than model identity (Lasso: repository group 0.285 versus model category 0.063 summed coefficient weight). Phase B shows the central tension: Antares-3B has a true negative rate of 0.034 on patched repositories and GPT-5.5 xhigh 0.279, while GPT-5 Nano reaches 0.868 and the inert Granite-4.0-1B a perfect 1.000 with 0.000 File F1.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vulnerability localization is a distinct capability from detection, repair or exploitation, and can be measured by scoring the file set an agent returns against the patch-touched implementation files.
- Repository-scale localization remains far from solved: the strongest system reaches 0.229 File F1 and 38.4% of tasks receive no correct localization from any of the 27 evaluated models.
- Repository structure predicts difficulty far better than model identity: the repository feature group carries 4.5x the summed Lasso coefficient weight of model category, and model-only features explain only R2 = 0.007 under cross-validation.
- Strong localization does not imply calibrated restraint: Antares-3B reaches 0.223 File F1 with a true negative rate of 0.034, while GPT-5.5 xhigh reaches 0.229 File F1 with TNR 0.279.
- Successful localization is associated with sustained, targeted and reliable tool use: mean commands per task correlates with File F1 at r = 0.72 across 27 model-level summaries.
Strengths
Localization is isolated from detection, repair and exploitation
Section 3 withholds advisory text, CVE identifiers, severity, affected version ranges and file hints, giving the agent only the generic MITRE CWE description and read-only terminal access. Table 1 documents that every prior benchmark either discloses the location (PrimeVul, JitVul, SecVulEval supply the function) or scores a downstream outcome whose location may be supplied explicitly (Vul4Py) or never observed (CyberGym).
A paired patched-snapshot phase that exposes the precision-restraint tension
Phase B re-runs the identical interface on the post-fix snapshot and scores whether the system reports nothing. This is what turns a leaderboard into a finding: GPT-5.5 xhigh leads Phase A at 0.229 File F1 but has TNR 0.279, while GPT-5 Nano scores 0.024 File F1 with TNR 0.868, and Antares-3B reaches 0.223 File F1 with TNR 0.034.
Release quality is genuinely high
Difficulty is attributed, not just measured
A deterministic, reproducible failure taxonomy
Weaknesses
Phase B's true negative rate is confounded with non-engagement
The 15-command budget is a binding constraint, not a neutral constant
Patch-touched files are a proxy for vulnerable files
Static-analysis baselines are structurally disadvantaged
Model coverage is budget-driven, not principled
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes a clean conceptual cut and then builds a benchmark around it. Detection asks whether code already selected is vulnerable; localization asks which code should have been selected. Table 1 substantiates the gap: every prior security benchmark either discloses the location (PrimeVul and JitVul supply the function, AutoTrace supplies the fixing commit) or scores a downstream outcome in which the location is supplied explicitly (Vul4Py) or never observed because correctness is decided by execution (CyberGym). VLoc Bench withholds advisory text, CVE identifiers, severity, version ranges and file hints, giving the agent only the generic MITRE CWE description and read-only terminal access. The scope is deliberately defensive and the threat model is stated up front: the target is the patch site, i.e. the files a defender would have to remediate.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is vulnerability localization and how is VLoc Bench different from other security benchmarks?
Vulnerability localization asks which files in a repository implement a given weakness, rather than whether a piece of code already selected is vulnerable. VLoc Bench gives an agent only a generic CWE description and read-only terminal access to an unfamiliar repository, with advisory text, CVE identifiers and file hints withheld, then scores the returned file list against the files modified by the real security fix. Most prior benchmarks either hand the model the code to judge or score a downstream outcome such as a patch or an exploit, so localization is never the scored output.
How hard is the benchmark for current AI systems?
Very hard. Across 27 models and four static-analysis tools, the best system (GPT-5.5 with xhigh reasoning effort) reaches 0.229 file-level F1, meaning precision 0.310 and recall 0.221. In 38.4% of the 500 tasks, no evaluated model localizes a single correct file. Performance also falls sharply with repository size, from a mean F1 of 0.598 on repositories under 100 KB to 0.058 on those over 10 MB.
What is Phase B and why does it matter?
Phase B presents the same agent with the patched repository under the identical interface, where the recorded vulnerability has been removed and the correct answer is to report nothing. It is scored by true negative rate. This catches a failure that localization-only benchmarks miss: systems that find vulnerable files well can still report unsupported locations on already-fixed code. For example Antares-3B scores 0.223 F1 in Phase A but a true negative rate of only 0.034 in Phase B.
Is the benchmark data and code publicly available?
Yes. The complete benchmark and evaluation code are published at github.com/cisco-foundation-ai/vulnerability-localization-benchmark with no gated access. The release includes a 500-task manifest with repository metadata, both commit SHAs, CWE labels and MD5 checksums, download and verification tooling, the Docker-based sandboxed evaluation harness, and the scoring and analysis scripts.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 could not be inspected; all values were read from the body text and the table and figure captions in the extracted file.
- Appendix D cost and runtime figures, and the per-ecosystem and severity bar charts, were read from captions rather than from the rendered plots.
- The 27-model leaderboard aggregates heterogeneous serving configurations, including provider-default decoding settings for closed-source models.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.