SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction
Yuanxiang Shi, Jiayi Lin, Xuanyong Lin, Liangcai Su, Yeheng Duan, Wei Wang, Qi Han, Bing Zhao, et al.
cs.SE · cs.AI
Quick verdict
Recasts vuln discovery as branch-targeted input prediction; sound design, consistent numbers, but single-run trials and no fuzzer baseline limit confidence.
Summary
SWE-Test reframes LLM-agent vulnerability discovery as input prediction with a closed ground truth. Using coverage-guided fuzzing (AFL++) over real C/C++ projects, the authors mine deep branches that a fuzzer newly reached, withhold the satisfying input, and ask an agent to construct one. Three modes decompose the ability: Open-loop (source only), Feedback-enabled (adds a coarse distance oracle), and Online Arena (no target; 12-hour runs scored by normalized coverage gain over a 72-hour-saturated fuzzer corpus). The fixed-target pool covers 60 tasks over 16 programs in 13 domains; Online Arena adds six programs, for 22 codebases and 15 domains. Across 15 default-effort model-scaffold configurations, Qwen3.8-Max with Claude Code leads at 55.0% Feedback-enabled pass rate; paired Claude Code means rise from 19.3% (Open-loop) to 36.4%. In Online Arena, five configurations produced 18 detections, manually consolidated to 13 distinct confirmed bugs across six programs; Qwen3.8-Max-Preview found seven. Failure analysis attributes most errors to constraint satisfaction rather than navigation, modulated by domain. Key limitations: one trial per task with no significance testing or confidence intervals; no classical baseline (e.g., budget-matched directed fuzzing toward the same branches) to establish the agent's marginal value; and manual bug confirmation whose criteria and reliability are not reported. The harness, tasks, and turnkey evaluation environment are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vulnerability discovery by LLM agents can be measured as branch-targeted input prediction with a closed, deterministic ground truth that resists contamination and avoids recall bias against an unknowable vulnerability set.
- The best of 15 default-effort model-scaffold configurations reaches 55.0% pass rate in the Feedback-enabled mode (Qwen3.8-Max with Claude Code), and paired Claude Code means rise from 19.3% (Open-loop) to 36.4% with feedback.
- Constraint satisfaction, not code navigation, is the dominant failure bottleneck; feedback raises target-function reach from 62.6% to 75.2% and near-to-pass conversion from 30.8% to 48.4%.
- In 12-hour Online Arena runs, five configurations produced 18 bug detections that reduce to 13 distinct manually confirmed bugs across six programs, with Qwen3.8-Max-Preview accounting for seven.
- Measured performance is materially sensitive to scaffold choice and reasoning effort, so uncontrolled model rankings are difficult to interpret.
Strengths
Closed, deterministic ground truth
Every fixed target comes from an observed AFL++ execution with a withheld satisfying input (Sec. 3.3), so grading is execution-based (Sec. 3.2) rather than recall against an unknowable vulnerability set — a genuine fix for the contamination and scoring biases the paper diagnoses in CyberGym/CTF-style benchmarks.
Clean ability decomposition via paired design
Open-loop and Feedback-enabled share the same 60 targets, making the feedback effect a controlled ablation: target-function reach 62.6%→75.2% and near-to-pass conversion 30.8%→48.4% (Sec. 4.3, Table 5); the leveled reward separates navigation failure from constraint failure.
Internally consistent reporting
Candid threat analysis and targeted ablations
Informative domain stratification
Weaknesses
Leaderboard built on single runs with no statistical treatment
Missing budget-matched classical baseline
Verifier signal underspecified; possible oracle leakage
Manual bug confirmation is an unstandardized instrument
Thin related work and a self-serving comparison table
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
SWE-Test: Benchmarking LLM Vulnerability Discovery via Input Prediction — Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SWE-Test benchmark?
SWE-Test measures how well LLM coding agents find software vulnerabilities by asking them to construct an input that reaches a specific deep branch in a real C/C++ program. The branches were discovered by AFL++ coverage-guided fuzzing, the satisfying input is hidden, and grading is purely execution-based, giving a closed ground truth instead of scoring against an unknowable list of vulnerabilities.
How does SWE-Test prevent data contamination and memorization?
Targets are branch conditions tied to specific builds of real programs, found by fuzzing and never published; the hidden input satisfying each branch has no public answer to memorize, unlike CTF-style benchmarks whose solutions leak into model training data (the paper cites a ~14% solution-leakage rate on InterCode-CTF).
Which models scored best on SWE-Test?
Qwen3.8-Max paired with the Claude Code scaffold leads: 55.0% pass rate in the Feedback-enabled mode and 7 of the 13 confirmed bugs found in the 12-hour Online Arena. Across seven Claude Code configurations, adding feedback raises the average pass rate from 19.3% to 36.4%.
What do LLM agents fail at when hunting for vulnerabilities?
Mostly satisfying the input constraints guarding a target branch rather than locating the relevant code: in the paper's experiments agents reach the target function in 75.2% of feedback-enabled trials but convert only 48.4% of those into full passes, and the dominant failure mode varies sharply by software domain (e.g., 82% navigation failures on SQL parsers vs 17% on PHP).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The separately provided metadata abstract is cut mid-sentence, but the embedded full text contains the complete abstract, so no content was lost.
- Figures 2-4 were assessed via captions and surrounding text only; their visual content is not recoverable from the text extraction (Figure 1 and Figure 5 data are embedded as text and were used).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.