Evaluating Coding Agents on Kernel Exploit Generation
Junyoung Jang, Gwanhyun Lee, Hwiwon Lee, Kyuheon Kim, Jongseong Kim, Jinho Jung, Lingming Zhang
cs.AI · cs.CR
Quick verdict
Claims first benchmark for verifier-defined kernel exploit primitives; results are credible but single-run and platform-primitive confounded.
Summary
KEX-bench evaluates coding agents on exploit-primitive generation against real kernels: 45 tasks over 40 Linux/Windows CVEs spanning kernel address leak, instruction-pointer control, heap read, heap write, and arbitrary address write. Each task runs in an isolated QEMU VM, exposes tools only through MCP servers under a tool-call budget, and decides success via deterministic verifiers checking per-run randomized kernel state. The strongest configuration (Claude Code with Opus 4.6) solves 14/25 Linux tasks (56.0%) and 1/20 Windows tasks (5.0%) without a reference PoC, and 31/45 (68.9%) with one. Trajectory analysis shows a trigger-to-primitive gap: agents reproduce crashes but fail to shape kernel state into verifier-checked primitives. Limitations include one trajectory per configuration, platform-primitive confounding, no equivalent adversarial validation of the Windows verifier, and limited contamination control.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- KEX-bench is the first benchmark to measure verifier-defined exploit-primitive generation by coding agents against real Linux and Windows kernels, with deterministic per-run-randomized success criteria.
- Current agents show a large capability gap by platform and setting: the strongest pair (Claude Code/Opus 4.6) solves 14/25 Linux (56.0%) vs 1/20 Windows (5.0%) tasks without a reference PoC, and 31/45 (68.9%) with one.
- A trigger-to-primitive gap exists: agents frequently reach crashes or KASAN-confirmed vulnerable paths but fail to convert kernel state into verifier-accepted LEAK/RIP/HEAPR/HEAPW/AAW primitives.
- Main bottlenecks are unreliable trigger discovery and failed trigger-to-primitive conversion under verifier checks, not merely insufficient tool budget.
Strengths
Verifier-defined primitives, not crash counting
Success requires primitive-specific kernel-state evidence: LEAK checks the reported _text against /proc/kallsyms; RIP scans the ring buffer for a General Protection Fault at a per-run random goal address; HEAPR/HEAPW use a custom heap_verifier module planting per-boot random values across nine kmalloc size classes, reported via panic/die notifiers even after crashes (§2.2.1). This directly operationalizes the trigger-versus-primitive distinction the paper is about.
Adversarially validated Linux verifiers
Appendix D validates the four Linux verifiers over 324 executions of 108 cases — output spoofing, trigger-only payloads, wrong targets, stale-target replay after reboot, and fail-closed execution faults — with zero false accepts/rejects in that campaign (Table 4). Randomized per-run targets also defeat replay of public exploit artifacts.
Controlled, fully logged evaluation pipeline
Unusually honest self-limitation
Weaknesses
One trajectory per configuration; rankings lack error bars
Platform fully confounded with primitive
No equivalent adversarial validation of the Windows verifier
Severe contamination exposure
Small per-primitive task counts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
KEX-bench targets a capability boundary that bug-finding benchmarks (Cybench, NYU CTF Bench, CVE-Bench) do not measure: whether a coding agent converts a real operating-system kernel vulnerability into a verifier-observable exploit primitive. The benchmark contains 45 task instances across 40 CVEs (20 Linux, 20 Windows) and five primitives — kernel text leak (KASLR defeat), RIP control, heap read, heap write (Linux only), and arbitrary address write (Windows only). Three central claims are made: (1) it is the first benchmark of verifier-defined kernel primitive generation; (2) the strongest configuration (Claude Code/Opus 4.6) solves 14/25 Linux (56.0%) vs 1/20 Windows (5.0%) w/o PoC and 31/45 (68.9%) w/ PoC (Table 2); (3) trajectories expose a trigger-to-primitive gap — agents reach crashes but fail to shape kernel state into the checked primitive.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is KEX-bench?
KEX-bench is a benchmark for evaluating AI coding agents on generating exploit primitives — kernel address leak, instruction-pointer control, heap read, heap write, and arbitrary address write — against real Linux and Windows kernel vulnerabilities (40 CVEs, 45 tasks). Success is judged by deterministic verifiers that check randomized kernel state per run, not merely a crash.
How well do AI agents perform on kernel exploitation tasks in this study?
The strongest configuration (Claude Code with Opus 4.6) solved 14 of 25 Linux tasks (56.0%) and 1 of 20 Windows tasks (5.0%) without a reference proof-of-concept. With a reference PoC provided, it solved 31 of 45 tasks (68.9%). Across all 990 configuration-task outcomes, agents solved 15.2%.
What is the 'trigger-to-primitive gap' reported in the paper?
It is the observed pattern that agents often manage to trigger the vulnerability or crash the kernel, but fail to shape kernel memory state into the exact primitive the verifier requires (e.g., leaking the kernel base or writing an exact value to an exact address). Among failed Claude Code/Opus runs on Linux without a PoC, 7 of 11 had already reached confirmed fault evidence.
Is the benchmark reproducible and publicly available?
The authors release configurations, prompts, verifiers, environment setup, and reproduction instructions at kex-bench.github.io. Licensed Windows images and binaries are not redistributed; authorized users rebuild them locally with provided scripts.
Can this benchmark results be trusted given the models may have memorized public exploits?
Partially: per-run randomized verifier targets prevent replaying stale exploit artifacts, but they do not prevent models from using memorized knowledge of public CVEs and write-ups. Only three distinct post-cutoff CVEs are available, so the paper reports coverage rather than a clean pre/post comparison.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3 and 4 are available only as text fragments; visual effort/token distributions could not be analyzed.
- Several referenced artifacts (Appendix B model details beyond the summary, full failure tables' semantics) are only partially present in the extracted text.
- The paper cites 'Claude Mythos Preview' (Carlini et al., 2026); this name could not be cross-checked and may reflect an extraction or naming anomaly.
- Pricing/version identifiers (GPT-5.4, Opus 4.6, Kimi K2.5, Gemma 4) are taken as stated in the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.