SyzHarness: Patch-Based Kernel Bug Reproduction with LLM-Synthesized Fuzzing Harnesses
Xingyu Li, Juefei Pu, Haonan Li, Arrdya Srivastav, Kareem Shehada, Srikanth V. Krishnamurthy, Zhiyun Qian
cs.CR · cs.AI
Quick verdict
Claims 78% (KernelCTF) / 73% (SyzDirect) kernel bug reproduction from patches via LLM harness + fuzzing; credible, honest ablations, but no code release.
Summary
SyzHarness targets patch-based reproduction of Linux kernel vulnerabilities: given a security patch but no PoC, it must both recover the trigger scaffold (syscall sequence, state construction) and find the concrete argument values that trigger the bug. The system uses an LLM agent (GPT-5.2-codex, grounded with code-navigation tools via MCP) to synthesize a parameterized fuzzing harness that fixes setup logic and exposes only uncertain, bug-critical parameters; the harness is mechanically translated into a Syzkaller pseudo-syscall plus syzlang description, and failed runs are refined using file/function/line reachability feedback across up to five iterations within a roughly 24-hour budget. A patch-differential oracle (crash on pre-patch kernel, absent post-patch, matching trace) validates candidates. Results: 78% on 100 KernelCTF cases (median TTR 1.2h), 73% vs 25% for a controlled SyzDirect rerun on its 100-case benchmark, and 40/50 on syzbot bugs fixed after March 2026; 45.3% operational yield on 64 unlabeled 2026 patches. Ablations show the LLM-only baseline at 65% and that hierarchical feedback mainly improves efficiency rather than final success (46/60 vs 47/60). Limitations: no released code or artifacts, a single frontier LLM, no statistical significance testing, and run-to-run variability (78-87% on the ablation subset) comparable in size to some reported deltas.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Patch-based kernel bug reproduction decomposes into trigger-scaffold recovery (suited to LLM reasoning) and concrete trigger-value discovery (suited to coverage-guided fuzzing).
- SyzHarness achieves 78% patch-validated reproduction on 100 KernelCTF cases with 2.9h mean time-to-reproduction.
- Under matched single-run settings and a common patch-differential oracle, SyzHarness reaches 73% on the SyzDirect benchmark versus 25% for a rerun of SyzDirect.
- Results are not explained by runtime retrieval of public PoCs or by training-data memorization, based on log inspection and knowledge-cutoff analysis.
- Hierarchical reachability feedback primarily improves time-to-reproduction rather than final reproduction success (Table IV).
Strengths
Rigorous validation oracle
§V-A patch-differential oracle: a PoC must reproducibly crash the pre-patch kernel, stop crashing post-patch, and match the patched function/root cause; ambiguous cases are manually excluded. This prevents crediting unrelated crashes, a rigor most LLM-reproduction papers lack.
Contamination-aware evaluation
Manual inspection of 30 agent logs found no web/PoC retrieval (§V-C1); a GPT-5.1-codex-max cutoff split shows 70.69% pre- vs 69.05% post-cutoff success; the 50 syzbot bugs fixed after March 2026 use only fix commits with public reproducers withheld.
Honest ablations and stability analysis
Clear decomposition with a worked example
Practical cost profile
Weaknesses
No statistical uncertainty quantification
Baseline comparison rests on an unpublished rerun
No artifact release
Dataset and environment skew
Unexplained race-condition mechanism
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SyzHarness (UC Riverside, arXiv 2609.23889v2) targets automated reproduction of Linux kernel vulnerabilities when only the fix patch is available — a real gap the authors substantiate via kernelCTF's 90-day PoC embargo and a collected set of 64 patches from 2026 with no public PoC or CVE (§I). The paper formulates reproduction as two subproblems (§III): recovering the trigger scaffold (syscall sequence, protocol state, object lifecycle) and finding the concrete trigger values. Its thesis: directed fuzzing (SyzDirect) is weak at the former, LLM-only generation at the latter. Headline claims: 78% patch-validated success on 100 KernelCTF cases (mean/median TTR 2.9h/1.2h, Table I); 73% vs 25% for SyzDirect on its own 100-case benchmark under matched single-run settings; 40/50 on syzbot UAF/OOB bugs fixed after March 2026 using fix commits only; 45.3% operational yield on 64 unlabeled patches (explicitly not framed as a success rate).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SyzHarness?
A research system from UC Riverside that automatically reproduces Linux kernel vulnerabilities when only the security fix patch is available. It uses an LLM to build the setup logic (syscall sequence and kernel state) as a parameterized Syzkaller fuzzing harness, then lets Syzkaller mutate only the uncertain, bug-critical values until the bug crashes. Candidates are confirmed with a patch-differential check: the input must crash the pre-patch kernel and stop crashing after the patch.
How successful is SyzHarness?
On 100 Google KernelCTF cases it patch-validated 78% of reproductions (median about 1.2 hours per success). On the 100-case SyzDirect benchmark, a controlled single-run comparison gave 73% for SyzHarness versus 25% for a rerun of SyzDirect. It also reproduced 40 of 50 syzbot bugs fixed after March 2026 using only their fix commits. On 64 unlabeled 2026 patches without public PoCs, it produced validated reproductions for 45.3%.
Does SyzHarness cheat by using public PoCs or memorized exploits?
The authors report two checks: manual inspection of 30 agent logs found no retrieval of public reproducers or writeups, and swapping to an older model with an earlier knowledge cutoff (GPT-5.1-codex-max) still yielded 70%, with nearly identical rates on pre- versus post-cutoff cases (70.69% vs 69.05%). The post-March-2026 syzbot results also post-date model training knowledge.
Is the SyzHarness code available?
The paper as published on arXiv does not mention a public code or artifact release. Implementation details and the full agent prompts are described in the paper's appendix, but the harness corpus, scripts, and per-case logs are not stated to be available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3-6 (pipeline diagram, per-iteration success distribution, run-frequency histogram) cannot be inspected directly; related numbers were taken from captions and inline narrative text.
- The abstract in the extracted text is truncated mid-sentence; headline numbers were cross-checked against §V.
- Review is based on arXiv v2 extracted text; camera-ready revisions may differ.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.