SCRIPTIOC-BENCH: A Benchmark for Recognizing Actionable Threat Intelligence from Script-Based Malware using LLMs
Hanna Kim, Jian Cui, Minkyoo Song, Hwanjo Heo, Seungwon Shin, Kimin Lee, Xiaojing Liao
cs.CR · cs.AI
Quick verdict
Claims LLMs still fail at static IOC extraction (best 65.4 F1); benchmark well built and honestly reported, but stats are thin and URL matching lenient.
Summary
SCRIPTIOC-BENCH measures whether LLMs can statically recover indicators of compromise (IOCs) from malicious scripts without executing them. The corpus comprises 634 manually verified JavaScript, PowerShell, and VBScript malware samples drawn from Filescan.io (Nov 2025–Mar 2026) and the Hynek Petrak collection (2015–2017), covering four IOC types (URLs, domains, IPs, filesystem artifacts), annotated by two security experts and stratified into Direct, Fixed-transform, and Program-dependent recovery levels. Ten open-weight models from 8B to 480B plus Gemini 2.5 Pro/Flash and GPT-5.6-terra are evaluated with map-reduce chunking for long scripts, against three regex baselines. The best model reaches only 65.4 F1; frontier models recover up to 91.7% of Direct IOCs but only 31–46% of Program-dependent ones. A ten-category false-positive taxonomy shows small models fall back to prompt-echo errors while frontier models produce sample-grounded reconstruction errors. On Qwen3-8B, deterministic string-utility tools plus SFT/GRPO adaptation raise precision from 31% to 48% without helping Program-dependent recovery (recall stays near 1–3%). Limitations: no confidence intervals or significance tests, no inter-annotator agreement statistics despite the two-expert protocol, lenient query-insensitive URL matching as the primary metric, acknowledged contamination risk on the historical subset, and mitigations validated on a single model.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Static IOC recovery without execution remains unsolved for current LLMs: the strongest model (GPT-5.6-terra) reaches only 65.4 overall micro F1 (Table 4), far above regex baselines (URLExtract 22.4) but far from reliable.
- Recovery difficulty stratifies sharply by annotation level: frontier models recall up to 91.7% of Direct IOCs but only 31–46% of Program-dependent IOCs (Table 5), with multi-operation transforms (≥3 ops: 19.0% for GPT-Terra) and loop-constructed values the hardest.
- Longer scripts degrade both reliability and capability: response-failure rates rise and Regex-missed Recall falls with token length (Figure 3), making inflated context a practical obstacle even for 1M-context frontier models.
- Smaller and larger models fail differently — small models echo prompt IOC examples while frontier models produce sample-grounded near-copy/granularity errors — and on Qwen3-8B, deterministic string tools plus SFT/GRPO adaptation are complementary, raising precision from 31% to 48% while leaving Program-dependent recovery near 1–3% (Figures 5–6).
Strengths
Ground truth curated for static recoverability — the key conceptual advance
§4.1 retains only IOCs fully determined by the script alone (no dynamic-only artifacts from platform reports), and each indicator is labeled Direct/Fixed-transform/Program-dependent via a released, deterministic fixed decoder (Appendix J, Table 9). This per-IOC stratification is what enables the paper's sharpest result: the 91.7% vs 31–46% recall cliff (Table 5).
Fine-grained, operational FP taxonomy with sensitivity analysis
Table 6 plus Appendix H defines ten categories with a deterministic precedence rule; numeric thresholds (diff ratio 0.90/0.80, edit distance ≤3) were tuned on a held-out split from a non-evaluated model, with sensitivity reported (±0.05 shifts ≤2% of FPs across adjacent categories). This makes the error analysis in §5.4 reproducible rather than anecdotal.
Honest operational reporting
Mechanism-level diagnosis beyond aggregate scores
Breadth of models, languages, and baselines
Weaknesses
No statistical rigor around headline numbers
Inter-annotator reliability never quantified
Primary metric is the lenient one
Mitigation study has a distribution-overlap confound and n=1
Some analytic cells rest on very few instances
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SCRIPTIOC-BENCH (arXiv 2609.06149) claims to be the first benchmark restricting IOC ground truth to *statically recoverable* indicators from script malware. Its four research questions map to four evaluation axes: overall capability (RQ1), difficulty stratification (RQ2), input-complexity robustness (RQ3), and failure-mode characterization/mitigation (RQ4). Central claims: best LLM reaches only 65.4 F1 (GPT-Terra, Table 4); frontier models recover up to 91.7% of Direct IOCs but 31–46% of Program-dependent ones (Table 5); length degrades reliability; and tools+adaptation lift Qwen3-8B precision from 31% to 48%. The claims are scoped honestly — the abstract and §7 do not overstate readiness for operational use.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the best performance LLMs achieve on SCRIPTIOC-BENCH?
The strongest model, GPT-5.6-terra (called GPT-Terra), reaches 65.4 overall F1, with Gemini 2.5 Pro close behind at 65.0. Performance splits by difficulty: frontier models recover up to 91.7% of IOCs that appear directly in the code, but only 31–46% of IOCs requiring sample-specific program logic such as decoding or reconstruction. Regex baselines top out at 22.4 F1.
What kinds of malware and indicators does the benchmark cover?
634 manually verified malicious scripts: 480 JavaScript (97 recent from Filescan.io plus 383 historical from the Hynek Petrak 2015–2017 collection), 75 PowerShell, and 79 VBScript. Four indicator types are annotated — URLs, domains, IP addresses, and filesystem artifacts — and each indicator is labeled with a recovery level: Direct (verbatim in source), Fixed-transform (undone by a standard decoder), or Program-dependent (needs following the script's own logic).
Can fine-tuning or giving tools to small LLMs fix IOC extraction?
Partly. In the paper's controlled study on Qwen3-8B, deterministic string-utility tools (Base64/hex/XOR decoding, concatenation, split/join) plus supervised fine-tuning and GRPO reinforcement learning raise precision from 31% to 48% and sharply cut hallucinated 'prompt echo' indicators. But recall on Program-dependent IOCs stays near 1–3% — deep, script-specific reconstruction remains unsolved even for frontier models.
Is the SCRIPTIOC-BENCH dataset and code publicly available?
The paper states that datasets, evaluation code, and a public leaderboard will be released. The appendices already document the full prompts, chunking budget formula, false-positive taxonomy rules with thresholds, and training hyperparameters (LoRA rank 128, reward weights), which is above-average reproduction support — contingent on the artifacts actually shipping.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–5 and 7–9 are extracted as interleaved inline text (axis labels, bar values, captions), so figure-only details such as the leaderboard screenshot (Appendix A) and exact heatmap gradations cannot be fully assessed; all numeric claims in this review come from the intact tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.