Reasoning Topology Matters: A Controlled Study of LLM-Based Cybersecurity Analysis
Jiling Zhou, Aisvarya Adeseye, Antti Hakkala, Seppo Virtanen, Jouni Isoaho
cs.CR · cs.AI
Quick verdict
GoT prompting beats CoT/few-shot by 9.8–12.2pp on 3 security tasks; trend credible, but Table II's perfectly uniform +4.0pp gaps undermine trust.
Summary
The paper proposes 'Security Reasoning Topology,' a framework that abstracts LLM reasoning structures in cybersecurity into three topologies — Linear, Branching, Graph — instantiated via Chain-of-Thought, Tree-of-Thought, and Graph-of-Thought prompts. Reasoning instructions are placed in the system prompt while task inputs stay in the user prompt, aiming to isolate topology effects. Five models (Llama 2 7B/13B/70B, GPT-5.1, Mistral Large 3) are evaluated on three datasets — MITRE ATT&CK-labeled network traffic, a 1,100-report CTI corpus, and CVE records — against few-shot, JSON-schema, and RAG baselines. Reported aggregate accuracy (Table I): GoT is best on all datasets (85.0/85.0/82.0), +9.8–12.2 points over few-shot and ahead of the strongest baseline RAG (80.6/80.8/76.8), with paired-bootstrap significance (p<0.001 vs few-shot; p<0.05 vs RAG). The GoT>ToT>CoT ordering reportedly holds for every model (Table II), and an ablation finds system–user prompt separation adds +1.9 (CoT), +2.9 (ToT), +3.7 (GoT) points. Key limitations: no token or latency cost analysis (ToT/GoT also receive a doubled 2048-token budget — a confound), no failure-mode analysis, and no released code, prompts, or sample sizes. Most seriously, Table II shows exactly +4.0-point gaps between consecutive topologies for all five models and Table I gives identical ATT&CK/CTI values for all topology rows — implausibly uniform patterns that undermine confidence in the reported numbers.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reasoning topology — organizing intermediate reasoning as Linear, Branching, or Graph via CoT/ToT/GoT system prompts — substantially changes LLM accuracy on cybersecurity analysis tasks.
- Graph (GoT) reasoning achieves the highest accuracy on all three datasets (85.0/85.0/82.0 on ATT&CK/CTI/CVE), improving over few-shot prompting by 9.8–12.2 percentage points and significantly over RAG (p<0.05).
- The topology ordering GoT > ToT > CoT is consistent across Llama 2 7B/13B/70B, GPT-5.1, and Mistral Large 3.
- Separating topology instructions into the system prompt improves accuracy, with gains growing with topology complexity (+1.9/+2.9/+3.7 pp for CoT/ToT/GoT).
Strengths
Cleanly controlled comparison design
Sec. IV-C fixes task inputs in user prompts and varies only system-level reasoning instructions; a dedicated prompt-organization ablation (user-only vs merged vs separated, Sec. V-C) quantifies the mechanism itself (+1.9/+2.9/+3.7 pp for CoT/ToT/GoT).
Breadth of the evaluation grid
Three heterogeneous datasets (ATT&CK traffic, 1,100-report CTI corpus, CVE records) × five models spanning families and scales (Llama 2 7B/13B/70B, GPT-5.1, Mistral Large 3), compared against three non-topological baselines (few-shot, JSON-schema, RAG).
Multiple runs with significance testing
Honest limitation disclosure
Weaknesses
Implausibly uniform result tables
Topologies are prompt text, not verified mechanisms
Compute confound favoring ToT/GoT
Irreproducible: no artifacts or sampling detail
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.24710, University of Turku) asks whether the *structure* of intermediate reasoning — linear, branching, or graph — affects LLM performance on cybersecurity analysis. It introduces 'Security Reasoning Topology' (Sec. III), instantiating the three structures as CoT, ToT, and GoT delivered through system-level prompts while task inputs remain in user prompts (Sec. IV-C). Central claims: (1) GoT is most accurate on all three datasets — 85.0 (ATT&CK), 85.0 (CTI), 82.0 (CVE) vs few-shot 75.2/72.8/70.2, i.e., +9.8/+12.2/+11.8 points (Table I); (2) the ordering GoT > ToT > CoT holds for Llama 2 7B/13B/70B, GPT-5.1, and Mistral Large 3 (Table II); (3) separating topology instructions into the system prompt adds +1.9/+2.9/+3.7 points for CoT/ToT/GoT (Sec. V-C).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Security Reasoning Topology?
It is the framework proposed by this paper: LLM reasoning on cybersecurity tasks is abstracted into three structures — Linear (sequential, instantiated as Chain-of-Thought), Branching (multiple candidate paths, as Tree-of-Thought), and Graph (interconnected, reusable states, as Graph-of-Thought) — controlled entirely through system-level prompts while task inputs stay in the user prompt.
How much does graph-style (GoT) reasoning improve accuracy according to the paper?
The paper reports GoT as the best method on all three datasets: 85.0 on ATT&CK traffic, 85.0 on CTI, and 82.0 on CVE, which is 9.8, 12.2, and 11.8 percentage points above few-shot prompting respectively, and also ahead of a RAG baseline (80.6/80.8/76.8).
Does the improvement hold for small and different LLMs?
The authors report the same ordering (Graph > Tree > Linear reasoning) for Llama 2 7B/13B/70B, GPT-5.1, and Mistral Large 3, suggesting scale-independence — though the reported gaps are suspiciously uniform (+4.0 points everywhere) and no code or raw results are available for independent verification.
Is code or data available for this paper?
No. The paper releases no code, no prompt texts, and no raw results. The ATT&CK dataset is public (UWF-ZeekData24), but the CTI and CVE datasets are unversioned Kaggle uploads, and sample sizes are not stated, making replication difficult.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (topology illustration) cannot be assessed from the extracted text.
- Extraction appears complete for this short conference-format paper (abstract through references); no appendix or prompt listings exist in the provided text.
- Table values were transcribed from PDF text; minor OCR risk exists, though all values reconcile internally.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.