A Three-Axis Stress Test of LLM vs Classical ML for Network Intrusion Detection under Distribution Shift and Adversarial Evasion
Muhammad Ebad Atif, Muhammad Haider Ali
cs.LG · cs.AI · cs.CR
Quick verdict
Claims no universal LLM-vs-classical winner across three NIDS robustness axes; credible, well-caveated study weakened by single-seed RoBERTa results.
Summary
This empirical study benchmarks XGBoost against RoBERTa-base fine-tuned with LoRA for binary NetFlow intrusion detection across three axes: same-dataset performance, cross-dataset transfer, and adversarial evasion, using two independently collected NetFlow v2 datasets (NF-UNSW-NB15-v2 and NF-CSE-CIC-IDS2018-v2) under one shared scoring harness. Same-dataset, the models are effectively tied (F1 gaps ≤0.004). Under cross-dataset shift, XGBoost wins decisively UNSW→CIC (F1 0.8116 vs 0.6582); RoBERTa-LoRA's false positive rate hits 0.7756 with balanced accuracy 0.5477 — barely above chance — while CIC→UNSW collapses for every model (F1 0.02–0.07). Under a feature-space drift-to-benign evasion attack, RoBERTa-LoRA leads by roughly 17 F1 points at ε=0.5, with both models' FPR below 0.01. A staged feature-leakage ablation improves transfer non-monotonically, suggesting the leakage signal is distributed across features. Key limitations: RoBERTa results are single-seed point estimates, LLM checkpoints were selected on ~13% of the same-dataset test split, the LLM arm sees feature names in serialized text while classical models do not, only two networks are studied, and the attack is feature-space only.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Same-dataset F1 gaps between XGBoost and RoBERTa-LoRA never exceed 0.004 on either dataset, making same-dataset evaluation non-discriminative between them.
- Under cross-dataset transfer, XGBoost beats RoBERTa-LoRA by 15.3 F1 points UNSW→CIC (0.8116 vs 0.6582), where RoBERTa-LoRA's FPR of 0.7756 and balanced accuracy of 0.5477 reveal a near-chance classifier, and CIC→UNSW collapses for all models (F1 0.0201–0.0730).
- Under a feature-space drift-to-benign evasion attack, RoBERTa-LoRA leads XGBoost by 17.3 F1 points at ε=0.5 (34.8 at ε=0.2), with both models' FPR below 0.01 throughout the valid range.
- Staged feature-leakage removal improves cross-dataset transfer non-monotonically (Set A 0.0488, Set B 0.0282, Set C 0.8116), indicating the leakage signal is distributed across the feature representation.
- NIDS model selection should follow the anticipated threat model and be evaluated on multiple independent robustness axes with more than one metric per axis.
Strengths
One shared harness across every condition
Sec. 3.6 scores all three axes with a single evaluation function; the ε=0 adversarial row reproduces the same-dataset UNSW confusion matrix exactly, a genuine pipeline-consistency check.
Multi-metric honesty instead of F1-only reporting
FPR and balanced accuracy convert RoBERTa-LoRA's moderate 0.6582 cross-dataset F1 into the near-chance reality (FPR 0.7756, balanced accuracy 0.5477, Table 4), and validate its adversarial robustness (FPR ≤0.0081, Table 6).
Unusually candid limitations section
Careful degeneracy handling in the attack sweep
Multi-seed classical arm built in from the start
Weaknesses
Single-seed LLM results drive headline reversals
'Statistically tied' without any statistical test
Checkpoint selection contaminates the reporting split
Information asymmetry between arms is untested
Narrow coverage on the adversarial axis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether LLM-based and classical ML classifiers for NetFlow intrusion detection can be ranked at all, and answers no: XGBoost and RoBERTa-base+LoRA tie same-dataset, XGBoost wins under cross-dataset shift, RoBERTa-LoRA wins under adversarial evasion (Sec. 4.4, Table 7). The design is a deliberately staged stress test over two independently collected NetFlow v2 datasets (NF-UNSW-NB15-v2, ~2.39M flows/3.98% attack; NF-CSE-CIC-IDS2018-v2, ~18.9M flows/11.95% attack) sharing a 43-feature schema, with Random Forest as a secondary classical baseline. A code repository is referenced (footnote 0). The central claims are consistent with the reported tables, but the two headline reversals rest partly on single-run LLM estimates.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is an LLM or XGBoost better for network intrusion detection?
According to this paper, it depends on the evaluation axis. Same-dataset they tie (F1 gaps ≤0.004). Transferring to a different network, XGBoost wins decisively (F1 0.8116 vs 0.6582 for UNSW→CIC, where RoBERTa-LoRA's false positive rate reaches 0.7756). Under a feature-space adversarial evasion attack, RoBERTa-LoRA wins by about 17 F1 points at ε=0.5, with both models keeping false positive rates below 0.01. The recommendation should follow the anticipated threat model.
What datasets does the paper use?
Two public, independently collected NetFlow v2 datasets from the University of Queensland collection: NF-UNSW-NB15-v2 (~2.39M flows, 3.98% attack) and NF-CSE-CIC-IDS2018-v2 (~18.9M flows, 11.95% attack), both re-extracted into an identical 43-feature schema. Experiments use balanced samples of 50,000 flows per class and binary (benign vs attack) labels, since the datasets' attack taxonomies are nearly disjoint.
Is the code or data for this study available?
Yes. The paper links a reproduction repository (github.com/ebadatif/IDS-three-axis-evaluation) and both datasets are publicly available. The classical models were run under four seeds (42–45), while RoBERTa-LoRA results are single-seed point estimates due to compute cost, which the authors flag as a limitation.
Why did RoBERTa-LoRA fail under cross-dataset transfer?
When trained on UNSW and tested on CIC-IDS2018, it flagged 77.6% of benign flows as attacks (FPR 0.7756), yielding balanced accuracy of 0.5477 — barely above chance — despite a moderate F1 of 0.6582. In the reverse direction every model collapsed to near-zero recall. The authors hypothesize, without direct verification, that CIC-trained models learn a volumetric 'attack = high volume' rule that misses UNSW's quieter, behavioral attacks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are present only as captions; the plotted curves could not be visually inspected, though Tables 1–7 cover every number cited in this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.