Not All Relations Are Equal: Relation-Balanced and Calibrated Graph Learning for Provenance-Based Intrusion Detection
Lijie Zheng, Ji He, Alessandro Brighente, Yulong Shen, Mauro Conti
cs.CR · cs.LG
Quick verdict
RECAL reports near-perfect F1 on DARPA E3 via relation-balanced learning and error calibration; strong, but thresholds are tuned on test data, inflating gains.
Summary
The paper introduces RECAL, an unsupervised provenance-based intrusion detection framework motivated by the observation that relation frequencies in CADETS audit data differ by roughly 140,000×, which the authors argue causes two failure modes: learning capacity concentrating on frequent relations, and reconstruction errors that are not comparable across relations. RECAL combines (1) relation-balanced masked graph autoencoding — masking rates inversely scaled to relation frequency, per-relation linear decoders over edge-type-conditioned GAT encodings, and equal-weight per-relation loss averaging — with (2) relation-calibrated detection, where per-relation reconstruction errors are converted to empirical benign quantiles and fused with Fisher's method after a KNN candidate-screening stage. On DARPA E3 CADETS, THEIA, and TRACE, RECAL reports F1 of 99.99%, 99.93%, and 99.99% and mean FPR of 0.0004%–0.0041%, exceeding the best baselines (STGAN, MAGIC, FLASH) by 0.42–0.88 percentage points and cutting FPR versus STGAN by roughly 105×, 4×, and 41×. Ablations in Table 3 attribute gains to both components, and the authors honestly report 18 undetected THEIA nodes. Main limitations: the decision threshold is selected on test data at the best-F1 operating point, five-seed means are reported without dispersion, some baseline rows (Unicorn, AEGIS) appear transcribed with missing FPRs, and the χ2 approximation underlying Fisher fusion is acknowledged to be approximate under dependence. Code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Relation-frequency heterogeneity in provenance graphs (≈140,000× spread in CADETS) causes capacity misallocation toward frequent relations and cross-relation score miscalibration in existing PIDSs.
- Relation-stratified masking rates, independent per-relation decoders, and equal-weight per-relation loss averaging mitigate frequent-relation dominance during learning.
- Mapping per-relation reconstruction errors to within-relation benign empirical quantiles and fusing them with Fisher's method makes anomaly evidence comparable across relations and cuts false positives.
- On DARPA E3 CADETS/THEIA/TRACE, RECAL reaches F1 of 99.99%, 99.93%, and 99.99%, exceeding the best baseline by 0.88, 0.82, and 0.42 percentage points and reducing mean FPR versus STGAN by roughly 105×, 4×, and 41×.
Strengths
Quantified, well-motivated problem framing
Fig. 1 measures a ~140,000× spread in per-relation event counts on CADETS benign graphs, and §1 derives two concrete failure modes (capacity misallocation; cross-relation score miscalibration) tied to specific prior systems, rather than asserting heterogeneity abstractly.
Symmetric fix on both learning and decision sides, with a real ablation grid
§2.3 (Eq. 2 frequency-inverse masking, independent per-relation decoders, Eq. 3 equal-weight loss) and §2.4 (empirical-CDF quantile calibration, Fisher fusion Eq. 4) target the two named problems; Table 3 ablates them jointly and separately (w/o BL: F1 99.26/99.94/99.44; w/o CD: 96.05/99.44/99.51) and also compares z-score, max, and mean fusion variants.
Honest disclosure of failure cases
Calibration/fusion design space is explored, not just module on/off
Transparent, very low compute cost
Weaknesses
Decision threshold tuned on test data
Five-seed means without dispersion or significance testing
Baseline comparability is questionable
Fisher fusion independence assumption is conceded violated, unchecked
Hyperparameters under-specified in the text
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a method paper (short-conference, ICASSP-style format) on unsupervised host-based intrusion detection over provenance graphs. The motivating measurement (Fig. 1) is a ~140,000× spread in per-relation event counts across CADETS benign training graphs, from which §1 derives two failure modes of prior PIDSs: capacity misallocation toward frequent relations, and cross-relation score miscalibration. The corresponding claims are that relation-stratified masking with per-relation decoders (§2.3) fixes the learning side, and that empirical-quantile calibration with Fisher fusion (§2.4) fixes the decision side. Headline results (Table 2): F1 of 99.99%/99.93%/99.99% on CADETS/THEIA/TRACE, gains of 0.88/0.82/0.42 pp over the best baseline, and mean FPR of 0.0004%–0.0041%, a claimed ~105×/4×/41× reduction versus STGAN, the lowest-FPR comparator.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is RECAL and what problem does it solve?
RECAL is an unsupervised intrusion detection framework for provenance graphs built from kernel audit logs. It targets the fact that system-call relation types (read, write, execute, etc.) occur at wildly different frequencies — about 140,000× apart in the CADETS dataset — which makes models over-fit frequent relations and makes anomaly scores incomparable across relation types. It fixes this with frequency-aware masking during graph-autoencoder training and by converting reconstruction errors to per-relation percentile scores before combining them.
How well does RECAL perform on the DARPA E3 datasets?
The paper reports F1 scores of 99.99% on CADETS, 99.93% on THEIA, and 99.99% on TRACE, with mean false positive rates between 0.0004% and 0.0041%. This is 0.42–0.88 percentage points above the strongest baselines compared (including STGAN, MAGIC, FLASH), and the FPR is claimed to be roughly 105×, 4×, and 41× lower than STGAN, the baseline with the lowest reported FPR. Note these numbers use a threshold chosen on the test set at the best-F1 point.
Is RECAL's code and data available?
The paper links to a public implementation at github.com/Jiex2001/RECAL, and the experiments use the publicly available DARPA Transparent Computing Engagement 3 datasets (CADETS, THEIA, TRACE) with THREATRACE-style node-level attack labels, so the evaluation should be reproducible, though several hyperparameter constants are only in the code.
What are the main limitations of RECAL?
Three stand out: (1) the alarm threshold is selected on test data, so reported numbers are best-case operating points; (2) results are averaged over five random seeds without variance estimates, which matters because the FPRs are so tiny; and (3) some baseline numbers appear to be taken from prior papers (Unicorn and AEGIS lack FPR values), making exact comparability unclear. The authors also note 18 genuinely undetected attack nodes on THEIA whose local behavior closely resembles benign activity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1, 3, 4, and 5 appear only as extracted vector-text labels; curve shapes and axis values (e.g., sensitivity ranges in Fig. 4, before/after separation in Fig. 3) could not be visually verified.
- Several hyperparameter constants (p0, pmin/pmax, decay λ, KNN screening threshold, Word2Vec setup) are not stated numerically in the text; reproducibility assessment assumes the released code supplies them.
- Per-baseline provenance (locally re-run vs transcribed) is not stated in the text and could not be verified.
- The paper is in short conference format (ICASSP-style); some evaluation-protocol details may be deferred to the repository.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.