Ranked by the Matcher: A Reproducibility Audit of Knowledge Graph Extraction from Threat Reports
Safayat Bin Hakim, Houbing Herbert Song
cs.CR · cs.AI · cs.CL
Quick verdict
Re-scoring 10 CTI extractors under 8 matchers flips 11/45 rankings; a candid, rigorous audit, but the hosted/offline serving split remains observational.
Summary
This paper asks how much published rankings of cyber threat intelligence knowledge-graph extractors are decided by the matching protocol rather than by the systems. Auditing twelve recent papers, the authors could reimplement the stated matching rule for only five. Re-scoring outputs of ten systems on fifty shared documents under eight protocols reverses eleven of forty-five pairwise orderings, and one fixed prediction set spans 0.16–0.70 triplet F1. On GRID's 378-item human-adjudicated calibration set, no mechanical matcher—lexical, embedding, or entailment—agrees with reviewers above roughly 0.71, while an LLM judge reaches 0.860. The authors also build CTIForge, a five-stage neuro-symbolic pipeline whose deterministic validation layer toggles while extraction stays byte-identical, and ablate it over seven backbones: all four hosted backbones gain precision under validation, all three offline ones lose it, tracking the share of validator actions disputing entity type. A serving-versus-decoding decomposition on Qwen2.5-7B attributes the sign to the serving stack (+0.0214), though on a single backbone only. Main limitations: the hosted/offline split is observational with confounded factors; two hosted deltas fall below the measured pass-to-pass noise; the ranking analysis covers fifty of 149 documents; the matching-rule audit was single-coder; and the judge figure is inherited from prior work. Code, protocols, and per-triple audit records are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The choice of matching protocol reverses 11 of 45 pairwise system orderings and moves one fixed prediction set from 0.1577 to 0.6994 triplet F1 across seven protocols.
- No mechanical matcher — lexical, embedding, or entailment — exceeds about 0.71 agreement with human adjudication on GRID's 378-item set, versus 0.860 for an LLM judge.
- With extraction held byte-identical, deterministic validation raises precision on all four tested hosted backbones and lowers it on all three offline ones, tracking the share of validator actions disputing entity type (roughly 1 in 8 hosted vs 1 in 3 offline).
- On the one backbone servable both ways (Qwen2.5-7B), the serving stack rather than decoding carries the sign of the validation effect (+0.0214 serving vs −0.0057 decoding), consistent with hand-written rules encoding frontier-model conventions.
- Corpus-pooled matching credits cross-report predictions and inflates true positives by 6.4% for CTIForge (5.9% for CTI-Nexus), about +0.03 F1, invisibly in any results table.
Strengths
Clean shared-extraction ablation design
Section 5.5 derives every ablation arm from one extraction pass with deterministic downstream stages, so arms differ only by the module under test; five replicate MiniMax-M3 passes put pass-to-pass SD at 0.0058 in ΔP, quantifying exactly the confound the design removes.
External calibration of the metric instead of assumption
Section 6.1/Finding 1 scores ten protocols (exact, soft-lexical, embedding τ, NLI τ) against GRID's 378 human-adjudicated items and reports the ceiling honestly: 0.598–0.706, entailment included specifically to test whether semantic matchers escape it (they do not), versus 0.860 for the LLM judge.
Unusual candor and self-limitation
Actionable recommendations each backed by a measured defect
Reproducibility infrastructure is genuinely strong
Weaknesses
Sign-split fragility at the noise floor
n=1 serving decomposition carrying an abstract-level claim
No inferential statistics anywhere
Single-coder, single-pass audit of the twelve papers
Ranking analysis characterizes a borrowed cache, not published systems
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether published rankings of CTI knowledge-graph extractors measure the systems or the scoring protocols. Four central claims: (1) matching rules are rarely specified well enough to reuse — Table 1 finds 5 of 12 papers reimplementable, 3 naming a similarity family without an encoder/threshold, 4 stating nothing; (2) orderings are protocol-sensitive — re-scoring ten systems from GRID's prediction cache on 50 shared documents under eight protocols reverses 11 of 45 pairwise orderings, and one fixed prediction set (the 149-document GPT-4o run) spans 0.1577–0.6994 triplet F1, a 0.5417 spread, across seven protocols (Fig. 3b); (3) on GRID's 378-item human-adjudicated set, ten mechanical protocols agree with reviewers at only 0.598–0.706, versus 0.860 for the LLM judge; (4) with extraction byte-identical, deterministic validation raises precision on all four hosted backbones and lowers it on all three offline ones (Table 3), tracking entity-type dispute share (≈1 in 8 actions hosted vs 1 in 3 offline, Fig. 4), with a serving×decoding 2×2 on Qwen2.5-7B attributing the sign to the serving stack (+0.0214 vs −0.0057 decoding, Appendix M).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do F1 scores for the same CTI extraction system differ so much between papers?
Because each paper uses a different rule for deciding when a predicted (subject, relation, object) triple counts as matching a gold annotation — string equality, name overlap, embedding similarity, or vague semantic matching. This audit shows the same predictions score anywhere from 0.16 to 0.70 triplet F1 depending only on the matcher, and 11 of 45 system orderings flip across eight protocols.
What is CTIForge?
A five-stage pipeline for building knowledge graphs from cyber threat reports: one LLM extraction call per text chunk, then deterministic validation (type constraints, indicator repair, evidence checks), canonicalization, and cross-document fusion. It is released as measurement apparatus — its validation layer can be switched on or off while the extraction output stays byte-identical, which is what lets the authors isolate downstream effects per backbone.
Are embedding-based matchers good enough to evaluate knowledge graph extraction?
Not on the evidence here. On GRID's 378 items adjudicated by three human reviewers, no mechanical matcher — lexical, embedding at any tested threshold, or textual entailment — agreed with humans above about 0.71; the dominant failure is under-matching (the strictest protocol misses 40.2% of human-credited matches). An LLM judge reached 0.860, though that figure is inherited from the GRID paper rather than re-measured.
Does adding a symbolic validation layer always improve LLM extraction?
No. Holding extraction fixed, validation raised precision for all four tested hosted backbones but lowered it for all three locally-served models, which removed more correct triples than bad ones. The authors attribute this to hand-written rules encoding the conventions of the frontier model they were developed against — deterministic does not mean model-agnostic. A serving-vs-decoding test on Qwen2.5-7B points to the deployment stack, not the weights, as the deciding factor.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal data (the Fig. 2 reversal matrix, Fig. 3a/3b points, Fig. 4 scatter, Fig. F1 distributions) survives extraction only as garbled axis-label digit streams; every number cited in this review comes from the prose and tables, which are intact and internally consistent.
- No truncation detected: the extracted text runs complete through Appendix M (Table M1), including all references and appendices A–M.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.