Benchmark Scores Are Pipeline-Dependent: A Reliability Audit of Cybersecurity LLM Benchmarks
Aymene Berriche, Cathrine Shalby, Mohannad Alhanahnah, Yazan Boshmaf
cs.CR · cs.AI · cs.CL
Quick verdict
Audits 8 cybersecurity benchmarks × 10 LLMs, showing scores shift >80pp with pipeline choices; 15 failure modes, rigorous, well-caveated meta-evaluation.
Summary
The paper treats LLM benchmarks as configurable measurement pipelines rather than fixed datasets, and audits eight cybersecurity benchmarks (48,662 questions, 23 tasks) across ten proprietary, open-weight, and security-specialized LLMs. Formalizing each benchmark as a five-stage pipeline (dataset, prompt, inference, extraction/scoring, aggregation) with nine configuration fields, the authors inspect all 120 benchmark–failure-mode pairs and identify 15 recurring failure modes. Documented effects are large: a newline stop sequence truncating reasoning outputs costs Qwen3.6 85.9 points on RedSage-Bench; an API rejecting SecEval's five-token budget leaves GPT-5.4 scored 0.3% on letter matches mined from error payloads; denominator conventions inflate Gemma-4's CTI-RCM score from 0.2% to 100%. PCA shows one component explains 95.25% of variance across the 23×10 score matrix, while semantically similar task pairs rank models only weakly consistently (Kendall τ-b 0.29, 0.24). Under a standardizing harness, nine of ten models shift at least three ranks somewhere; Spearman ρ between original and standardized rankings falls to −0.47 on MMLU-CS, with bootstrap analysis ruling out sampling noise. Judge dependence is controlled via an independent judge (99.6% agreement). Limitations: the standardization choices are themselves contestable (Table 12), only ten models are tested, perturbation analysis is admittedly opportunistic, generation-level variance is unestimated, and 44 of 72 pipeline fields remain unreproducible from documentation. Code and the sayf-eval harness are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cybersecurity LLM benchmark scores are pipeline-dependent: a single configuration choice can shift a model's score by more than 80 percentage points (85.9 pp from a newline stop sequence; 81.1 pp from a rejected token budget) without changing model or dataset.
- Fifteen recurring failure modes span all five pipeline stages across the eight audited benchmarks (34 observed incidents over 120 inspected benchmark–failure-mode pairs).
- Semantically similar task pairs (CTI-Bench vs AthenaBench vulnerability scoring and attacker attribution) rank the same ten models differently (Kendall τ-b 0.29 and 0.24) due to incompatible scoring conventions, metric directions, and alias handling.
- Under an evaluation harness standardizing nine pipeline fields, nine of ten models shift by at least three ranks on at least one benchmark (only GPT-5.4 does not), and these shifts are robust to question-sampling bootstrap.
- 44 of 72 pipeline configuration fields (61%) across the eight benchmarks cannot be reproduced from documentation alone (36 undefined, 8 contradicted).
Strengths
End-to-end audit with full coverage, not anecdotes
All 120 benchmark–failure-mode pairs were inspected (Table 4: 34 incidents; stage-level denominators in Table 18), grounded in a formal five-stage pipeline model with nine configuration fields (§2). The specification census (App. C: 44/72 fields, 61%, unreproducible from documentation; 8 contradicted between paper and released code) is independently valuable.
Disciplined evidence typology
Table 16 classifies each failure as re-scoring (outputs held fixed, isolating downstream fields exactly), re-generation, or paired evaluation; peer-gap estimates are explicitly marked † and never laundered into controlled-ablation claims (App. F.1 concedes perturbation is 'opportunistically' possible).
Above-norm uncertainty quantification
Judge circularity confronted directly
Actionable deliverables
Weaknesses
Headline rank shifts are conditional on contestable conventions
Gold-label audit is indirect and incomplete
Limited external validity and no generation-level variance
Some effects are peer gaps, not ablations
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether cybersecurity LLM benchmark scores measure models or measurement pipelines. It formalizes each benchmark as a five-stage pipeline Sb(m) = Ab(Eb(Ib(m, Pb(Db)))) with nine configuration fields (§2), audits eight benchmarks (Table 1: MMLU-CS, CyberMetric, SecBench, SecEval, SECURE, CTI-Bench, AthenaBench, RedSage-Bench — 23 tasks, 48,662 questions) on ten models (Table 2), and claims three things: (1) 15 recurring failure modes span all five stages; (2) single configuration choices shift scores by >80 pp and reorder rankings; (3) standardization moves nine of ten models by ≥3 ranks on at least one benchmark. Scope is explicitly measurement *reliability*, not construct validity (Limitations), and the harness is framed as diagnostic rather than a uniquely correct evaluator — a framing that pre-empts the most obvious objection before reviewers raise it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does it mean that LLM benchmark scores are pipeline-dependent?
A benchmark score depends not only on the questions and the model but on configurable evaluation choices: prompt template, decoding parameters, token budget, stop sequences, answer extraction, scoring rule, denominator, and aggregation. This paper shows that in cybersecurity benchmarks a single such choice can change a model's score by more than 80 percentage points — for example, a newline stop sequence cut one model from 85.9% to 0.0%, and a rejected five-token budget left GPT-5.4 'scoring' 0.3% on letter matches inside HTTP error messages.
Which benchmarks and models were audited?
Eight cybersecurity benchmarks — MMLU-CS, CyberMetric, SecBench, SecEval, SECURE, CTI-Bench, AthenaBench, and RedSage-Bench — totaling 23 tasks and 48,662 questions, evaluated on ten LLMs including GPT-5.4, Claude Sonnet 4.6, Gemma-4, Qwen3.6, Llama-3.3, GPT-OSS, and four cybersecurity-specialized models (Primus-Nemotron, Primus-Merged, Foundation-Sec, RedSage-Qwen3).
Is the code and evaluation harness available?
Yes. The audit code is released at github.com/qcri/cyberbench-audit and the evaluation harness (sayf-eval) is an open-source Python package on PyPI. Raw model outputs, pipeline configurations, and meta-evaluation results (published to EveryEvalEver) are released, so extraction and scoring changes can be re-scored without re-running the models.
Does this mean LLM leaderboard rankings are meaningless?
No. The paper argues scores are conditional measurements, not that evaluation is impossible. Its recommendations are to publish the full pipeline with each score (an 'evaluation pipeline card'), report invalid-response rates and rank stability, store raw outputs, and avoid comparing scores produced under incompatible conventions — treating leaderboard positions as pipeline-dependent rather than intrinsic model properties.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt (head and tail) with three marked [...sampling gap...] regions covering parts of §4.3–§4.4, the middle of the references, and portions of the appendices; some main-text failure narratives (e.g., F2(I)–F3(I)) were read via the surviving appendix evidence, and appendix content beyond the sampled regions could not be verified.
- Figures 1–3 and the full Table 13/28 contents are available only through extracted captions and surrounding text; fine-grained visual or cell-level details were not independently verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.