FUSE: An Evaluating Framework for Dangerous Capabilities of LLMs
Zhengyi Jin, Ru Zhang, Xiao Chen, Xinbo Liu, Jiaxuan Lin, Jia Huang, Jianyi Liu, Zhen Yang
cs.AI
Quick verdict
Modular K/D/H framework profiles dangerous capabilities of 12 commercial LLMs; internally consistent and honest, but harm claims limited by judge reliability.
Summary
The paper introduces FUSE, a modular evaluation framework that measures dangerous capabilities of LLMs along three pipelines—Knowledge (IRT-estimated ability on multiple-choice items), Defense (adaptive five-round red-team elicitation scored on a 4×25 rubric), and Harm (rubric-scored actionability of open-ended responses, conditional on non-refusal)—aggregated with derived indicators (DL, BR, ∆, ER) into a seven-dimensional capability profile φ. Instantiated with a chemical-biological module (298 scenario seeds, 3,773 MCQs, 200 harm queries), it evaluates 12 commercial models from the Claude, GPT, DeepSeek, and Kimi families released 2023–2026, plus a 16-scenario cyber pilot for three overlapping models. Reported findings: family profiles diverge sharply (DeepSeek combines the highest knowledge with the lowest defense and BR>1.09 erosion; Claude defends near-perfectly yet shows 67.8–78.6% silent refusal); harm scores cluster in a narrow 68–78 band across all models and the 3.5-year window; defense rankings survive judge substitution (bootstrap Spearman ρ>0.79 for four of five alternative judges), while harm-judge agreement is only moderate (ρ 0.42–0.58). Main limitations: per-model harm sample sizes are unreported and harm estimates for strong refusers rest on small compliance-conditioned subsets; there is no human-expert validation of judges; the red-team attacker model and prompts are undisclosed; temporal evolution claims lack trend statistics; code and dual-use assets are available only upon request.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A unified three-pipeline protocol (K, D, H) plus derived indicators yields a seven-dimensional dangerous-capability profile φ comparable across models and domains, with weakly coupled dimensions (D–H ρ=0.318).
- Commercial LLM families show sharply divergent risk structures: DeepSeek pairs the highest knowledge with the lowest defense and escalating compliance (BR>1.09), while Claude combines near-perfect defense with 67.8–78.6% silent refusal.
- Harm is inelastic: compliant outputs stay in a narrow 68–78/100 actionability band across all 12 models and a 3.5-year release window, so risk growth comes from rising knowledge and inconsistent refusal, not more harmful content.
- Defense rankings are robust to judge choice (bootstrap Spearman ρ>0.79 for 4 of 5 alternative judges), while harm scoring is only moderately reliable (ρ 0.42–0.58).
Strengths
Comparability by construction
The four-module architecture and formal pluggable-module interface (§3.1–3.2, Table 1) achieve cross-model comparability through one protocol, one judge, and one model population rather than post-hoc normalization; the cyber pilot (§5.6) transfers the rubric structure unchanged to a second domain.
Verifiable internal arithmetic
Derived quantities recompute exactly from raw tables: ∆(Opus-4)=(97.0−82.0)/97.0=+0.155 and ∆(V4-Pro)=(67.2−80.1)/80.1=−0.161 (Tables 4, 6); family means in §5.7 (Claude D=91.3, DeepSeek H=75.8) match Tables 4–5; Table 10 aggregates follow scenario-count weighting (Opus-4 D=89.9=(156·97.0+142·82.0)/298).
Honest reliability reporting
Phenomenological yield beyond aggregate scores
Serious dual-use ethics
Weaknesses
Unreported per-model H sample sizes
No human-expert validation of judges
Undisclosed red-team agent
Temporal claims under-inferenced
Marginal IRT payoff
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
FUSE (arXiv 2609.02168) is a framework paper: a modular evaluation infrastructure for *dangerous capabilities* of LLMs, instantiated with a chemical-biological (CB) module and piloted on cyber. Each model passes through three pipelines — Knowledge K (2PL IRT ability on 3,773 MCQs), Defense D (five-round adaptive red-team induction across 298 scenario seeds, scored on a 4×25 rubric), Harm H (rubric-scored actionability of open-ended responses, conditional on non-refusal) — plus derived indicators DL, BR, ∆, ER, aggregated into a seven-dimensional profile φ (Eq. 20). Claims: (1) a unified protocol makes φ vectors comparable across models and domains; (2) 12 commercial models (Claude, GPT, DeepSeek, Kimi; 2023–2026) show sharply divergent family profiles; (3) temporal analysis shows knowledge compounding, defense diverging by family, harm 'inelastic' in a 68–78 band; (4) reliability is grounded in cross-judge agreement (bootstrap ρ>0.79 for 4/5 judges) and weak inter-dimension correlation (ρ∈[0.32, 0.52]).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the FUSE framework for LLM dangerous capabilities?
FUSE is a modular evaluation infrastructure that scores each LLM along three pipelines—Knowledge (multiple-choice items analyzed with item response theory), Defense (adaptive five-round red-team elicitation), and Harm (rubric-scored actionability of compliant responses)—and aggregates them with derived indicators into a seven-dimensional dangerous-capability profile intended for cross-model and cross-domain comparison.
Which models did FUSE evaluate and what did it find?
Twelve commercial models from the Claude, GPT, DeepSeek, and Kimi families released between 2023 and 2026, tested on a chemical-biological danger module. It found sharply different family profiles—DeepSeek pairs the highest knowledge with the weakest defense, Claude defends almost perfectly but silently refuses up to about 79% of turns—while harm scores stayed in a narrow 68–78 out of 100 band for every model and time point.
Is FUSE's code or data publicly available?
Not directly. The paper states code and data are available upon reasonable request, and provides reviewers a minimal non-sensitive reference implementation; the chemical-biological scenario seeds, harm probes, MCQ bank, and raw interaction logs are deliberately withheld for dual-use safety reasons.
How reliable are FUSE's scores?
Defense rankings are fairly robust: four of five alternative LLM judges agree with the primary judge (GPT-4o-mini) at bootstrap Spearman ρ above 0.79. Harm scoring is weaker, with judge agreement of ρ 0.42–0.58, and no human-expert validation is reported, so harm comparisons between models should be read cautiously.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as jumbled text fragments (axis labels, point annotations of Figures 3, 7, 8, 11–16); their visual content could not be inspected, though every number cited in this review comes from tables or prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.