How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE
cs.CR · cs.AI · cs.CL · cs.LG
Quick verdict
Single-direction weight edit cuts a 320B MoE's refusal by 41–89pp; careful, honest study, but one model and hook-based decomposition limit scope.
Summary
The paper tests whether directional ablation — training-free removal of a model's refusal by projecting a single 'refusal direction' out of the residual-stream writer weights — still works on GLM-5.3-Flash, a 320B-parameter mixture-of-experts with 288 routed experts, a four-stream hyper-connection (mHC) residual and block-FP8 weights, none of which the original recipe (validated on dense models up to ~70B) assumes. Enumerating all 12,479 writer matrices by tensor structure and iterating FP8 requantization 32 times (leakage 13.4%→5.2%), refusal on seven harmful benchmarks falls from 0.51–1.00 to 0.08–0.48 (41–89 percentage points, McNemar p<10⁻⁷), with no detected change on four capability benchmarks and XSTest false refusal dropping to zero. Lossless-hook decomposition of all seven writer-group subsets shows strong non-additivity: attention, dense and expert groups alone remove 0.039/0.016/0.148 while jointly they remove 0.776, so name-matching implementations silently remove only 0.066. The mHC scalar stream mixing provably preserves orthogonality, and writer-level editing beats layer-boundary projection (0.174 vs 0.265, p=3×10⁻⁵). A random orthogonal direction removes nothing, yet violence/sexual/hate residue survives all twelve category-fitted subspaces at ranks 1–12. Main limitations: a single model, judge-based scoring without human labels, an exploratory non-nested rank sweep, and withheld code.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Directional ablation transfers to a 320B sparse-MoE with a four-stream mHC residual and block-FP8 weights, cutting refusal on seven harmful benchmarks by 41–89 percentage points (0.51–1.00 → 0.08–0.48) with no detected capability change and zero false refusal on XSTest.
- The removal effect is strongly non-additive over writer groups: attention, dense and routed experts remove 0.039/0.016/0.148 alone but 0.776 jointly, so 74% of the effect exists only under the joint intervention.
- Name-matching implementations reach only 0.066 of the 0.776 on an MoE because fused expert tensors are invisible to module traversal, producing a silent failure rather than an error.
- mHC's scalar cross-stream mixing provably preserves orthogonality, so writer-level ablation remains valid — and editing the writers is more thorough than a layer-boundary projection (0.174 vs 0.265 paired, p=3×10⁻⁵).
- Removal is direction-specific (a random orthogonal direction removes nothing), yet a residue concentrated in violence, sexual content and hate survives every rank-1-to-12 subspace fitted on those categories.
Strengths
Order-free writer-group decomposition
§5.2 measures all seven non-empty subsets of {attention, dense+shared, routed experts} with lossless hooks (Table 7) instead of an order-dependent ladder, and explicitly refuses to read the Shapley shares (experts 54%, attention 32%, dense 14%) as statements about where refusal lives.
Contamination-controlled evaluation
Clean AdvBench/JailbreakBench/StrongREJECT variants draw only from outside the 256-prompt fitting slice (Table 4/Table 5); the clean-vs-contaminated differences (e.g., 0.152 vs 0.131 on AdvBench) fall inside CIs, and all generalization claims rest on the clean rows.
Statistical discipline
Engineering honesty with measured costs
Candid limitations section
Weaknesses
Single-model evidence base
Decomposition measured at the hook ceiling
Rank sweep is confounded and underpowered
Judge-only safety scoring
Code withheld
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether directional ablation (Arditi et al., 2024) — training-free removal of refusal by projecting one direction out of the residual-stream writers — survives three architectural shifts at once: sparse MoE (288 routed experts, top-8), a four-stream manifold-constrained hyper-connection (mHC) residual, and block-FP8 checkpoints. The subject is GLM-5.3-Flash (320B total / 18B active, 45 layers + MTP). Central claims: (1) the attack survives, cutting refusal on seven harmful benchmarks from 0.51–1.00 to 0.08–0.48 (Table 5; 41–89 pp, all McNemar p<10⁻⁷) with no detected capability change (Table 6) and XSTest false refusal 0.024→0; (2) the effect is strongly non-additive — attention/dense/expert groups alone remove 0.039/0.016/0.148 while the joint edit removes 0.776 (Table 7), so the conventional name-matching recipe reaches 0.066 and fails silently; (3) scalar mHC mixing preserves orthogonality, and writer-level editing beats a layer-boundary projection (0.174 vs 0.265, paired p=3×10⁻⁵); (4) removal is direction-specific (a random orthogonal control does nothing) yet a violence/sexual/hate residue survives twelve category-fitted subspaces at ranks 1–12 (Table 10).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does the 'refusal direction' ablation attack still work on mixture-of-experts models?
Yes for the one model tested. Applied to GLM-5.3-Flash (a 320B-parameter MoE), a single-direction weight edit cut refusal on seven harmful benchmarks by 41–89 percentage points. But it only works if the edit reaches every residual-writer weight — including the 12,384 expert down-projections hidden inside fused tensors that conventional name-matching code silently skips; touching only the conventional targets removes just 0.066 of the 0.776 achievable.
What is directional ablation?
A training-free technique that removes an LLM's refusal behavior: estimate a 'refusal direction' as the difference of mean residual-stream activations between harmful and harmless prompts, then project that direction out of every weight matrix that writes to the residual stream. It needs no gradients or optimization — only a few hundred prompts and a forward pass — which is why it circulates widely for uncensoring open-weight models.
Is the uncensored GLM-5.3-Flash checkpoint publicly available?
Yes. The paper studies an already-public edited checkpoint (orcarouter/GLM-5.3-Flash-Uncensored-FP8 on Hugging Face), released under the same license as the base model. The authors do not release their editing code, the fitted refusal direction, or the category subspaces they fitted.
What part of the model's safety training resisted removal?
Refusal on violence, sexual content, and hate/harassment prompts survived every attempted edit: subspaces fitted specifically on those categories at ranks 1 through 12 still left refusal between 0.20 and 0.47, while prompts about illegal procedures and cybercrime were almost fully released. The paper cannot explain what carries this residue.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract supplied with the task is cut mid-sentence ('...measurable refusal at every'), but the embedded full text contains the complete abstract and body through Appendix A, so no review-affecting truncation exists.
- The extraction contains tables but no figure content; if the published PDF includes figures, their content could not be assessed from this text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.