When Safety Routing Breaks: Understanding Alignment Fragility under Benign Fine-Tuning
Yitong Guo, Xiaoyi Chen, Siyuan Zhang, Xiaofeng Wang, Haixu Tang
cs.CR · cs.AI
Quick verdict
Proposes Fisher-geometry account of safety collapse under benign fine-tuning; convergent evidence but geometry on one family, 2 seeds, no significance tests.
Summary
The paper asks why safety alignment in LLMs collapses so readily under benign fine-tuning. Rejecting gradient-conflict as the primary cause, the authors propose a Fisher-geometric account: the safety Fisher is low-rank (top-1 concentration 0.491 vs 0.271 for utility), alignment flattens it by roughly two orders of magnitude, and 100 benign fine-tuning examples selectively re-sharpen late output-side MLP blocks — final-layer down_proj sharpness rises 11.2× for safety versus 1.2× for utility. Behaviorally, full-parameter SFT on 100 Alpaca samples raises HEx-PHI attack success rate from 0% to 59.6–85.7% across Llama-3.1-8B and Qwen2.5-7B alignment variants, while utility (MMLU, BoolQ, ARC-E) degrades far less at that scale. Logit-lens read-outs and cross-condition activation patching — a final-layer patch restores 96.9% of the aligned refusal margin — support a routing account in which safety representations persist but lose control of the output. Random, high-conflict, and low-conflict sample subsets all collapse safety, so gradient conflict modulates severity but is not necessary. LoRA and ASAM suppress early collapse yet fail at 5,000 samples, and 10 safety examples (50 after a 5k attack) or a fixed refusal prefix restore refusal, indicating intact representations. Limitations: SFT/DPO alignment only, 7–8B open models, English-only benchmarks, two random seeds without significance testing, and Fisher geometry reported for a single model family.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Benign fine-tuning breaks safety by selectively re-sharpening a low-rank output-side Fisher routing subspace (final-layer down_proj 11.2× for safety vs 1.2× for utility), not through gradient conflict, which only modulates severity (Table 5).
- Alignment compresses the safety Fisher by roughly two orders of magnitude while preserving a routing pathway; after FT-100, safety-relevant representations remain intact while the output mapping to refusal is disrupted.
- Final-layer activation patching restores 96.9% of the aligned refusal margin, causally implicating late-layer computation in the refusal/compliance switch.
- LoRA and ASAM suppress early collapse but their protection fails at 5,000 samples, indicating sharpness control alone cannot preserve alignment at scale.
- 10 safety examples (50 after a 5,000-sample attack) or a fixed refusal prefix restore refusal, showing the broken component is output routing rather than safety knowledge.
Strengths
Convergent multi-level evidence for one mechanism
Fisher curvature drift (Fig 3: final-layer down_proj 11.2× for safety vs 1.2× for utility), logit-lens refusal–compliance margins (Fig 4), and causal activation patching (Fig 5: a final-layer patch restores 96.9% of the aligned refusal margin) all localize the failure to the same late-layer output-side locus.
Directly tests the dominant prior hypothesis instead of ignoring it
§4.3 deploys two published gradient-conflict sampling pipelines (He et al. 2024; Guan et al. 2025) and shows that even bottom-conflict 100-sample subsets exceed 52% ASR across all alignment conditions (Table 5), grounding the rejection of the outlier-conflict account in data.
Unusually broad behavioral grid for a mechanism paper
Reversibility results with practical significance
Weaknesses
Only two random seeds, no inferential statistics
Mechanistic core demonstrated on a single model family
Mitigation comparison confounded by optimization strength
Asymmetry claim needs a scale qualifier the abstract omits
Introduction overstates the rejection of gradient conflict
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a well-defined question: why does refusal behavior collapse under *benign* fine-tuning while the utility task itself survives? The authors reject gradient-conflict explanations (§4.3, Table 5) and advance a Fisher-geometric mechanism with four linked claims: (1) the safety Fisher is low-rank — top-1 spectral concentration 0.491 vs 0.271 for SciQ utility, rising to 0.718 vs 0.223 in the final layers (Table 1), with Kaiser-criterion ranks of 8 vs 11; (2) alignment flattens the safety Fisher by roughly two orders of magnitude (Fig 2); (3) FT-100 selectively re-sharpens late output-side MLP blocks — final-layer down_proj sharpness rises 11.2× for safety vs 1.2× for utility (Fig 3); (4) safety representations survive while output routing breaks: a final-layer activation patch restores 96.9% of the aligned refusal margin (Fig 5), and a fixed refusal prefix cuts Align-256 ASR from 85.70% to 34.85% with no parameter update (§4.5). A fifth claim covers mitigation: LoRA and ASAM suppress early collapse but fail at 5,000 samples (§4.4, Fig 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does fine-tuning an LLM on harmless data break its safety alignment?
The paper argues refusal is implemented as a low-rank, output-side routing pathway. Even 100 benign examples selectively re-sharpen the curvature (Fisher) of late MLP modules — 11.2× in the final layer for safety vs 1.2× for utility — so small updates flip the refusal/compliance switch while the underlying safety knowledge stays intact. Random benign data suffices; adversarially selected outliers are not required.
Can safety be restored after benign fine-tuning breaks it?
Yes, cheaply. Fine-tuning on 10 safety examples reversed a 100-sample benign attack (ASR back to 0%), and 50 examples sufficed after a 5,000-sample attack. Even without parameter updates, forcing a refusal prefix like 'I'm sorry' cut ASR from 59.60% to 2.12% on one aligned Llama-3.1 variant, showing representations survive and only the routing is disrupted.
Do LoRA or sharpness-aware minimization (ASAM) prevent safety collapse?
Only partially. Both suppress the attack-success-rate jump at 100 fine-tuning samples (e.g., Align-10k: LoRA 11.0% vs full SFT 59.6% ASR), but protection fades by 5,000 samples, where LoRA itself climbs to 31.2% ASR. The authors conclude sharpness control alone cannot preserve alignment at scale.
What is the Fisher-geometric explanation of alignment fragility?
The empirical Fisher measures how sharply refusal behavior responds to parameter changes per module. The paper finds the safety Fisher is more concentrated (top-1 mass 0.491 vs 0.271 for utility), alignment flattens it by roughly 100×, and benign fine-tuning re-sharpens only late output-side MLP blocks — localizing fragility to a small routing subspace rather than global knowledge loss.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 are graphics: captions, axis labels and annotations (e.g., '11.2x', '1.2x') were extracted, but curve shapes and per-layer bar details could not be visually verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.