The Geometry of Refusal: Why Post-Hoc Safety Is Fragile and Pretraining-Time Safety Persists
Srikanth Malla, Chiho Choi, Joon Hee Choi
cs.LG · cs.AI
Quick verdict
Argues post-hoc safety is a thin Fisher-orthogonal gate, pretraining co-training persists; strong framework, but evals are in-distribution, small-n.
Summary
The paper offers a geometric account of why post-hoc safety alignment (RLHF/DPO) is fragile while safety co-trained during pretraining persists. Projecting the safety update Δ onto the top-k eigenspace of an empirical capability Fisher, the authors find every measured post-hoc install sits in a 'suppression' regime: overlap ρ runs 21–66× a random floor yet 2–3 orders below the 'erasure' boundary ρ_part=10⁻², with qnorm curvature in the hundreds. A kernel-immobility lemma implies such thin, near-orthogonal updates mask rather than remove capability; accordingly 100 benign Alpaca fine-tuning steps cut AdvBench refusal on Qwen-2.5-7B (93→58.2%) and Llama-3-8B (49.2→10.9%) at preserved capability. A nine-revision OLMo-2-1B sweep dates the representational 'substrate' refusal engages to a sharp window around 6B–60B pretraining tokens. Constructively, from-scratch continuous safety co-training holds 87–98% refusal across 410M–6.9B that stays at 84–91% after the attack — but coverage is in-distribution (AdvBench-like), with HarmBench and StrongReject remaining low. Main limitations: no matched post-hoc baseline at equal refusal strength, regex-based refusal scoring (a Llama-Guard cross-check exists), mostly single seeds with n=64–256, and NLL capability metrics shown to be format-confounded.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Every measured post-hoc safety install (DPO/RLHF across five families, 1B–8B) lies in a suppression regime: ρ is 21–66× the random floor yet ≥2 orders below ρ_part=10⁻², with qnorm reaching 200+.
- A kernel-immobility lemma shows updates in the capability-Fisher kernel can mask but not first-order remove capability, so 100 benign Alpaca steps cut refusal 35–38 pp on Qwen-2.5-7B and Llama-3-8B at preserved general capability.
- The representational substrate a refusal circuit engages emerges sharply in OLMo-2-1B between roughly 6B and 60B pretraining tokens, with per-revision DPO geometry transitioning at the same point.
- From-scratch continuous safety co-training at 410M–6.9B installs 87–98% refusal that survives the benign-SFT attack at 84–91% (in-distribution), while a compute-matched windowed schedule installs none — persistence of the safety signal, not its timing, drives robustness.
Strengths
A unifying, well-specified geometric instrument with a proper null
The (ρ, qnorm) plane (Sec. 3, App. B) comes with a random-subspace null and z-scores of 157–517 SDs (Table 4), turning scattered fragility findings (Qi et al. 2024; Arditi et al. 2024; Wei et al. 2024) into one measurable regime classification.
Both sides of the regime boundary are probed empirically
An adversarial-SFT control engineered to saturate the capability eigenspace reaches max ρ=1.04×10⁻² with qnorm 33,945 (App. G, Table 12), showing 'erasure' is reachable while all DPO/RLHF installs stay ≥50× below ρ_part.
Unusually thorough internal cross-checks
Clean causal isolation in the constructive result
High reproducibility posture
Weaknesses
The erasure boundary is anchored by only one of two controls
Thin statistical power where the headline numbers live
Co-training robustness is in-distribution only, with train/eval overlap
Capability preservation judged by a metric the paper itself shows is unreliable
'Capability directions' is a narrow proxy
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper unifies several known failures of post-hoc LLM safety alignment under one geometric description, then uses it constructively. Claims: (C1) every measured post-hoc install (shipped Qwen/Llama/Mistral instruct deltas; self-trained DPO on Pythia-1.4B) lands in a 'suppression' regime — update overlap ρ with the capability-Fisher top-k is 21–66× the k/d random floor yet ≥2 orders below the erasure boundary ρ_part=10⁻², with qnorm up to 220.6 (Table 8); (C2) a kernel-immobility lemma (App. H) implies such updates cannot reduce capability loss at leading order, so benign fine-tuning restores refusal: 100 Alpaca steps take Qwen-2.5-7B from 93.0→58.2% and Llama-3-8B from 49.2→10.9% AdvBench refusal (Table 1); (C3) the refusal substrate in OLMo-2-1B emerges sharply around 6B–60B pretraining tokens (Table 13); (C4) from-scratch continuous co-training at 410M–6.9B holds 87–98% refusal, 84–91% post-attack (Table 3), while a compute-matched windowed schedule installs none (Table 2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does benign fine-tuning make safety-aligned LLMs unsafe again?
This paper argues post-hoc safety training (RLHF/DPO) installs a thin, high-curvature 'refusal gate' that is nearly orthogonal to the model's capability directions. Because the underlying capability is masked rather than removed, as few as 100 benign instruction-tuning examples reroute the model to its original behavior — AdvBench refusal fell from 93% to 58% on Qwen-2.5-7B-Instruct and 49% to 11% on Llama-3-8B-Instruct.
What is the difference between the suppression and erasure regimes?
The paper classifies weight updates by ρ, the fraction of the update lying inside the top-128 eigenspace of an empirical Fisher matrix of capability losses. Suppression (ρ far below 10⁻²) masks capability expression and stays recoverable; erasure (ρ at or above 10⁻²) actually moves capability directions. All measured RLHF/DPO installs were suppression; only a deliberately adversarial SFT control reached the erasure boundary.
Does training safety during pretraining work better than RLHF afterwards?
In the paper's from-scratch Pythia experiments (410M–6.9B parameters, 10B tokens), continuously mixing a safety loss into pretraining produced 87–98% refusal that stayed at 84–91% after a benign fine-tuning attack, versus 35–38 percentage-point losses for post-hoc installs. Important caveat: this held only on in-distribution harmful prompts (AdvBench-like); coverage of HarmBench and StrongReject stayed low, and no equally strong post-hoc baseline was available for a matched comparison.
Can I reproduce this paper's experiments?
The authors state the full pipeline is in the supplementary code, all models are public checkpoints (Pythia, OLMo-2, Qwen-2.5, Llama-3, Mistral), recipes include seeds and hyperparameters, and total compute was roughly 80 B200-GPU-hours. The geometry measurements and attacks on public instruct models are the most accessible entry point; from-scratch co-training at 410M needs about 14 GPU-hours per run on a B200-class node.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- In-figure axis labels and tick text (Figs. 1–5) are garbled by PDF extraction, but every number cited in this review comes from body text or tables, which extracted cleanly.
- The abstract string in the task metadata is cut mid-sentence ('reach 87'), but the full abstract inside the provided paper text is complete; this was not treated as truncation.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.