Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
Hoang Cuong Nguyen, Mark Dras, Usman Naseem
cs.CL
Quick verdict
Compares SFT/Ra-SFT/ORPO refusal circuits across 3 LLMs; finds objectives reshape circuits, an alignment trilemma; solid but single-run steering stats.
Summary
This paper asks whether the choice of post-training objective — beyond training data — determines how refusal is implemented inside language models. The authors fine-tune Llama-3.1-8B, Gemma-2-9B and Qwen3-8B from base checkpoints under three objectives, holding data and hyperparameters fixed: supervised fine-tuning (SFT), reasoning-augmented SFT (Ra-SFT, trained on chains justifying safety decisions), and ORPO. Using difference-in-means refusal directions, activation and attribution patching, and ActAdd/ITI steering, evaluated on WildJailbreak, StrongREJECT, XSTest and an MMLU subset, they find Ra-SFT produces a distinct, MLP-dominant and bootstrap-stable refusal circuit across all three architectures; SFT concentrates causal effects in few attention heads (Llama head 25, -0.33); and steerability is architecture-dependent (Gemma Ra-SFT recognition-layer steering cuts ASR 18.8pp at alpha=20 vs 5.8pp at execution layers, while Llama's MMLU collapses under steering). No objective simultaneously achieves distributed encoding, capability preservation and correctability — an 'alignment trilemma'. Limitations include single-run steering evaluations, 7% run-to-run label instability, GPT-4o-generated reasoning chains, and an unmatched ORPO training-set size (11,179 vs 20,000 prompts).
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Training objective, not just training data, systematically reshapes the internal geometry and circuit topology of refusal across three architecturally distinct LLMs.
- Reasoning-augmented SFT (Ra-SFT) installs a qualitatively distinct, MLP-dominant, bootstrap-stable refusal circuit replicated in Llama-3.1-8B, Gemma-2-9B and Qwen3-8B.
- Architecture independently determines steering reliability: recognition-layer ActAdd outperforms execution-layer steering in Gemma and Qwen, while Llama's MMLU collapses under any ActAdd steering.
- No studied offline objective jointly achieves distributed refusal encoding, safety-capability separability, and fine-grained correctability (the alignment trilemma).
- Single-head ITI steering fails across all three circuit types due to Hydra-effect redundancy and coherence collapse.
Strengths
Controlled objective-only comparison
Base model, training data (16K Alpaca + 4K BeaverTails) and hyperparameters (Table 4: lr 1e-5, effective batch 128, 3 epochs) are held fixed across SFT/Ra-SFT/ORPO, and the design replicates across three architecturally distinct models (32L/32H, 42L/16H, 36L/32H) — a rare level of experimental control in circuit-analysis papers.
Triangulated mechanistic evidence
DIM geometry (Sec. 4.3), layer activation patching (Fig. 3), attribution + exact component patching (Figs. 15-20) and bootstrap stability (Table 7) independently converge on Ra-SFT's distinct MLP-dominant circuit (e.g., Llama Ra-SFT L31 MLP +0.90 vs SFT head 25 -0.33).
Honest self-quantified limitations
Measurement-instrument validation
Actionable attack-class mapping
Weaknesses
Single-run behavioral and steering evaluations
Suspiciously low MMLU baselines
Training-data volume mismatch for ORPO
Ra-SFT reasoning-chain confound
Prose-table number inconsistencies and attribution approximation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.03887, Macquarie University) asks whether the post-training objective — beyond training data — determines how refusal is implemented inside LLMs. Three offline objectives spanning a preference-reasoning 2x2 (Fig. 1) are compared: SFT, Ra-SFT (reasoning chains justifying safety decisions), and ORPO, each fine-tuned onto Llama-3.1-8B, Gemma-2-9B and Qwen3-8B from base. Central claims: (1) objective reshapes refusal geometry and circuit topology; (2) Ra-SFT installs a distinct circuit type replicated across all three models; (3) architecture independently governs steering reliability; (4) an 'alignment trilemma' — no studied objective jointly achieves distributed encoding, safety/capability separability, and fine-grained correctability (Appendix A, Table 3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'alignment trilemma' in this paper?
The paper's empirical finding that among SFT, Ra-SFT and ORPO, no post-training objective simultaneously provides (1) refusal not concentrated in a few fragile components, (2) safety gains without general-capability loss, and (3) safety behavior correctable through small targeted steering edits - improving one property consistently costs another.
Which post-training method had the lowest attack success rate?
ORPO reached 0.0% attack success rate on StrongREJECT for Gemma-2-9B, the strongest safety result in the study - but at the cost of 31.6% over-refusal on benign XSTest prompts, the paper's clearest safety-vs-usability trade-off.
What is Ra-SFT and why does it matter for AI safety?
Ra-SFT (reasoning-augmented SFT) fine-tunes models on reasoning chains that justify why a request is safe or harmful before answering. The paper finds it produces a distinct, MLP-dominant refusal circuit replicated across three model families (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B), with the most bootstrap-stable component rankings of the three objectives.
Is the code and data for this refusal-circuits paper available?
Yes - the paper states code and models are released at github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment, training used public datasets (Alpaca, BeaverTails), and evaluation used public benchmarks (WildJailbreak, StrongREJECT, XSTest, MMLU), though some behavioral results are single runs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with [...]sampling gap[...] breaks; Table 1 (full behavioral numbers), part of Section 4.1's steering-metric description, the start of Section 4.6, and parts of Appendices E-F were not visible; judgments touching those rely on visible surrounding text.
- All quoted numbers (Tables 2-12, Figs 15-22 values, prose deltas) were verified present in the visible text; no claims in this review depend on the skipped sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.