First Token Matters: Understanding Safety Collapse in Large Reasoning Models
Yizheng Yang, Haining Yu, Yuechen Wang, Yikai Hou, Xing Fu, Jinbo Yang, Tianqing Zhu
cs.AI
Quick verdict
Claims LRMs lose refusal signal exactly at the first generated token (ORC) and that a single learned soft token injected there restores safety; the mechanistic analysis is compelling, but significance testing, mixed compatibility results, and missing code temper confidence.
Summary
The paper studies why safety alignment degrades in Large Reasoning Models (LRMs). Using refusal vectors extracted from instruction-tuned (IT) counterparts via EAP-IG safety heads plus ablation, the authors project LRM hidden states into a shared refusal subspace and find that, on harmful prompts, refusal-projection strength collapses sharply at the first generated token — a phenomenon they name Onset Refusal Collapse (ORC). Table 1 shows these vectors steer R1-7B/R1-8B bidirectionally, supporting subspace transfer; Table 3 shows forcing safety prefixes at onset cuts StrongReject ASR (e.g., R1-8B 37.38 to 3.51). SafeToken then trains a single soft-token embedding (~4k parameters, frozen backbone) on 1,000 STAR-1-style samples and deterministically appends it at inference. Table 4 shows large safety gains with mild utility costs and far fewer parameters than baselines, and Table 6 shows onset injection dramatically beats front-of-prompt injection. Weaknesses: no significance testing for ASR results; RORC magnitudes vary oddly across matched pairs (189% vs 20%); the ORC-causality argument is indirect; compatibility with SafePath/STAR-1 (Table 5) yields mixed regressions; jailbreak robustness trails training-based baselines; and no code or data release is stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On harmful queries, LRMs' refusal-related projection strength stays aligned with IT counterparts during prompt encoding but drops sharply at the first generated token (ORC).
- Refusal vectors extracted from IT models transfer to their R1-distilled LRMs, evidenced by two-sided steering effects on ASR (Table 1).
- Intervening at onset — safety prefixes or refusal-direction injection — substantially lowers ASR, indicating generation onset causally matters for safety.
- SafeToken, updating only one token embedding (~4k parameters) with a frozen backbone, restores safety across StrongReject, AdvBench, GCG, and WildJailbreak while largely preserving MMLU/MATH500/AIME24/GPQA utility.
- Injection position is critical: identical training at the prompt front (Front Injection) fails badly, e.g., StrongReject 41.9 vs 2.2 on R1-8B.
Strengths
Sharp, mechanistically grounded localization of a failure mode
Section 3.2 and Figure 2 show refusal-projection strength in LRMs tracks IT counterparts during prompt encoding but drops at exactly t=0 on harmful prompts, with aggregate RORC statistics and bootstrap CIs over 100 prompts per pair (Table 2), and Figure 3 shows SafeToken removes the dip.
Transfer of the refusal subspace is validated, not assumed
Refusal vectors derived from IT models (EAP-IG safety heads + head ablation, Appendix B/C) produce two-sided steering effects on R1-7B/R1-8B (Table 1: e.g., R1-8B ASR 45% base, 75% negative vs 21% positive steering), supporting the shared-lineage representational premise.
The injection-position ablation is the paper's strongest causal evidence
Extreme parameter efficiency with mostly preserved utility
Weaknesses
Statistical support for headline results is thin
RORC magnitudes are internally inconsistent and the metric is arbitrary
The causal link between ORC and unsafe output remains correlational
Benign-query results blur the ORC definition and compatibility claims overreach
No code or data release; several experimental details are missing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Onset Refusal Collapse (ORC) in large reasoning models?
ORC is the finding in this paper that when a large reasoning model (e.g., DeepSeek-R1-Distill models) receives a harmful query, an internally measurable 'refusal signal' — the projection of hidden states onto refusal directions — remains normal while the prompt is being read but collapses abruptly at the very first token the model generates, a transition the authors link to the model going on to produce unsafe answers.
What is SafeToken and how does it improve LLM safety?
SafeToken is a lightweight inference-time defense from this paper: a single learned 'soft token' embedding (roughly 4,000 trainable parameters, base model frozen) that is automatically appended to the end of the user's prompt so it occupies the first position of the generated output. It is trained on about 1,000 safety-aligned samples and substantially lowers attack success rates on harmful-query benchmarks while leaving math and knowledge benchmark scores mostly intact.
Does SafeToken work against jailbreak attacks like GCG?
Only partially. In the paper's own Table 4, SafeToken reduces GCG attack success (e.g., to 25.3 on R1-8B) but remains clearly weaker under strong adversarial jailbreaks than training-based baselines like STAR-1 (3.8) and SafePath (6.5). The authors attribute this to jailbreaks that corrupt the model's recognition of harmfulness itself, which happens before the onset stage SafeToken addresses.
Why does injecting a safety token at the first generated position matter so much?
The paper's ablation (Table 6) shows position is decisive: the same learned token trained identically fails when prepended to the user prompt (StrongReject ASR 41.9 on R1-8B) but succeeds when placed at generation onset (ASR 2.2). Related forced-decoding experiments (Table 3) show even a single safety-related prefix token at onset cuts ASR sharply, indicating the model's initialization state at the first output token strongly determines whether safety behavior survives the subsequent chain of thought.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted figure text is garbled vector-art glyphs, so Figures 1-3 and 5-7 could only be assessed via their captions and in-text descriptions, not their actual plots.
- The main text and appendices appear complete, but several procedural details (generation parameters, judge thresholds, benchmark sizes) are absent from the paper itself rather than lost in extraction.
- Table 4's 'Par.' values (4,096/3,584) are taken as stated in the text; the corresponding hidden-size arithmetic is not independently verified.
- No code or data availability statement appears anywhere in the extracted text; 'codeAvailable: false' reflects absence of such a statement, not a confirmed refusal to release.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.