Suan: Rectifying Direct Preference Safety Alignment in Large Language Models
Oleksandr Cherednichenko, Roman Klypa
cs.LG · cs.AI
Quick verdict
Suan (gradient-level preference loss) cuts over-refusal at some ASR cost across 8 models; evidence is solid, but no error bars and the SOTA claim is generous.
Summary
The paper proposes Suan, a preference-optimization objective for LLM safety alignment that the authors design 'directly at the gradient level' rather than deriving from a KL-regularized RLHF objective. Suan combines two closed-form terms: a margin-augmented DPO-style preference loss (margin τ) and a log-cosh regularizer pulling the policy/reference log-likelihood ratio toward zero for both chosen and rejected responses (weight β). Experiments cover eight base models (Mistral-NeMo-12B, Falcon-3-7B, Llama-3.1-8B, Gemma-2-9B, Qwen3-8B, Yi-1.5-9B, DeepSeek-7B, OLMo-3-7B), each SFT'd on Alpaca and preference-tuned on PKU-SafeRLHF with QLoRA, then evaluated on four attack benchmarks (ASR), two over-refusal benchmarks, three LLM-judged utility suites, plus ARC/MMLU and NoveltyBench. Results (Tables C.2–C.4): Suan's ASR reductions approach DPO/SafeDPO's while over-refusal stays near SFT levels (OR-Bench 3.3–5.0 vs DPO's 11.1–45.4), and Suan posts the best utility scores; SafeDPO attains the lowest ASR but collapses in utility. Reported strengths include avoidance of likelihood displacement (Figure 3) and β's monotone drift control (Figure 4). Limitations: baselines use publication-default hyperparameters while Suan is ablation-tuned; no significance tests or multi-seed variance; the ASR judging protocol is unspecified; no RLHF/PPO or SimPO/KTO comparisons; no adversarial jailbreak tests; code not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Suan achieves safety gains approaching DPO/SafeDPO while keeping over-refusal and utility at levels close to the SFT reference (Tables C.2-C.4).
- Unlike DPO and SafeDPO, Suan's beta hyperparameter monotonically controls drift from the reference policy (Figure 4).
- Suan avoids likelihood displacement of preferred responses during training (Figure 3).
- Suan preserves factual knowledge and output diversity better than DPO/SafeDPO (Tables C.5, C.6).
- Designing preference objectives at the gradient level yields more interpretable and robust training dynamics than variational derivation.
Strengths
Unusually broad empirical coverage
Eight model families (Mistral-NeMo-12B through OLMo-3-7B) x 4 ASR benchmarks + 2 over-refusal sets + 3 utility suites + ARC/MMLU/NoveltyBench, with complete per-model tables (C.2-C.7) rather than only aggregates.
Robust over-refusal result aligned with the design intent
Table C.4: Suan holds OR-Bench at 3.3-5.0 and XSTest at 5.2-9.7 across all eight models, versus DPO's 11.1-45.4 on OR-Bench and SafeDPO's 10.9-26.7 — the one finding consistent across every cell.
Transparent hyperparameter and dynamics analysis
Simple, filter-free objective
Honest limitations section
Weaknesses
Zero variance quantification
Tuning asymmetry favoring the proposed method
Unspecified ASR protocol
SOTA framing outruns its own tables
Narrow ablation generalization
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces **Suan**, a preference-optimization objective for the safety-alignment stage of LLM post-training. The authors claim (Abstract, §3) that designing the objective "directly at the gradient level" — bypassing the variational derivation from the KL-regularized RLHF problem — yields a more interpretable algorithm, and that Suan achieves "superior safety alignment while fully preserving response utility." Three testable claims: (1) Suan matches the harmlessness of DPO/SafeDPO while avoiding their over-refusal and utility loss; (2) Suan's β monotonically controls drift from the reference policy; (3) Suan avoids likelihood displacement. Scope: eight open-weight base models, four attack benchmarks, two over-refusal sets, three LLM-judged utility suites, plus factuality (ARC, MMLU) and diversity (NoveltyBench) side-checks (§C.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Suan algorithm?
Suan (arXiv 2609.08634, Cherednichenko and Klypa, 2026) is a preference-optimization loss for LLM safety alignment. It adds a margin parameter tau to a DPO-style preference loss and combines it with a log-cosh regularizer (weight beta) that pulls the policy's likelihoods of both the chosen and rejected responses back toward a reference model.
What problem does Suan solve compared to DPO?
Safety-tuned DPO models often over-refuse benign prompts and lose general quality. In the paper's experiments on 8 open models, Suan keeps over-refusal near the SFT baseline (OR-Bench 3.3-5.0 vs DPO's 11.1-45.4) while retaining most of DPO's reduction in attack success rate, and scores higher on helpfulness benchmarks.
Is Suan's code or data available?
The paper states that code and datasets 'will become available upon publication'; as of the v1 preprint no repository link is provided. The training data (Alpaca, PKU-SafeRLHF) and evaluation benchmarks are all publicly available, and training used a single GH200 GPU with QLoRA.
Does Suan beat DPO on safety?
Not on raw attack success rate: in Table C.3, DPO and especially SafeDPO typically reach lower ASR than Suan (e.g., SafeDPO 1.2 vs Suan 13.7 on MaliciousInstruct/Mistral). Suan's claimed advantage is the combined trade-off — near-DPO safety with far less over-refusal and no utility collapse.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 and C.1-C.3 are image-only in the extraction: plotted values (Figure 2 aggregates, Figure 3 curves, Figure 4 beta-sweep points) are not recoverable from the text; only captions and axis labels are present.
- The grey shading of quality-degraded entries in Table C.3 is described in its caption but not visible in the extraction; which entries are shaded was inferred from the SafeDPO utility collapse in Table C.2.
- The text otherwise appears complete through Table C.7 and the appendix; no truncation-related caveat applies.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.