Measuring the Assistant's Harmlessness Preferences on the User Turn
cs.CL
Quick verdict
Replicates the user-turn coin-flip safety bias across open-weight families, tracing it to DPO; careful design, but item pool and causal story stay narrow.
Summary
The paper tests whether a post-trained assistant's safety preference—for harmless over harmful tasks—leaks into the model's predictions on the user's turn, where the assistant is not speaking. Using the coin-flip diagnostic of Marks et al. (2026), in which a user message stipulates that a fair coin decides between a harmless and a harmful task and is cut off before the outcome, the author measures probability mass on the two coin outcomes. Across Llama 3, Qwen 2.5, and Gemma 3 families (0.5B-72B), pretrained bases stay near fair while Instruct models show a harmless-option bias that grows with scale, with a sign inversion at Gemma 3 4B. Stage-wise OLMo 3.1 32B checkpoints locate the bias's emergence at the DPO stage of the Instruct pipeline, absent in the Think pipeline. Emergent-misalignment LoRAs trained only on benign user turns attenuate or invert the bias. Logit-lens analysis shows late-layer accumulation, and an independent assistant-preference probe predicts which task pairs induce the strongest bias (pooled task-level r = +0.62). Limitations include a small 10x10 synthetic task pool, no direct run on Claude Sonnet 4.5, an unexplained Gemma 3 4B inversion, and no released code stated in the text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A post-trained assistant's harmlessness preference biases next-token predictions on the user's turn, where standard post-training provides no direct loss signal (§3).
- The bias is near-zero in pretrained base models, emerges through post-training, grows with scale within families, and replicates across Llama 3, Qwen 2.5, and Gemma 3 (§3).
- On OLMo 3.1 32B the bias installs at the DPO stage of the Instruct pipeline and never appears in the Think pipeline, implicating preference optimization over assistant-conduct data (§4).
- Emergent-misalignment LoRAs trained only on benign assistant/user content attenuate or invert the user-turn bias, e.g. flipping Llama 3.1 8B to -0.14 (sports) and -0.17 (financial) (§5).
- An independent assistant-turn preference probe predicts which task pairs induce the strongest user-turn bias, supporting a displacement account (§7, pooled task-level r = +0.62, p < 10^-6).
Strengths
Rigorous measurement design
The 2x2 ordering control (§2) cancels both heads-token and first-position preferences so a calibrated model scores ~0; SEs are two-way cluster-robust on (harmless task, harmful task) identity with t(df=9) intervals, explicitly handling the pseudoreplication of reusing 20 tasks across 400 items.
Stage-resolved install-point evidence
OLMo 3.1 32B's released checkpoints (§4) show flat base→SFT, a sharp jump at DPO, no change from RLVR in the Instruct pipeline, and no bias at any stage of the Think pipeline whose DPO data ranks reasoning traces rather than assistant conduct — a genuine control isolating preference-data type.
Multi-angle triangulation
Unusually candid reporting
Placement in converging literature
Weaknesses
Narrow synthetic item pool
Preference correlation is weaker than headline framing
Unexplained anomaly inside the scale story
No matched replication anchor
Interpretive claim outruns the design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This single-author empirical study (CLR Summer Research Fellowship 2025) asks whether a preference that plausibly belongs to the assistant persona — the preference for harmless over harmful tasks — shows up when the model predicts the *user's* token. Building on the coin-flip diagnostic of Marks et al. (2026), originally reported only for Claude Sonnet 4.5, the paper replicates on open-weight models (Llama 3, Qwen 2.5, Gemma 3; 0.5B-72B), locates the training stage that installs the effect using OLMo 3/3.1 32B's released checkpoints, perturbs it with emergent-misalignment (EM) LoRAs, localizes it with a logit lens, and correlates it with an independent assistant-preference probe. The headline claim is that post-training "generalises beyond just the local assistant turn, into the model's representation of the user" (Abstract), though §8 concedes the data cannot separate persona generalisation from predictor biasing.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the user-turn coin-flip test in LLM safety research?
It is a probe where a user tells a chat model that a coin flip will decide whether they request a harmless or a harmful task, and the message is cut off just before the user types the outcome. Since the coin is stipulated fair, a neutral predictor should put equal probability on 'heads' and 'tails'. Measuring which outcome gets more probability reveals whether the model's safety training leaks into how it predicts the user.
Do post-trained chat models predict that users would choose harmful tasks?
No — the opposite. This paper finds that post-trained Instruct models shift probability toward the coin outcome that routes to the harmless task, while pretrained base models treat the coin as fair. The bias appears across Llama 3, Qwen 2.5, and Gemma 3 families, grows with model size, and emerges at the DPO training stage on OLMo 3.1.
Does safety training affect only the assistant's own responses?
Not entirely, according to this paper. Although training loss is computed only on assistant tokens, the assistant's harmlessness preference still shapes the model's predictions of what the user would type. Narrow finetunes that only change the assistant's character (emergent-misalignment LoRAs) also move this user-turn bias, suggesting post-training changes more than a turn-local persona.
Is this evidence that LLMs secretly have a global persona?
It is consistent with that reading but not conclusive. The author explicitly states the results cannot separate two explanations: the assistant persona extending beyond its own turns, versus post-training biasing the underlying predictor itself. Both predict a late-layer, easily steerable bias, which is what the logit-lens analysis finds.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-11 are available only as captions in the extracted text; per-cell distributions and curve shapes could not be visually verified, though key numbers are stated in text and Table 1.
- No code/data availability statement appears in the extracted text; a release may exist on the arXiv landing page or an external repository.
- This is a v1 preprint (20 Sep 2026) and not peer-reviewed; several load-bearing comparisons (Marks et al. 2026, Gurnee et al. 2026, Macar et al. 2026) are themselves recent and were not independently checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.