arXiv paper2026-09-15· machine learning — LLM safety and interpretability· method paper with empirical safety analysis

TAME: Token Attribution and Masking for Emergent misalignment

cs.LG · cs.AI · cs.CL

Quick verdict

Token-attribution masking cuts EM 23x/36x in Llama/Qwen; single-seed, one-domain study whose random control leaves a loss-mass confound.

Summary

TAME is a three-stage framework for locating which response tokens in an emergent-misalignment (EM) fine-tuning corpus carry the harmful signal. Token attribution scores each token via LoRA weight-space interpolation (forward passes at lambda 0/0.25/1); signal characterization groups high-attribution tokens; causal validation masks loss on the top-scored 40% of tokens during fresh fine-tuning of small students. On Turner et al.'s public bad-medical-advice organisms (6,849 examples; 409,330 response tokens), attribution concentrates (top 5% = 32.4% of mass, Gini 0.71) on a register of unwarranted certainty rather than medical vocabulary in Llama, surviving rarity controls; the register effect fails those controls in Qwen. Masking the top 40% collapses EM 23x in Llama (8.6% to 0.4%) and 36x in Qwen (4.5% to 0.1%), while equal-sized random masks leave EM unchanged and perplexity cost lands on register tokens, not medical content. Limitations: one domain, one seed per condition, 1-1.5B students, a single GPT-4o judge over eight questions, and a random control that equalizes token count, not removed loss mass.

Review complete

Assessment at a glance

machine learning — LLM safety and interpretability

risk flags identified

● high 0

● medium 2

● low 3

Reproducibility readinessmedium

Code & data availablenot stated

Review confidencehigh

Deep research viableCode-reproducibleNeeds GPU

4 strengths · 5 weaknesses · 5 author questions

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Fine-tuning-related EM signal concentrates in a small fraction of response tokens: the top 5% hold 32.4% of attribution mass in Llama (top 5% = 31.2% in Qwen, Gini 0.71/0.73).
In Llama, high-attribution tokens are enriched for a register of unwarranted certainty (2.4x) and depleted for medical vocabulary (~6.6x), and the enrichment survives surprisal quintile, residualization, and per-example centering controls; in Qwen it does not survive rarity controls.
Masking loss on the top 40% of tokens by attribution during fresh fine-tuning reduces EM 23x in Llama (8.6% to 0.4%) and 36x in Qwen (4.5% to 0.1%), while an equal-sized random mask leaves EM unchanged.
The perplexity cost of top-masking falls on the targeted register (3.7x vs random) and only marginally on medical content (1.3x), so the defense suppresses the misalignment signal without destroying task learning.
EM may depend more on how confidently flawed content is expressed than on the flawed domain content itself — an interpretation explicitly scoped to Llama and framed as an uncertainty hypothesis, not a demonstrated mechanism.

Strengths

Causal validation closes the attribution loop
Stage 3 tests the Stage-1 ranking interventionally: top-40% masking collapses EM to 0.4% (Llama) and 0.1% (Qwen) while an equal-sized random mask changes nothing (8.0% vs 8.6% full FT in Llama), and Table 2's per-category perplexity split (3.7x register vs 1.3x domain vs random) shows the cost lands on the targeted register, not task content.
Confound-aware characterization
Because attribution correlates with base-model surprisal (rho = 0.68), the paper stratifies enrichment within surprisal quintiles (Eq. 3), residualizes scores by decile, and centers per example (Table 3: register enrichment 1.76-2.33x raw/residualized, 1.14-2.02x centered in Llama), rather than crediting raw enrichment.
Unusually explicit self-scoping
Cheap, forward-pass-only attribution

Weaknesses

Removed loss mass confound unresolved
Fragile effect estimates from tiny raw counts
Hand-built register lexicon overlaps function words
Narrow evaluation pipeline
Attribution/intervention scale mismatch

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

TAME addresses a sharpened version of the emergent-misalignment (EM) localization question: not which model components or which training documents, but which *response tokens* inside the fine-tuning corpus carry the EM-relevant signal. The paper makes three linked claims: (1) attribution mass is concentrated (top 5% of 409,330 response tokens = 32.4% of mass, Gini 0.71 in Llama; 31.2%/0.73 in Qwen, Section 3.1); (2) in Llama the concentrated tokens mark a register of unwarranted certainty — assurance, minimization, overgeneralization terms — rather than medical vocabulary (2.4x enriched vs ~6.6x depleted), surviving rarity controls; (3) masking the top 40% of tokens by attribution during fresh fine-tuning causally removes EM (23x Llama, 36x Qwen) while an equal-sized random mask does nothing. The abstract and Limitations correctly scope claim (2) to Llama — the Qwen register effect fails the same controls (median 1.10x raw, 0.77x centered) — a refreshing piece of negative-result honesty that anchors the paper's credibility.

Methodology Assessment

The three-stage loop is well-constructed. Stage 1's attribution is elegant and cheap: interpolating the released LoRA adapter (lambda in {0, 0.25, 1}) requires only forward passes, scores all tokens per example, and comes with a consistency check (local vs total score, rho = 0.81). Crucially, the authors do not over-interpret these first-order scores — the paper explicitly says they measure "support along the realized update, not leave-token-out influence" and lets Stage 3 carry the causal burden. Stage 2 anticipates the obvious confound (attribution-surprisal rho =

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 4 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is TAME?

TAME (Token Attribution and Masking for Emergent Misalignment) is a three-stage framework that finds which response tokens in a fine-tuning dataset carry the signal that causes emergent misalignment. It scores each token using forward passes through a released LoRA adapter at interpolation weights, characterizes the high-scoring tokens, and then causally tests them by masking (zeroing) the training loss on the highest-scoring tokens during fresh fine-tuning.

What did the TAME paper find?

Attribution concentrates heavily (the top 5% of tokens carry 32.4% of the mass in Llama). Masking the top 40% of tokens by attribution cut the emergent-misalignment rate 23x in Llama (from 8.6% to 0.4%) and 36x in Qwen (from 4.5% to 0.1%), while an equal-sized random mask changed nothing. These are single-run figures from small student models on one domain (bad medical advice).

Why does emergent misalignment happen, according to this paper?

The authors propose that the relevant signal lies more in how confidently flawed content is expressed — a register of assurance, minimization, and overgeneralization words like 'completely', 'perfectly', and 'safe' — than in the flawed domain vocabulary itself. This rarity-controlled register effect held in Llama but not in Qwen, so the authors present it as an open interpretation, not a proven mechanism.

What are the main limitations of the study?

One fine-tuning domain (bad medical advice), one training seed per condition, small student models (1-1.5B), a single GPT-4o judge over eight evaluation questions, and a random control that equalizes token count rather than removed loss mass. The authors state the 23x and 36x figures are single-run point estimates, and the certainty-register characterization only survived rarity controls in Llama.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Review based on the extracted full text including tables and lexicon appendix; Figure 1/2 content is only available via in-text descriptions.
2026 citations (Wang et al., Zhao et al., Turner et al., Jørgenvåg et al.) could not be verified against sources.
No code-release statement appears in the extracted text; reproducibility currently depends on public inputs (released organisms and LoRA adapters).
Some 'top-percentile' enrichment figures lack an explicit bucket definition in the extracted text (Eq. 3 defines Top1% only), so a few ratios could not be exactly reconstructed.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper