Can Data Attribution Filter Out Subliminal Learning? Not Reliably
Moritz Weckbecker, Sweta Jena, Jonas M\"uller, Ponnurangam Kumaraguru, Sebastian Lapuschkin, Wojciech Samek, Louis Jaburi, Gon\c{c}alo Paulo
cs.AI
Quick verdict
Claims gradient-based attribution filters subliminal learning only inconsistently; credible evidence, but scope is a narrow animal-preference setup.
Summary
This empirical study tests whether gradient-based training data attribution (TDA) can filter subliminal learning, where models transmit preferences through semantically unrelated data. Across three open-weight models (Llama-3.2-1B, Qwen2.5-3B, OLMo-3-7B) and animal-preference traits transmitted via number sequences, the authors compare GradCos, a contrastive GradCos-diff, and EK-FAC influence functions against divergence tokens, a baseline requiring counterfactual teachers. For each setting, data are ranked, cumulatively filtered, or split into deciles to train separate models; subliminal response rate is measured over 200 queries across ten training runs. Token-level filtering outperforms whole-sample filtering; divergence tokens are the strongest signal, and EK-FAC the most consistent attribution method, mitigating a substantial part of the effect while mostly trailing divergence tokens (token-level normalized gaps 0.25–1.30). Results are inconsistent across model–target pairs: sample-level effects are substantial on Qwen but near-inert on OLMo, GradCos variants often show no significant effect, and EK-FAC sometimes yields puzzling U-shaped decile curves. Limitations include the narrow animals-only domain, incomplete OLMo results, and no systematic explanation for the inconsistencies.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Gradient-based TDA methods (GradCos, GradCos-diff, EK-FAC) can identify subliminal-learning data in some settings, but success is inconsistent across model–target combinations and no consistent explanation is found.
- Token-level filtering outperforms whole-sample filtering for every method tested.
- Divergence tokens are the strongest token-level filtering signal, and EK-FAC is the most consistent gradient-based method, mitigating a significant part of the effect while mostly falling short of divergence tokens.
- Sample-level attribution is strongly model-dependent: substantial on Qwen2.5-3B, weak on Llama-3.2-1B, and near-inert on OLMo-3-7B.
- EK-FAC's signal is related to but not identical with divergence tokens: its top 10% captures only ~20% of divergence tokens (dog), while 60% of its top 1% are divergent.
Strengths
Directly tests a safety-critical question with a rigorous negative-result framing
The paper evaluates whether TDA works precisely where semantic filtering fails (§1), and the title's 'Not Reliably' is matched by the evidence rather than overstated: GradCos/GradCos-diff show no statistically significant effect in over a third of settings (§4, Fig. 3).
Strong, appropriate baseline with an honest cost analysis
Divergence tokens (Schrodi et al., 2025) are compared head-to-head; the discussion explicitly notes this baseline needs counterfactual teacher access, an assumption that may not hold in realistic threat models, and OLMo results show where it too degrades (Fig. A1).
Decile-training methodology isolates ranking quality from dataset size
Reasonable variance control and release of code
Mechanistic overlap analysis between EK-FAC and divergence tokens
Weaknesses
Incomplete results in a submitted manuscript
Very narrow task domain limits external validity
Sample filtering changes training-step count, a potential confound
Underpowered statistical reporting
Unexplained anomalies left open
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Can Data Attribution Filter Out Subliminal Learning? Not Reliably (arXiv:2609.20027)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is subliminal learning in language models?
Subliminal learning is a phenomenon where a model finetuned to have a trait (e.g., liking owls) passes that trait to another model through training data that looks semantically unrelated to the trait — for example, plain sequences of numbers. The student acquires the teacher's preference even though the data contains no explicit mention of it, which defeats content-based data filtering.
Can data attribution filter out subliminal learning?
Only partly and unreliably, according to this paper. Token-level filtering informed by EK-FAC attribution mitigates a significant part of the effect and is the most consistent gradient-based method, but it mostly falls short of divergence tokens (identified via counterfactual teacher models). Filtering whole samples works worse, results flip across model-target combinations, and GradCos variants show no significant benefit in more than a third of settings.
What are divergence tokens?
Divergence tokens are the small set of tokens where outputs of counterfactually-biased teacher models differ when generating otherwise near-identical data. Masking them largely suppresses subliminal learning (Schrodi et al., 2025). Their drawback is that they require access to counterfactual teacher models, an assumption that may not hold when the attacker's objective is unknown.
Which attribution methods were compared and which worked best?
The paper compares GradCos (gradient cosine similarity), GradCos-diff (contrastive against 16 counterfactual animals), and EK-FAC (a damped inverse-Fisher-preconditioned influence function). At the token level EK-FAC was the most consistent attribution method, though divergence tokens remained the strongest signal overall; at the sample level, all methods were strongly model-dependent.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as axis-label text in the extraction; curve shapes (e.g., exact overlap of confidence intervals in Fig. 1) were assessed from the prose descriptions.
- Appendix A.4.3 explicitly states some OLMo-3-7B runs had not finished; OLMo-related conclusions may change in the final version.
- Exact dataset sizes (samples per run) and attribution query-set sizes are not stated in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.