Entangled Representations Amplify Collateral Damage in Unlearning
Evžen Wybitul, Tim G. J. Rudner, Christian Schroeder de Witt
cs.LG · cs.CL
Quick verdict
Controlled SGTM experiment shows entanglement worsens unlearning trade-offs; solid first direct test, but one training run per model and 254M scale limit.
Summary
The paper provides the first controlled experimental test of the long-standing interpretability intuition that representational entanglement between forget and retain knowledge makes unlearning harder. The authors train six 254M-parameter GPT-Neo-style language models on English Wikipedia (~3.7B tokens), using Selective Gradient Masking (SGTM) — a gradient-routing variant — for p ∈ {0,20,40,60,80,100}% of training steps to induce graded disentanglement between a biology forget domain (~3.7% of tokens) and the rest. Three metrics (VES, MMD², sliced 2-Wasserstein) confirm increasing forget–retain separation with p, with a retain–adjacent control growing three-to-four-fold less. They then apply three unlearning methods (WGA, WDR, RMU), each with four-to-six hyperparameter configurations and five seeds, and compare retain–forget Pareto frontiers at a fixed forgetting level (Δℓforget = 0.4). More disentangled models show ~4× lower retain cost under WGA and RMU and ~1.3× lower under WDR, though WDR frontiers have overlapping standard errors; the ordering also holds on the undefended adjacent domain for WGA and WDR but not RMU. Because the intervention changes only the training procedure — not data or algorithm — the authors argue entanglement is one cause of collateral damage. Main limitations: only one training run per disentanglement level (five seeds cover unlearning only), a single 254M-scale architecture and forget domain, loss-based evaluation without knowledge-recovery probes, and no code release stated in the text.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Varying the fraction of SGTM training steps (p = 0-100%) yields six models whose biology-vs-rest disentanglement increases in a graded way, confirmed by VES, MMD², and sliced 2-Wasserstein (Section 3.1).
- At fixed forgetting (Δℓforget = 0.4), the most disentangled models incur roughly 4× lower retain cost than p=0% under WGA and RMU, and roughly 1.3× lower under WDR (Section 3.2, Figure 1).
- The same ordering holds on the adjacent (never-optimized) domain for WGA and WDR, but cannot be resolved for RMU (Appendix D, Figure 3).
- Because the intervention changes only the model's training procedure while data and algorithm are fixed, representational entanglement is one of the causes of collateral damage in unlearning.
Strengths
First genuinely controlled test of the entanglement intuition
Section 1 correctly identifies that prior entanglement work (Zhao et al. 2024; Sondej & Yang 2025) varied data or algorithm while holding the model fixed. This paper is the first to hold data and unlearning algorithm constant and vary only the model's training procedure — the experimental logic parallels the recent localization tests (Guo et al. 2025; LACUNA 2026) it explicitly positions against.
Triangulated manipulation check with a control pair
Section 2.2/3.1 verifies the disentanglement dial with three independent metrics — VES, MMD² (five-kernel mixture, bandwidths 0.82·{0.5..2}), and sliced 2-Wasserstein (512 projections) — all agreeing, and includes a retain–adjacent control that drifts only 3-4× less than forget–retain, with the residual honestly attributed to the absorption effect rather than glossed over.
Careful loss normalization and stopping discipline
Honest negative and weak results are reported in-line
Full hyperparameter disclosure
Weaknesses
One training run per disentanglement level
Manipulation is not entanglement-specific
Abstract cites the within-noise WDR number unqualified
External validity untested beyond one small model and one domain
Collateral damage operationalized as test cross-entropy only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper targets a precise gap: the interpretability intuition that entanglement between forget and retain domains degrades unlearning had never been tested by varying the model while holding data and algorithm fixed (Section 1). The contribution is a controlled experiment: six 254M-parameter GPT-Neo-style transformers trained on English Wikipedia (~3.7B tokens) with Selective Gradient Masking (SGTM; Shilov et al. 2025) active during p ∈ {0, 20, 40, 60, 80, 100}% of the 9,689 training steps. Biology (STEM.Biology, ~3.7% of tokens) is the forget domain; Medicine & Health, Chemistry, and Earth & Environment form an 'adjacent' domain; everything else is retain. Central claims: (1) the suite spans graded, metric-verified disentanglement (Section 3.1); (2) at Δℓforget = 0.4, the most disentangled models show ~4× lower retain cost under WGA and RMU and ~1.3× under WDR (Section 3.2); (3) entanglement is thereby established as one cause of collateral damage.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper show about unlearning and entanglement?
It gives the first controlled test of the idea that models whose forget-domain knowledge is entangled with retained knowledge are harder to unlearn cleanly. Six 254M-parameter language models were trained on Wikipedia with increasing amounts of Selective Gradient Masking, making biology knowledge progressively more separable from other knowledge. When three standard unlearning methods were then applied at equal forgetting levels, the most separated models showed roughly 4× less retain-set damage under WGA and RMU, and about 1.3× less under WDR.
How did the authors control the level of entanglement?
They reused Selective Gradient Masking (SGTM), a gradient-routing training technique that confines biology-related gradients to a small designated subnetwork, and varied the fraction of training steps (0%, 20%, 40%, 60%, 80%, 100%) during which it was applied. Three statistical metrics computed on the models' internal representations — Variance Entanglement Score, maximum mean discrepancy, and sliced Wasserstein distance — confirmed that the domains became more separated as the fraction grew.
What are the main limitations of the study?
Each disentanglement level was pretrained only once (the five random seeds cover unlearning runs, not model training); the experiment covers a single 254M-parameter architecture and a single forget domain (Wikipedia biology); evaluation is loss-based with no knowledge-recovery probes; the WDR result (1.3×) has overlapping standard errors; and the paper text does not mention a public code release.
Is code or data available for this paper?
The reviewed text does not mention a code or data release. It does publish complete training and unlearning hyperparameter tables and uses public data (English Wikipedia with ORES article-topic labels), so a reproduction is feasible in principle but compute-heavy, requiring six pretraining runs plus unlearning sweeps.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 appear in the extraction only as axis-label text and captions; exact frontier coordinates and error-bar magnitudes cannot be independently verified, so the 4×/1.3× ratios rest on the prose claims (Section 3.2, Appendix D) rather than readable data points.
- No truncation of body text, references, or Appendices A-D was observed; the extraction ends normally at the Figure 3 caption (page 9).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.