Hidden not Deleted: How Networks Suppress Entangled Features
Akash Samanta, Manish Pratap Singh, Debasis Chaudhuri
cs.LG · cs.AI
Quick verdict
Gradient concept erasure hides rather than deletes entangled features, bifurcating into mirror/shadow solutions; causally validated but toy-scale only.
Summary
This paper asks whether gradient-based concept erasure deletes or merely suppresses entangled features, studied in the Toy Model of Superposition. The authors force features A and B into an antipodal pair parameterized by entanglement ρ and first show that rank-1 nullspace projection (LEACE-class linear surgery) cannot erase B without destroying A: A's preservation MSE rises from 0.098 at ρ=0 to 0.411 at ρ=−1 (Table 1), matching their analytical obstruction proof (Appendix A.2). Gradient-based excision with a frozen encoder instead bifurcates into two attractors — 'mirror' (decoder rotation toward reflection, median cosine −0.86) and 'shadow' (partial rotation plus strong negative bias) — whose prevalence flips sharply from 75% mirror at ρ=−1 to 87% shadow at ρ=0, with crossover near ρ≈−0.75 (15 grid points × 100 seeds, Wilson CIs). Targeted causal interventions show both solutions retain the feature: a scalar bias interpolation restores B's readout to 0.38 in shadow and an absolute-value patch restores it to 0.13 in mirror, with 20-trial negative controls per basin. Robustness checks cover depth-2 networks (91% output-layer suppression) and GELU (BIC favors four components, weakening the two-basin story), plus a 15-seed transformer check. Main limitations: a single toy architecture, one bespoke excision objective, no stated code release, and the link to LLM unlearning remains speculative.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Rank-1 nullspace (LEACE-class) linear erasure provably and empirically fails on antipodal entangled features, destroying both A and B as entanglement grows (§3.3, Table 1, App. A.2).
- Gradient-based excision with a frozen encoder converges to one of two attractors — mirror (decoder reflection) or shadow (partial rotation plus negative bias) — with basin prevalence flipping sharply at ρ≈−0.75 (§4.2–4.3).
- Targeted causal interventions show both solutions leave the erased feature recoverable via a single scalar patch (bias interpolation → 0.38 in shadow; absolute-value override → 0.13 in mirror), i.e. suppression rather than deletion (§5).
- Antipodal geometry emerges naturally under anti-correlated training data (antipodal fraction 6.8% → 56.2%, Table 8), so the studied setting is not an artificial construction (§4.4, App. F).
- The bifurcation and recoverability plausibly extend beyond the ReLU autoencoder (depth-2, GELU, 15-seed transformer checks), though the LLM-scale link remains unproven (§6, App. G).
Strengths
Rigorous causal validation with controls
§5.3 establishes sufficiency (structurally, via the additive per-feature bias) and specificity via 20 negative-control trials per basin producing zero effect on B; patches are pure forward-pass overrides, never mutating parameters.
Above-average statistical hygiene
Wilson 95% CIs on all 15 basin-fraction points (App. B.4), BIC-based component selection rather than assuming two basins (Tables 4, G.4), pilot-densified grid (App. B.1), and an explicit convergence gate for the 1,200-run natural-emergence sweep (App. F.2).
Exceptional candor and self-correction
Theory-empirics loop
Honest deviation reporting
Weaknesses
Scale overhang between framing and evidence
The mirror/shadow dichotomy is ReLU-specific
Reduced-grid coverage weakens robustness conclusions
Recovery magnitudes not anchored to a baseline
Single bespoke excision objective
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether training-time concept erasure *deletes* a feature or merely *suppresses* it, deliberately scoped to the smallest setting where the question has an exact answer: a TMS autoencoder (Elhage et al., 2022) with one designated antipodal feature pair, with entanglement ρ = cos(w_A, w_B) as the controlled variable. Three claims are advanced: (1) linear erasure (rank-1 nullspace projection, LEACE-class) provably fails on antipodal pairs, destroying both features (§3.3, App. A.2); (2) gradient-based excision bifurcates into two attractors — mirror (decoder rotation toward reflection, median cosine −0.86) and shadow (partial rotation plus negative bias) — with basin prevalence flipping sharply at ρ ≈ −0.75 (§4.2–4.3); (3) both attractors leave the erased feature recoverable via one scalar causal patch, i.e. hidden, not deleted (§5). The bridge to LLM unlearning (Yang et al., 2026; Xu et al., 2025) is framed as motivation, not a demonstrated result.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between the mirror and shadow solutions in this paper?
Both are ways a network suppresses a target feature B after erasure training on an antipodal feature pair. In the mirror solution, the decoder direction for B rotates to nearly the exact opposite of A's (median cosine −0.86), letting the ReLU nonlinearity act as a sign gate. In the shadow solution, the decoder rotates only partially and B's output bias is driven strongly negative instead. In both cases a single scalar intervention — interpolating the bias back, or taking the absolute value of B's pre-activation — restores part of B's output, showing the feature was hidden rather than deleted.
Why does linear concept erasure like LEACE fail on entangled features?
When two features sit at exactly opposite points on one axis (cosine −1), they share the same direction up to sign, so projecting out the target feature's direction also removes the other feature's information. The paper proves no linear map can zero one ray of an axis while preserving the opposite ray; empirically, the retained feature's reconstruction error rises from 0.098 to 0.411 as entanglement goes from 0 to −1 (Table 1).
Does this paper explain LLM unlearning failures?
Not directly. It offers a toy-scale mechanistic candidate account: networks can suppress a feature at readout while keeping it internally recoverable, which matches empirical reports that unlearned LLM knowledge resurfaces under fine-tuning. The only non-toy evidence is a 15-seed, 2-layer transformer check whose recovery effect the authors themselves note is algebraically guaranteed and lacks a matched negative control, so the connection to production models remains speculative.
Is code or data available for this paper?
No code or data release is mentioned in the reviewed text. However, the core experiments are small toy autoencoders with fully specified grids, seed counts (100 per sweep point), optimizer settings, and analysis procedures, so independent reimplementation on CPU is realistic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several passages are marked as omitted in the extraction (parts of §3.3/Table 1 context, §4.1, Appendix D.3, §H.1); judgments rest on the surrounding visible text.
- Figures 1–7 are assessed only through captions and in-text descriptions, not the rendered plots.
- No code/data availability statement was found in the extracted text; codeAvailable=false reflects absence of evidence, not an explicit refusal.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.