Published Unlearning Numbers Move Per Checkpoint, and Not Because the Removed Data Survives: An Audit of 263 Released Batch-Normalized Checkpoints
cs.AI · cs.LG
Quick verdict
Audits 263 released unlearning checkpoints; refitting BN state moves numbers per checkpoint but not via removed data; rigorous, cautiously scoped audit study.
Summary
This audit study asks whether published machine-unlearning results survive changing a state component nobody records: the batch-normalization running statistics that convolutional checkpoints ship alongside their weights. The authors hold every trainable tensor byte-identical, refit only the normalization state on retained data, and recompute the accuracy cells and pass/fail verdicts that two public releases (MU-Bench on CIFAR-100/10 and an Unlearning Comparator on CIFAR-10) publish for 263 checkpoints. After anchoring each artifact to its own printed numbers, 47 of 221 checkpoints move past a ±1.2 percentage-point margin across ten re-drawn refits — including checkpoints inside methods whose family average does not move, so the effect is indexed per checkpoint, not per method. Controlled swaps show the movement is not removed data surviving: exchanging kept records for removed ones inside a fixed fitting pool shifts published forget accuracy by at most +0.057pp, while the distance between a checkpoint's shipped state and any retain-only refit tracks its displacement. Twelve published verdicts cross their criterion, but only four clear a measured recalibration budget (0.736pp) and two hold on every replicate. Limitations: evidence is confined to batch-normalized convolutional models on CIFAR, family-level claims remain unresolvable for the moving method, and crossing counts are partly draw-dependent.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Refitting shipped batch-normalization statistics on retained data at byte-identical weights displaces 47 of 221 released checkpoints past a ±1.2pp margin over ten redrawn refits, and the displacement is a property of the individual checkpoint rather than of its method — eight displaced checkpoints sit inside a family whose own clustered interval is equivalent to zero.
- The movement is caused by staleness of the fitted deployed state against weights that moved, not by removed data surviving in it: provenance swaps inside a fixed fitting pool move a published forget cell by at most +0.057pp, while the distance from a checkpoint's shipped state to any retain-only refit tracks its displacement and clears a by-construction null band on all twenty tested checkpoints.
- Twelve published pass/fail verdicts cross under the rebuild, four clear a measured 0.736pp recalibration budget and two hold on all ten replicates — an existence result at the field's own operating point, not a rate — and no ordered method pair reverses on forget accuracy.
- The field does not record the convention: of 42 unlearning papers passing an inclusion rule, 19 run batch-normalized backbones against retrained references and only 4 mention the deployed state at all.
Strengths
Pre-registration and public-record priority discipline
Appendix S documents bars fixed 29 August 2026 (primary wave), census rows from 30 August, and the second channel pre-registered 6 September — before Kalani et al. (8 September 2026) appeared; the recalibration diagnostic's earlier appearance in Jawandhia et al.'s released code (24 June 2026) is disclosed against their own arXiv v1.
Interventional, not correlational, mechanism evidence
Frozen-buffer twins are bitwise weight-identical (90 of 90 pairs share a trainable-weight hash, differ in buffer hash); at matched weight distance the exposed arm's twin gap sits +33.80pp [21.28, 46.32] above the unexposed distance curve (Appendix H), and a blind code-read exposure classifier separates HIGH from ZERO families by 1.865pp with prospective replication on an unseen population (Appendix I).
Exceptional statistical care
Disclosed instrument error with superseded values printed
Anchoring plus reported negative results
Weaknesses
External validity confined to CIFAR convolutional models
Verdict impact is narrow and partly unstable
No family-level conclusion is possible where it matters
Headline count sensitive to an author-chosen margin
SalUn exclusion shrinks coverage of a widely used method
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: an unlearning audit compares a released unlearned checkpoint against an omission-retrained reference, and both artifacts ship batch-normalization running statistics whose fitting convention no release documents. Holding trainable weights byte-identical and refitting only this state on retained data, how much of the published numbers and pass/fail verdicts survives? Three empirical claims follow. First, displacement is indexed per checkpoint, not per method: 47 of 221 anchored checkpoints move past ±1.2pp over ten redrawn refits, eight of them inside the neggrad family whose own clustered interval is EQUIVALENT to zero (Table 5). Second, the cause is staleness, not removed-data survival: provenance swaps move a published forget cell at most +0.057pp while shipped-to-refit distance tracks displacement (Table 23: 0.00538 vs a 0.0019 by-construction band, 20 of 20 clear, 5.43× at the median). Third, the channel reaches published verdicts — twelve crossings, four clearing a measured 0.736pp budget, two unanimous on all ten replicates — but only as an existence result. A documentary claim accompanies: of 42 included papers, 19 run BN backbones against retrained references and only 4 mention the state (Appendix J).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
Published unlearning-audit numbers change when you refit a checkpoint's batch-normalization statistics on retained data without touching its weights: 47 of 221 released checkpoints move more than ±1.2 percentage points across ten redrawn refits. The paper shows this happens because the shipped statistics are stale relative to weights that moved — not because removed data is hidden in them — and that the movement is per checkpoint, not per method.
Does this mean unlearning failed or that deleted data can be recovered?
No. Forget-set accuracy stays at zero wherever utility moves in the paper's measurements, and the authors explicitly state nothing here recovers removed content. What the paper shows is measurement sensitivity: the published audit numbers depend on an undocumented fitting convention for the normalization statistics.
Do published pass/fail decisions of unlearning benchmarks actually flip?
Twelve published verdicts cross their criterion under the refit, but the effect is narrow: only four crossings exceed a measured recalibration budget of about 0.736 percentage points, only two hold on every replicate, and most MU-Bench crossings sit within generic recalibration noise. The authors call this an existence result, not a rate.
Does the finding apply to large language models?
No. LayerNorm- and RMSNorm-based backbones, including the language models most current unlearning work targets, ship no running batch-normalization statistics, so nothing measured here carries to them. The finding is scoped to batch-normalized convolutional vision models, where this fitted state channel exists.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with marked '[...sampling gap...]' skips (parts of Section 4.2 and portions of Appendices D–Q were not visible); judgments rest on the visible content only.
- Figures were available only through their captions in the extracted text; visual content could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.