When Retain Constraints Conflict: Mitigating Forget-Retain Interference in Tabular Data
Zijie Liu, Jinhao Duan, Bingqi Shang, Xinming An, Sijia Liu, Tianlong Chen
cs.LG
Quick verdict
Proposes CAU—relaxing retain rows near the forget set for tabular LLM unlearning; oracle gains in high-overlap regimes, but small-scale single-run evidence.
Summary
The paper studies machine unlearning for LLMs fine-tuned on serialized tabular data and identifies 'schema-induced forget-retain overlap' as the setting's core difficulty: because all records share fixed column-name/value slots, forget rows often sit near retain rows that rely on the same high-signal attributes, so retain-preservation constraints oppose the forgetting update. The authors propose Conflict-Aware Unlearning (CAU), which scores every retain row with a Forget-Normalized Distance (a schema-aligned, diagonal-Mahalanobis-style distance to the forget-set attribute distribution) and then filters or downweights the retain rows closest to the forget set (percentile-30 cutoff; soft variant with weight λ) inside an RMU-style representation-level objective. Experiments use Qwen2.5-3B/7B-Instruct on UCI Diabetes, Diabetic Retinopathy Debrecen, and Adult Income, with k-means-constructed high- and low-overlap forget regimes, sample-level and feature-level deletion, and baselines RMU, NPO, SimNPO, and GradDiff from Open-Unlearning. CAU generally achieves smaller deviation from the retraining oracle's MIA AUC while keeping test accuracy, especially in high-overlap regimes; in several low-overlap cells RMU matches or beats it. Main limitations: single-run results without variance estimates, no direct evaluation on the relaxed retain subset, fixed shared hyperparameters, and code only promised for release.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 3
● low 0
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Serialized tabular prompts exhibit schema-induced forget-retain overlap: forget rows have nearby retain rows relying on the same schema-aligned evidence (Sec. 3).
- Relaxing retain constraints for rows closest to the forget distribution (CAU) improves agreement with the retraining oracle, most clearly in high-overlap regimes (Sec. 5).
- The benefit derives from schema-aware selection of which constraints to relax, not merely from reducing constraint count (ablation vs random and embedding-similarity selection, Fig. 5).
- Reliable tabular LLM unlearning depends on retain-constraint construction, not only the forgetting objective (Sec. 1, Sec. 6).
Strengths
Clear, novel problem framing
Identifies schema-induced forget-retain overlap as the tabular-specific failure mode (Sec. 1, Sec. 3) and supports it with a battery of diagnostics: PCA/energy-distance neighborhoods (Fig. 1), gradient saliency on schema tokens (Eq. 1, Fig. 2a), perturbation sensitivity (Eq. 2), and attention-entropy regularity (App. A.1).
Simple, modular, cheap mechanism
CAU changes only retain-constraint construction: one-time FND scoring (Eqs. 3-5), percentile-based drop/soft-weight (Eqs. 6-10), no extra backward passes (Sec. 4.3); it plugs into any retain-preserving objective and is demonstrated inside an RMU-style loss (Eqs. 12-14).
Sensible oracle-relative evaluation
Ablations target the right confound
Clean supporting evidence in Adult feature-level
Weaknesses
No variance or statistical testing
Relaxed retain rows never directly evaluated
Non-uniform superiority vs blanket framing
Fixed shared hyperparameters, no sensitivity analysis
Unexplained anomalies in reported numbers
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper addresses LLM unlearning for serialized tabular records and argues the setting is qualitatively different from free-form text unlearning: all rows share one schema (repeated column-name/value slots, similar value ranges, a common output space), producing what the authors call schema-induced forget-retain overlap (Sec. 1, Sec. 3). Three claims carry the paper: (1) this overlap makes retain preservation actively oppose the forgetting update; (2) CAU — which filters or downweights retain rows nearest the forget distribution — matches a retraining oracle better than RMU, NPO, SimNPO, and GradDiff; (3) the gain comes from schema-aware selection, not merely fewer constraints. Claims (1) and (3) are reasonably supported; claim (2) holds in high-overlap regimes but not uniformly, and the evidence base carries no variance reporting.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Conflict-Aware Unlearning (CAU)?
CAU is a modification to LLM unlearning for tabular data: before unlearning, each retained row is scored by a Forget-Normalized Distance (a z-scored, schema-aligned distance to the forget set's per-column attribute statistics), and the retained rows closest to the forget set (below the 30th percentile) are dropped or downweighted in the retain-preservation loss. The forgetting objective itself is unchanged.
Does CAU outperform RMU and NPO?
Not uniformly. It usually attains smaller deviation from the retraining oracle's membership-inference AUC in high-overlap forget regimes (e.g., |Δ|=0.0418 vs 0.1214-0.4832 for baselines on Adult Income feature-level with Qwen2.5-3B), but in several low-overlap settings RMU matches or beats it (Adult Income 3B low-overlap: RMU |Δ|=0.0420 vs CAU 0.2850).
What data and models were used?
Qwen2.5-3B/7B-Instruct fine-tuned on three public UCI tabular datasets (Diabetes, Diabetic Retinopathy Debrecen, Adult Income), with k-means-based high- and low-overlap forget sets, evaluated by test accuracy and a loss-based MIA AUC against a retrain-from-scratch oracle.
Is code available for this paper?
Not yet as of this arXiv version: the appendix states that full configs and scripts 'will be released with the code'. The method is built on the public Open-Unlearning framework, and all datasets are from the UCI repository.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1's Diabetes block lost its method-row labels in extraction and several cells are garbled (e.g., Test Acc '4.6364', |Δ| '4.1269'); those cells were not cited and results from that block are avoided.
- Figures 4 and 5 — the sample-level Diabetes/Retinopathy results and the retain-selection ablation — exist only as graphics; their underlying numbers are not recoverable from the text, so those results could not be numerically audited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.