Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning
Pengyang Shao, Chuanpeng Lu, Wei Qin, Yanzheng Jin, Xiaohao Liu, Xi Ai, Kenji Kawaguchi, Richang Hong
cs.CL · cs.AI
Quick verdict
CLLPU: 10-language benchmark separating insufficient from excessive unlearning propagation; rigorous build, single-model evidence, no significance tests.
Summary
The paper introduces CLLPU, a benchmark that reframes multilingual LLM unlearning as a propagation-boundary problem. Beyond the common-goal setting, where target knowledge should be suppressed in every language, it adds a language-conditioned setting in which suppression should stay confined to a designated source language. The benchmark covers ten languages and was built from 80 post-Llama-3.1 Wikipedia topic pairs through knowledge-unit extraction and schema-aware relation matching, producing 800 forget-retain unit pairs plus 800 holdout units, then dual-anchor translation verified by blind back-translation, GPT-5.5 cross-model review, and a 2,000-QA human audit, yielding 72,000 QA instances. Evaluating GA, GD, NPO, SimNPO, and two BalDRO variants on Llama-3.1-8B-Instruct, the authors find opposite failure modes: cross-lingual forgetting stays incomplete under the common-goal objective (GA drops Source forget ROUGE-L to 0.0192 but Cross remains 0.0437, with strong source-language dependence and directional asymmetry), while language-conditioned forgetting over-propagates (best Cross SS only 0.5103 versus 0.8910 for the original model). MIA results are validated against Original (AUC 0.93-0.97) and Retrain (roughly 0.5) references. Main limitations: all conclusions rest on one model with SFT-injected knowledge; Tables 1-3 lack variance or significance testing; hyperparameter checkpoints were screened on the evaluated source-language examples; and the language-conditioned objective has no retraining oracle. Code is released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing multilingual unlearning evaluations measure cross-lingual transfer but cannot distinguish insufficient propagation (common-goal) from excessive propagation (language-conditioned).
- Under common-goal forgetting, all six evaluated methods forget far more weakly across languages than in the source language, with strong source-language dependence and directional asymmetry (GD: bn-to-en ROUGE-L 0.519 vs en-to-bn 0.009).
- Under language-conditioned forgetting, no method confines suppression to the source language: the best Cross SS is 0.5103 versus 0.8910 for the Original model (Table 2).
- General multilingual utility (Belebele) conceals localized damage: GA retains 0.7868 Belebele accuracy while matched retain accessibility collapses in the language-conditioned setting.
- Sentence-similarity metrics overestimate residual knowledge because relation-matched forget/retain answers have baseline BGE-M3 similarity of 0.48-0.50 (Appendix C).
Strengths
Novel problem formulation
Sections 1 and 3.1 define common-goal vs language-conditioned forgetting, separating insufficient from excessive propagation that prior cross-lingual transfer evaluations (cited [13,17,34]) conflate; the language-conditioned objective is explicitly framed as a stress test rather than a deletion oracle (Section 3.1.1), an unusually honest scoping move.
Contamination-controlled, multiply verified construction
Only post-Llama-3.1 (after 2024-07-23) topics with recorded temporal evidence (Appendix A.1); 4,649 units distilled to 800 relation-matched forget-retain pairs plus 800 unit-level-disjoint holdout units via a deterministic schema-aware matcher (Table 4); dual-anchor translation with blind back-translation, GPT-5.5 cross-model review, and a 2,000-QA human audit spanning all nine target languages (A.3).
Matched-pair probes expose damage invisible to general utility
Self-anchored MIA protocol
Honest metric introspection
Weaknesses
Single-model, SFT-injected evidence
No uncertainty or significance reporting
Checkpoint selection on evaluated data
Language-conditioned objective lacks an attainability check
Verification pipeline self-selects its audit sample
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CLLPU (Sections 1-3) reframes multilingual LLM unlearning as a propagation-boundary problem. Beyond the conventional common-goal setting — target knowledge suppressed regardless of query language — it adds a language-conditioned setting in which suppression should stay inside a designated source language. The paper claims three things: prior evaluations cannot separate insufficient from excessive propagation; CLLPU operationalizes both objectives with matched, verified multilingual probes; and six representative methods fail in opposite directions on Llama-3.1-8B-Instruct. The authors are commendably careful to frame language-conditioned forgetting as a benchmark-specific stress test rather than a data-deletion oracle (Section 3.1.1) and to scope model claims explicitly (Section 5.1.1). The core claims are supported by the reported evidence, subject to the generalization caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CLLPU benchmark?
CLLPU is a multilingual benchmark for evaluating LLM unlearning (removing specific knowledge from a model). It spans ten languages, contains 800 matched forget-retain knowledge-unit pairs and 800 holdout units (72,000 QA instances total), and evaluates two objectives: common-goal forgetting (knowledge suppressed in every language) and language-conditioned forgetting (suppression confined to one designated language).
What are the paper's main findings?
On Llama-3.1-8B-Instruct with six unlearning methods (GA, GD, NPO, SimNPO, and two BalDRO variants), forgetting is incomplete when it should propagate to all languages, yet over-propagates when it should stay in a single language. It also shows general utility benchmarks like Belebele can miss severe damage to semantically related knowledge.
Is the CLLPU code and dataset publicly available?
Yes. The paper states CLLPU and its full construction pipeline (prompts, code, human-review workflow) are publicly released at github.com/CLLPU/CLLPU-bench.
Why does the benchmark use topics from after July 2024?
All topics postdate the Llama 3.1 release (July 23, 2024) so the target knowledge is unlikely to have been seen during pretraining; the knowledge is instead injected into the model via supervised fine-tuning, giving a controlled starting point shared by all unlearning methods.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Heatmap and bar values (Figures 2-5, 6-10) are only available through numbers quoted in the prose; per-cell values could not be independently read from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.