$μ^2$-Bench: A Multilingual Machine Unlearning Benchmark
Kyomin Hwang, Hyeonjin Kim, Hyunho Lee, Yearim Kim, Yeji Song, Nojun Kwak
cs.CL
Quick verdict
Presents a 12-language unlearning benchmark; multilingual-specific unlearning best reaches hold-out languages, but evidence is single-run and code unreleased.
Summary
The paper introduces μ²-Bench, a benchmark for multilingual machine unlearning (MMU) that tests whether targeted knowledge is truly removed from LLMs in all languages, including those never seen during unlearning. From 250 fictitious profiles with 20 attributes each, the authors build 5,000 English QA pairs, machine-translate them into 12 languages (60,000 pairs total) spanning Latin and non-Latin scripts and high/low-resource languages, then fine-tune Gemma3-12B and Qwen2.5-7B on four training languages per setting (two script-balanced settings), apply GA, GAGDR, NPO, and LingTea at 1/3/5% forget ratios, and evaluate on four held-out languages using probability, LLM-judged semantic equivalence, privacy leakage, knowledge-separability (KSS), and TyDi QA utility metrics. LingTea, the only multilingual-specific method, ranks first in 58/84 training-language and 72/84 hold-out-language evaluations, with its average margin widening from +14.96 to +27.94. The authors also show that unlearning induces code-mixed outputs (e.g., Spanish 'lobos' replacing 'wolves'), motivating semantic over surface-form metrics. Key limitations: no significance testing or seed variance across the 84 comparisons; template-based synthetic data; translation validated without native speakers; the GPT-4o-mini LLM judge is unvalidated; and no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing MMU benchmarks evaluate language-wise and never test knowledge that spread into hold-out languages (Sec. 2.3), so they cannot tell whether a fact was removed across all its multilingual expressions.
- Aggregated over 2 models × 2 settings × 3 forget ratios, LingTea ranks first in 58/84 training-language and 72/84 hold-out-language metric evaluations, with its average margin over the strongest competitor widening from +14.96 to +27.94 (Sec. 5.2).
- Unlearning induces code-mixed outputs in which target knowledge stays retrievable in another language (Fig. 4: Spanish 'lobos' overtakes English 'wolves' after NPO on Gemma3-12B), so surface-form metrics like ROUGE-L underestimate residual knowledge.
- General knowledge, measured as the average of TyDi QA F1 and EM, remains comparable to the memorized model after unlearning across all configurations (Table 2, last column; e.g., Gemma3 S1 p1: 30.19 → 30.84).
Strengths
Evaluation axes prior MMU benchmarks lack
Two script-balanced settings separate training from hold-out languages (Table 1: Setting 1 trains on EN/DE/ID/AF, holds out ES/LV/ZH/UR; Setting 2 trains on non-Latin ZH/EL/MK/TH, holds out EN/LV/KO/UR), directly testing whether knowledge acquired via cross-linguistic spread (memorized-model hold-out PS ~55.8 vs ~99.99 in-training, Table 2) survives unlearning — something TOFU-translation benchmarks (Lizzo & Heck 2026; Farashah et al. 2026) and FAME (Savelli et al. 2025) do not do.
Knowledge-wise metrics with a principled gold standard
KSS-Norm (Eq. 7) and PrivLeak (Eq. 5) both normalize against a reference model fR trained only on non-target knowledge, giving a well-defined target for 'ideal' unlearning rather than ad hoc thresholds; evaluation aggregates across the knowledge instance's 12 language expressions instead of per-language scores.
Unusually well-documented dataset pipeline
Novel and practically relevant code-mixing finding
Controlled, fair comparison grid
Weaknesses
No statistical rigor behind the headline counts
Synthetic template domain limits external validity
Translation QC without native speakers (admitted, but consequential)
Unvalidated LLM judge and model overlap
No artifact release; novelty partly inherited from prior work
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
μ²-Bench (arXiv 2609.20945, cs.CL) addresses a genuine evaluation gap: multilingual machine unlearning (MMU). The authors argue in Sec. 2.3 that existing benchmarks — TOFU translated into multiple languages (Lizzo & Heck 2026; Farashah et al. 2026) and FAME (Savelli et al. 2025) — score each language separately and never test whether knowledge that propagated into hold-out languages via cross-linguistic spread is actually gone. Their prior framework (Hwang et al. 2026) partially addressed this but only with Latin-script hold-outs. The paper's four claims are: (1) a 12-language synthetic QA benchmark spanning scripts and resource levels; (2) a memorize→unlearn→evaluate pipeline separating training from hold-out languages (Table 1); (3) knowledge-wise evaluation metrics (PS, SE, PrivLeak, KSS-Norm, TyDi QA); (4) the empirical conclusion that multilingual-specific unlearning is necessary, plus a code-mixing side-effect finding. Claims 1-3 are supported by construction; claim 4 is directionally credible but statistically under-supported.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is μ²-Bench?
μ²-Bench is a benchmark for multilingual machine unlearning (MMU): testing whether an LLM can truly 'forget' targeted knowledge in every language, including languages that were never shown to the model during the unlearning process. It uses 60,000 synthetic question-answer pairs across 12 languages, built from 250 fictitious person profiles with 20 attributes each, and evaluates unlearning methods on both training languages and held-out languages.
What is cross-linguistic spread and why does it matter for machine unlearning?
Cross-linguistic spread is the phenomenon where facts an LLM learns in some languages become retrievable in languages it never saw them in. For unlearning this means deleting a fact in the training language is not enough — the paper shows models can still answer questions about 'forgotten' facts in held-out languages, so unlearning must be evaluated (and performed) multilingually.
Which unlearning method works best for multilingual LLMs according to this paper?
In experiments on Gemma3-12B and Qwen2.5-7B, LingTea — the only multilingual-specific method tested — ranked first in about 69% of training-language metric evaluations and about 86% of held-out-language evaluations, beating generic methods GA, GAGDR, and NPO. The gap widens on held-out languages, supporting the conclusion that multilingual-aware unlearning methods are needed.
Is the μ²-Bench dataset or code publicly available?
The paper text contains no repository, dataset, or code release URL. The benchmark is synthetic (machine-translated, human-refined QA pairs) with full prompts and hyperparameters documented in the appendix, but the exact 250 profiles and refined translations cannot currently be obtained from the paper alone. Check the authors' pages or arXiv listing for an artifact link.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables 1 and 2 were extracted as run-on text; row/column alignment (e.g., Setting 2 training languages ZH/EL/MK/TH vs hold-out EN/LV/KO/UR) was reconstructed by inference, so individually cited cell values could be misattributed.
- Figures 1-4 are available only via captions; visual details such as the token probabilities in Fig. 4 come from caption text and garbled inline numbers.
- codeAvailable=false reflects the absence of any release URL in the extracted text, not a confirmed refusal to release artifacts.
- Reference list contains several 2026-dated works (Hwang et al. 2026, OpenUnlearning, Farashah et al. 2026) that could not be verified from this text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.