Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models
Kangwook Ko, Jaehyuk Jang, Wonjun Lee, Hee-Seon Kim, Changick Kim
cs.CL · cs.CV
Quick verdict
Proposes localized retain-free MLLM identity unlearning (PAVA); strong ablations, but single-seed results and test-set hyperparameter tuning limit certainty.
Summary
This paper tackles identity unlearning in multimodal LLMs without a retain set. The authors first localize identity knowledge via three analyses on LLaVA-1.5-7B (MLLMU-Bench): causal tracing shows identity retrieval is mediated by MLP computation at content tokens in modality-specific layer bands (VQA L7–12, QA L0–5); weight transplant shows MLP weights alone restore ~79% of identity knowledge (attention only 58%); Fisher overlap is lowest for decoder MLPs between identity-knowledge and visual-attribute signals. Building on this, PAVA confines LoRA updates to the top-indirect-effect decoder MLP layers (K=L/4), applies NPO to identity queries, and adds a Visual-Attribute Anchor distilling the pre-unlearning model's own answers to identity-agnostic visual questions about the forget images. On MLLMU-Bench (5% forget), PAVA reduces Qwen2.5-VL forget Correctness from 84 (vanilla) to 36 while retaining ROUGE-L 0.408 — the best trade-off among forget-only methods and competitive with retain-based GD/KL; results replicate on ReMem, on LLaVA-1.5, and on Qwen3-VL-8B, with sequential QA→VQA unlearning and a relearning attack (lowest recovery, +2.3 overall). Main limitations: all results come from a single seed-42 run with no variance estimates; the GPT-4o-mini judge defining both headline metrics is unvalidated against humans; hyperparameters (λ, learning rates) are tuned on the test benchmark; and curated synthetic profiles may overstate how sharply localization transfers to in-the-wild identities.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Identity knowledge in MLLMs is stored chiefly in early-to-mid decoder MLP layers, as shown by converging causal tracing, weight-transplant (~79% MLP-only recovery) and Fisher-overlap evidence.
- These layers can be modified with comparatively low interference to visual processing because identity-knowledge and visual-attribute Fisher signals overlap least in decoder MLPs.
- PAVA — NPO confined to top-IE MLP layers plus a visual-attribute anchor distilled from the forget images — achieves the best forget–retain trade-off among forget-set-only methods and is competitive with retain-based baselines on MLLMU-Bench and ReMem.
- Localized, pathway-aware updates compose for sequential modality-specific unlearning and show the lowest recovery under partial relearning attack (+2.3 overall vs GA's +19.7).
Strengths
Triangulated localization that heeds known pitfalls
§3 explicitly designs around Hase et al. (2023)'s finding that causal localization ≠ editing recipe: causal tracing (§3.1, modality-specific MLP bands VQA L7–12 / QA L0–5) is complemented by weight transplant testing storage vs transit (§3.2: MLP-only restores 79% vs 58% attention-only) and Fisher overlap testing editability (§3.3: decoder MLPs lowest). Replicated on Qwen2.5-VL and Qwen3-VL-8B (C.5–C.7).
Genuinely retain-free preservation signal
VAA (§4.3) turns identity-agnostic content in the forget images (hair, clothing, background) into a self-distillation anchor, addressing the availability/privacy dilemma motivating the paper. Table 3 shows it deepens forgetting (F-C 50→36) at roughly constant retention rather than trading one for the other.
Ablations that test mechanism, not just performance
Evaluation beyond static generation metrics
Reproducibility-minded appendices
Weaknesses
All evidence from a single run
Unvalidated LLM judge defines both headline metrics
Hyperparameters selected on the evaluation benchmark
Missing comparison with closest cited competitors
Synthetic, redundant-free identity setting undercuts the deployment story
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets post-deployment identity unlearning in MLLMs (GDPR-adjacent motivation, §1): remove a target individual's facts using only the forget set, since retain corpora are unavailable after deployment and rebuilding one re-creates the privacy exposure. Two claims anchor the work: (1) identity knowledge is stored in early-to-mid decoder MLPs, established by three independent probes; (2) editing only those layers, with a self-generated visual anchor (PAVA), yields the best forget–retain trade-off among forget-set-only methods and remains competitive with retain-based baselines. Evaluation spans MLLMU-Bench (5% and 10% forget ratios) and ReMem (single-identity, multi-hop) on LLaVA-1.5-7B and Qwen2.5-VL-7B, plus Qwen3-VL-8B in the appendix.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PAVA?
PAVA (Pathway-Aware Visual-attribute Anchoring) is a machine unlearning method for multimodal LLMs. It first localizes where a person's identity knowledge is stored (the decoder MLP layers with the highest causal indirect effect), then updates only those layers with an NPO forgetting loss plus a 'visual-attribute anchor' that distills the pre-unlearning model's own answers to identity-agnostic visual questions (hair color, clothing, background) about the forget images, so perception survives while the person's facts are suppressed.
Does PAVA need a retain set?
No. Unlike prior MLLM unlearning methods (MANU, MMUnlearner, GD, KL), PAVA uses only the forget data: the removal signal and the preservation signal both come from the forget images themselves. The paper reports it matches retain-based GD/KL on the forget–retain trade-off on MLLMU-Bench and ReMem without ever seeing retain examples.
Does it work on real people or deployed production models?
Not demonstrated. All experiments use MLLMU-Bench and ReMem, which contain synthetic, curated identity profiles injected by fine-tuning 7–8B open models. The authors themselves flag in Limitations that real-world identities reinforced across redundant web data may not localize as sharply, and that in-the-wild validation is future work.
Is code or data available for this paper?
The paper text contains no code or data release link, and the datasets/models used (MLLMU-Bench, ReMem, LLaVA-1.5, Qwen2.5-VL, Qwen3-VL) are all public with documented licenses, so reproduction would require reimplementation from the detailed appendices (layer lists, LoRA configs, judge prompt).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures extract as axis/heatmap text only; figure-level quantities (e.g., Fig 6 grounding-gap values, Fig 8 trajectories) are taken from the prose captions, not verified against plot pixels.
- No code or data release URL appears anywhere in the provided text; codeAvailable=false reflects the absence of a statement, not a stated refusal to release.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.