Not All Patches Are Equally Forgettable: Spatially Localized Domain Unlearning in Vision-Language Models
Akanksha Singh, Vinod K. Kurmi
cs.CV
Quick verdict
Patch-selective attenuation for CLIP domain unlearning beats ADU with good ablations, but gains are modest and the 3.8% retention claim is unverified.
Summary
The paper targets approximate domain unlearning in vision-language models: suppressing recognition behavior tied to a forget domain while preserving accuracy on retained domains. The authors first argue empirically that forget-domain evidence is spatially localized across ViT patch tokens: on Office-Home, suppressing only the top ~10% domain-scoring patches cuts forget accuracy from 63.23 to 37.20 while retained accuracy drops just 1.11 points. They then propose a two-stage framework on frozen CLIP ViT-B/16. Stage I trains a lightweight linear patch scorer (BCE plus margin ranking loss, image-level domain labels inherited by patches) at intermediate layer 5, and computes semantic utility as the maximum cosine similarity between projected patch embeddings and class text embeddings. Stage II attenuates patches whose Localized Domain Effect (domain score over semantic utility) exceeds a threshold, scaling them by (1 - alpha*r). Across ImageNet/ImageNet-Sketch, Office-Home, and Mini DomainNet with one to three forget domains, the method improves the harmonic-mean tradeoff over ADU (e.g., 80.15 vs 75.56 on Mini DomainNet) with roughly comparable retention, and beats global and random suppression baselines. Ablations cover components, patch granularity, intervention layer, and the margin objective. Main limitations: gains over ADU are often 1-2 points with no variance reported; the abstract's claim of up to 3.8% retained-accuracy improvement is not traceable in the main tables; forgetting is measured only by classification error, without information-leakage or relearning tests; DomainNet and unseen-domain results are deferred to supplementary material.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Forget-domain evidence is spatially localized in ViT patch tokens: suppressing roughly 10% of top-scoring patches reduces Office-Home forget accuracy from 63.23 to 37.20 with only a 1.11-point retention drop (Table 1).
- A two-stage framework that scores patch domain sensitivity against semantic utility (LDE) and attenuates high-LDE patches yields better forgetting-retention harmonic means than ADU, global suppression, and random suppression across ImageNet, Office-Home, and Mini DomainNet for |Df| = 1-3.
- Layer 5 of frozen CLIP ViT-B/16 is the deepest point where domain information remains linearly separable (probe accuracy 0.76) before class consolidation completes (0.70), making it the effective intervention site.
- Retained-domain accuracy stays essentially unchanged relative to ADU while forgetting improves, indicating gains come from more precise suppression rather than retention degradation.
Strengths
Empirically grounded central hypothesis
RQ-1 (Table 1) quantifies the core claim: suppressing only the top ~10% domain-scoring patches reduces Office-Home forget accuracy 63.23 to 37.20 with a 1.11-point retention drop; Fig. 4(b) layer probes (domain 0.76 vs class 0.70 at layer 5) ground the choice of intervention depth rather than leaving it arbitrary.
Lightweight, frozen-backbone design
Only a linear patch scorer is trained (SGD, 50 epochs, lr 0.0025); no suppression modules or backbone updates are introduced. Table 5 shows the tradeoff is stable across 3x3 to 28x28 patch granularities (H 69.36-71.63), suggesting robustness to that design choice.
Informative ablations and controls
Consistency across forgetting difficulty
Weaknesses
Unverifiable headline claim in the abstract
Thin statistical reporting
Weak operationalization of 'unlearning'
Evaluation leans on unavailable supplementary material
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is approximate domain unlearning?
It is the task of making a pretrained model stop exhibiting recognition behavior tied to a chosen 'forget' domain (for example a sketch or clipart style) while keeping its accuracy on the remaining domains, without full retraining. Performance is typically reported as retained-domain accuracy versus forget-domain error, combined into a harmonic mean.
How does this method decide which patches to suppress?
A lightweight linear scorer is trained on intermediate ViT patch tokens to rate each patch's sensitivity to the forget domain using image-level domain labels, and each patch also receives a semantic-utility score from its similarity to class text embeddings. Patches whose domain score is high relative to their semantic utility are scaled down before the remaining transformer layers process the image.
What results does the paper report?
On frozen CLIP ViT-B/16 evaluated on ImageNet/ImageNet-Sketch, Office-Home, and Mini DomainNet with one to three forget domains, the method reports higher harmonic-mean forgetting-retention tradeoffs than the ADU baseline (for example 80.15 versus 75.56 on Mini DomainNet with one forget domain) while retained-domain accuracy stays roughly unchanged.
Is code available for this paper?
The paper text does not mention a released code or data repository; the full DomainNet results and several analyses are stated to be in the supplementary material.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary material (full DomainNet results, unseen-domain and supervision-budget experiments, dataset-level overlap analysis, layer-selection details) is absent from the extracted text, so abstract claims relying on it could not be verified.
- Figures 1-5 are available only via captions; the qualitative localization claims rest on visualizations that could not be inspected.
- Table 2's |Df|=3 ADU row appears to contain one more numeric value than there are columns, so exact per-metric comparisons for that setting are uncertain.
- No code availability statement appears in the extracted text; codeAvailable=false reflects absence of a statement rather than a confirmed negative.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.