ResLRP: The Role of Residual Cancellation in Attribution Instability in Vision Transformers
Jim Berend, Reduan Achtibat, Daniel Sch\"affer, Alexander Binder, Wojciech Samek, Sebastian Lapuschkin, Maximilian Dreyer
cs.CV · cs.AI · cs.LG
Quick verdict
Identifies residual cancellation as driver of ViT LRP instability and proposes ResLRP; well supported by causal interventions and broad benchmarks, marred by minor table inconsistencies.
Summary
The paper identifies residual-connection cancellation as a key cause of noisy, unfaithful LRP attributions in Vision Transformers. When attention or MLP updates oppose the residual stream, the merged activation becomes small, and standard epsilon-rules divide by this small denominator, amplifying relevance into large opposing pairs ('relevance explosion'). The authors quantify cancellation (C) and amplification (A), finding Pearson correlation 0.99 across models, with ViTs worse than LLMs and early layers worst. Their fix, ResLRP, applies an LRP-gamma rule at residual merges that favors sign-consistent branches; it is exactly conservative and provably bounds per-site amplification at 1+2/gamma. Causal support comes from channel-wise interventions: correcting only the top 0.5% cancellation-prone channels improves SRG by 7.9%, while treating low-cancellation channels hurts. Benchmarks across seven ViT checkpoints, FunnyBirds ground truth, SAE feature localization, and three VLMs consistently favor ResLRP, with the largest VLM gains (+27-29% localization, up to 3.4x SRG). Limitations: guarantees are local per-merge only and the global bound is vacuous in practice; end-to-end conservation depends on the AttnLRP attention rule; several baseline values are inconsistent across tables without stated protocol differences; and the method itself is a one-line adaptation of an existing gamma-rule, so the contribution rests mainly on the diagnosis.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Residual-path cancellations (zin ~ -zup) are a primary cause of relevance explosion and attribution instability in ViT LRP, substantially stronger than in language transformers and concentrated in early layers.
- Applying an LRP-gamma rule at residual merges (ResLRP) yields exact conservation (Rin+Rup=Rout) and bounds per-merge amplification at 1+2/gamma, where the standard rule admits no finite bound.
- Targeted channel-wise interventions show the quality gains stem specifically from correcting residual cancellation rather than generic regularization (top-0.5% channels +7.9% SRG vs bottom-75% channels -5.5%).
- ResLRP improves faithfulness and localization across ViT families and three VLMs (+27-29% relative localization, up to 3.4x SRG), and best localizes SAE features to input space.
- The residual amplification measure A is an architecture-level diagnostic: register tokens reduce A, and high A predicts where attribution methods degrade, most notably in VLMs.
Strengths
Causal isolation of the mechanism, not just correlation
Sec. 5.2 / Tab. A.8: correcting only the top 0.5% most cancellation-prone channels improves SRG by +7.9% while treating 75% of low-cancellation channels gives -5.5%; SRG correlates with unstable relevance removed (Spearman rho=0.79) far more than with channel count (rho=0.50), and the pattern replicates on DINOv2-B.
Exact, checkable theory at the modified site
Sec. 4.1 / A.2 prove conservation (Phi(a,b)+Phi(b,a)=1) and a tight per-merge amplification bound |Rin|+|Rup| <= (1+2/gamma)|Rout| with a short, verifiable case analysis; the proof handles both sign cases and the symmetry argument is sound.
Triangulated evaluation across independent evidence types
Deployment-relevant demonstrations with statistical care
Practical simplicity and hyperparameter robustness
Weaknesses
Internal inconsistencies in reporting
Theoretical guarantees are local and partly vacuous at scale
Modest methodological novelty
Limited protocol coverage in key benchmarks
Unexamined sensitivity of the localization metric design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ResLRP?
ResLRP is a modification of Layer-wise Relevance Propagation (LRP) for Vision Transformers. It adds a gamma-rule at each residual (skip) connection that favors branches whose contribution has the same sign as the merged output. This prevents the small-denominator amplification of relevance that occurs when the residual stream and an attention or MLP update cancel each other, while keeping relevance exactly conserved.
Why do LRP explanations look noisy or checkered on Vision Transformers?
According to this paper, because attention and MLP updates frequently oppose the incoming residual stream, producing small merged activations. Standard LRP rules divide relevance by this small value, blowing it up into large opposing positive-negative pairs ('relevance explosion') that appear as checkerboard-like noise. The effect is strongest in early layers and much weaker in language transformers.
Does ResLRP require hyperparameter tuning?
Essentially no. It has a single hyperparameter gamma, and the untuned default gamma=1 reaches 98-100% of the best swept value on most evaluated models, with a broad performance plateau for any gamma > 0; gamma=0 recovers the standard AttnLRP baseline.
How much better is ResLRP than previous methods on vision-language models?
On Qwen2.5-VL-3B, Qwen3-VL-4B, and Gemma-3-4B-it, ResLRP improves localization over AttnLRP by 27-29% relative and improves the SRG occlusion-faithfulness score by factors of about 1.9 to 3.4, with all differences statistically significant under paired Wilcoxon signed-rank tests. The largest gains occur exactly where residual cancellation is strongest.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures and heatmaps (Fig. 1-3, A.1-A.5) cannot be analyzed from extracted text; qualitative claims about map cleanliness are taken from captions on trust.
- Equation rendering is partially garbled by text extraction (Eqs. 1, 3, 4); interpretations were reconstructed from the surrounding prose and the A.2 proof.
- Some cross-table value discrepancies could partly stem from extraction errors, though each table is internally consistent as extracted.
- The appendix appears complete through A.16 with no evident truncation, so confidence in coverage of experiments is high.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.