Influence Score and Transformers interpretability: Measure of the Effective Impact of Attention Heads at inference time
Lisa Bouger, Yannick Teglia, Philippe Loubet Moundi
cs.CL
Quick verdict
Head influence score (logit projection × ALTI contribution) for a DeBERTa injection classifier: promising, but single-model, weak stats, unvalidated labels.
Summary
The paper introduces an inference-time influence score that quantifies how much each attention head in a Transformer classifier drives a binary decision. The score multiplies three terms: the projection of a head's output onto the direction separating the two class logits (w = W[INJECTION] − W[BENIGN]), the head's ALTI-style L1-proximity contribution within its layer, and the layer's normalized contribution to the residual stream; it requires neither gradients nor ablations. The method is applied to a publicly available DeBERTa-v3-base prompt-injection detector (12 layers) evaluated on roughly 30,000 prompts relabeled by an LLM-as-a-judge consensus of four models. Analyses at head, layer, and network scales show that correct and erroneous predictions follow distinct internal dynamics: the global score is unimodal for true positives but trimodal for true negatives (three K-means clusters matching long structured, medium instruction, and micro prompts), while errors exhibit late-layer imbalances concentrated in L10–L11 heads (Table 1). Ablating the six top-scored heads degrades F1 by 0.007–0.015 depending on the selection criterion; in the true-positive group the combined score outperforms direction-only and contribution-only selection, but in the true-negative group direction-only selection yields a larger degradation, tempering the claimed superiority. Main limitations: a single model and task, no statistical testing of small ablation deltas, unvalidated LLM-judge labels, inconsistent dataset statistics between main text and Appendix B, and code available only on request.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Neither directional logit influence nor structural contribution alone characterizes a head's effective role; their product S = Δ·C·α does.
- Ablating heads selected by the combined score degrades performance more than heads selected by direction or contribution alone, showing the score identifies causally involved components.
- Correct and incorrect predictions have distinct internal dynamics: TP decisions are unimodal and stable, TN decisions arise through three multimodal regimes, and errors stem from late-layer (L10–L11) imbalances.
- The score provides a systematic, gradient-free compromise between fine-grained circuit analysis and output-only interpretability methods for Transformer classifiers.
Strengths
Well-motivated, gradient-free multi-scale score
Section 5 combines logit-direction projection with ALTI contribution, filling a real gap identified in Section 2: ALTI (Ferrando et al., 2022) lacks decision polarity, the logit lens lacks component attribution; the score computes at inference with no gradients or ablations.
Ablation compared against both component criteria
Table 2 ablates six heads selected by combined score vs Δ-only vs C-only; in the TP group the combined score yields the largest degradation (ΔF1 0.014 vs 0.009 and 0.007), supporting the design where it matters most (attack detection).
Rich descriptive phenomenology of correct vs erroneous circuits
Honest limitations discussion
Weaknesses
TN-group ablation contradicts the central superiority claim
No statistical rigor for small effects
Unvalidated, internally inconsistent evaluation data
Single model, single task, vanilla-Transformer assumptions
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the influence score for attention heads proposed in this paper?
It is an inference-time score S = Δ × C × α for each attention head of a Transformer classifier. Δ is the projection of the head's output onto the direction separating the two class logits; C is the head's ALTI-style proximity-based contribution within its layer; α is the layer's normalized contribution to the residual stream. The score requires no gradients or ablations and can be aggregated per head, per layer, or over the whole network.
Which model does the paper analyze?
A publicly available DeBERTa-v3-base classifier fine-tuned for prompt injection detection (protectai/deberta-v3-base-prompt-injection on Hugging Face), a 12-layer Transformer with a binary classification head. The evaluation used roughly 30,000 prompts assembled from public datasets and relabeled by a consensus of four LLM judges (Mistral, Gemma 2, Llama 3.2, GPT-4o-mini).
Did the ablation experiments validate the influence score?
Partially. Removing the six highest-scoring heads lowered F1 by 0.007–0.015. In the true-positive group, heads chosen by the combined score caused a larger drop (ΔF1 0.014) than direction-only (0.009) or contribution-only (0.007) selection. In the true-negative group, however, direction-only selection caused a bigger drop (0.015) than the combined score (0.009), so the claimed superiority holds for only one of the two groups.
Is the code or evaluation dataset available?
Not publicly. A footnote states that code and datasets can be made available upon request for academic reproduction. The underlying classifier and the source datasets are public, but the LLM-judge-relabeled evaluation set and the analysis code are not released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 appear only as captions in the extracted text; distribution shapes (e.g., the TN trimodality in Figure 3) are taken from the authors' in-text descriptions and could not be verified from the plots.
- The extraction ends at Appendix C / Table 4, including a stray trailing 'TN.' fragment likely from the table footer; any content after Appendix C is absent from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.