Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O'Neill, Guillaume Bourgeon, Houssem Chatbri, Christopher Martin, et al.
cs.AI
Quick verdict
Asks if LLMs' cited top-3 features match measured necessity/sufficiency of their decisions; sound design, but synthetic data, 3-trial scores, no data release.
Summary
This paper asks whether the top-three factors that LLMs cite in their own explanations actually correspond to what influences their decisions. Across two synthetic use cases — financial advisor recommendation and prompt-risk monitoring — the authors query eight configurations (Claude Opus 4.6, Sonnet 5, Haiku 4.5; GPT-5.4 with reasoning disabled plus Low, Medium and High effort; Gemini 3.5 Flash) at temperature 0, then apply black-box input interventions: necessity is estimated by changing one feature and counting output changes; sufficiency by retaining one feature, masking the rest, and counting output preservation. Mean Spearman correlations between cited rank and measured scores are 0.349 (necessity) and 0.354 (sufficiency) for advisor recommendation, and 0.431 and 0.580 for prompt monitoring. An uncited feature outscores the weakest cited one in 57.6%/58.1% of advisor responses and 25.8%/8.9% of prompt-monitoring responses. The authors conclude that explanations carry useful but inconsistent signal, with feature selection and ordering being distinct reliability properties. Limitations: synthetic, unreleased data partly generated by an evaluated model; only three trials per intervention; no significance testing or code release; conclusions depend partly on the chosen intervention strength.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The order of the cited top three only partially agrees with intervention-measured influence: mean Spearman correlations are 0.349 (necessity) and 0.354 (sufficiency) for advisor recommendation, 0.431 and 0.580 for prompt monitoring, across eight model configurations.
- The cited top three frequently omit a stronger feature: an uncited feature outscores the weakest cited one in 57.6%/58.1% of advisor responses and 25.8%/8.9% of prompt-monitoring responses under necessity/sufficiency.
- Feature selection and feature ordering are distinct reliability properties, e.g. Opus 4.6 omits fewer strong features than Sonnet 5 in advisor recommendation while Sonnet 5 has better rank agreement (0.445 vs 0.178).
- Citation frequency tracks measured influence in advisor recommendation (ρ=0.903 necessity) but shows a salience gap in prompt monitoring, where the most-cited category (jailbreak_attempt) is not the strongest under either criterion.
- Higher GPT-5.4 reasoning effort reduces sufficiency-based omissions (76%→40% advisor; 25%→3% monitoring) but does not consistently improve the ordering of cited features.
Strengths
Dual-criterion intervention framework with a real conceptual payoff
Separating necessity (PN) from sufficiency (PS) exposes that the two orderings of the same cited features disagree (Appendix D.2: Spearman 0.18-0.35 advisor, 0.38-0.87 monitoring), grounding the paper's claim that selection and ordering are distinct reliability properties.
Elicitation is controlled before evaluation
Section 3.6/Appendix A.1 compare 13 prompt formats; fixed feature names cut positional instability from 0.929 to 0.235 (advisor) and 0.858 to 0.240 (monitoring), and the authors explicitly disclaim that reproducibility is not evidence of faithfulness.
Interventions are validated, not assumed
Practically consequential negative result
Honest scoping
Weaknesses
Three trials per intervention make headline rates statistically fragile
Rank-agreement conclusions are intervention-dependent beyond acknowledged absolute shifts
No data or code release
Elicitation prompt may bias toward the tested interpretation
Generator-evaluator overlap in prompt monitoring
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: when an LLM component returns a decision plus a cited top-3 list of influential factors, do those features — and their stated order — agree with the feature's measured behavioural influence, operationalised as intervention-based necessity (PN: fraction of trials where changing the feature changes the output) and sufficiency (PS: fraction of trials where retaining the feature while masking the rest preserves the output). Two synthetic deployment-motivated use cases (advisor recommendation: 100 client profiles against 13 fixed advisors; prompt-risk monitoring: 100 prompts with 3-4 of 8 risk segments) and eight configurations (Claude Opus 4.6/Sonnet 5/Haiku 4.5; GPT-5.4 at four reasoning-effort settings; Gemini 3.5 Flash) at temperature 0. Central claims: (1) cited rankings correlate only partially with PN/PS (means 0.349/0.354 advisor, 0.431/0.580 monitoring); (2) the cited top three frequently omit a higher-scoring feature (57.6%/58.1% advisor, 25.8%/8.9% monitoring); (3) feature selection and ordering are distinct reliability properties. Claims are appropriately scoped to observable behaviour, not internal reasoning.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper measure about LLM explanations?
Whether the top-three factors an LLM cites in its own explanation match what actually drives its decision, judged by black-box input interventions: necessity (change the factor — does the output change?) and sufficiency (keep the factor, remove the rest — is the output preserved?).
What are the paper's main findings?
Agreement is partial: mean Spearman correlations between cited rank and measured scores are 0.349/0.354 for advisor recommendation and 0.431/0.580 for prompt monitoring, and an uncited feature beats the weakest cited one in 57.6%/58.1% of advisor responses and 25.8%/8.9% of prompt-monitoring responses. Explanations are informative but unreliable as oversight evidence.
Which models were evaluated?
Eight configurations at temperature 0: Claude Opus 4.6, Sonnet 5 and Haiku 4.5; GPT-5.4 with reasoning disabled plus Low, Medium and High effort; and Gemini 3.5 Flash, across two synthetic tasks (financial advisor recommendation and prompt-risk monitoring).
Is the code or data available?
No. The authors state the synthetic records cannot be released (proprietary evaluation schemas, potentially harmful prompt content) and provide only input schemas, prompts and intervention procedures; no code release is mentioned.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure interiors (data behind Figures 1-13) are not extractable from the PDF text; in-figure quantitative claims were verified only via captions and numbers repeated in the body text. The extracted text otherwise appears complete (abstract, Sections 1-5, references, Appendices A-D).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.