Limitations of Automated Simulatability: LLM Simulators Can Bypass Explanations
Antonin Poché, Fanny Jourdan, Nils Feldhus, Qianli Wang, Jing Yang, Simon Ostermann, Nicholas Asher, Philippe Muller, et al.
cs.CL · cs.LG
Quick verdict
Persuasive self-critique: LLM simulators lean on task priors and leak anonymized labels; evidence mostly correlational, ≤15B models, but fixes actionable.
Summary
This paper self-critically audits ConSim (Poché et al., 2025), the authors' own automated-simulatability framework in which an LLM simulator replaces human explainees and explanation quality is scored by how well the simulator predicts a task model's outputs. The authors first replicate the original qualitative ranking (Vanilla SAE, ICA, Semi-NMF above the no-explanation baseline; PCA/SVD at the bottom) across six datasets, three explanation families, and four open simulators (Qwen-3.5-9B, Gemma-4-12B, Phi-4, Llama-3.1-8B), using 50 seeds grouped into ten 100-prediction observations. They then document two failure modes. First, in non-anonymized prompts, explanation gains over matched baselines are statistically significant but small: a class-names-only baseline (B1) already exceeds 0.7 on several datasets, prompt types agree 77–90%, and on IMDB simulators agree with gold labels at ~95% while the task model agrees only ~50% — consistent with a shortcut where simulators solve the task from priors. Second, ConSim's class anonymization is defeated by a new classes-as-concepts baseline that trivially names each class; it tops all methods under anonymization (win rates 55–88%, mean advantage up to +0.138), rewarding label leakage. Findings are presented as consistent-with rather than proving the shortcut; caveats include simulators ≤15B without reasoning, one anomalous simulator (Llama-3.1-8B reverses gains), untested contamination, and rationales judged by a same-family model. Recommendations: error over-sampling, leakage baselines, multiple simulators, harder tasks, effect-size reporting.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- In non-anonymized automated simulatability, explanations yield only small gains over no-explanation baselines across concepts, attributions, and rationales, consistent with simulators solving the classification task from task priors (the shortcut hypothesis).
- ConSim's class anonymization does not force explanation-reliant evaluation: a classes-as-concepts baseline that trivially names each class ranks first under anonymization (win rates 55–88%, mean advantage up to +0.138), rewarding hidden label-mapping leakage.
- Convergent evidence includes a class-names-only B1 baseline exceeding 0.7 on several datasets, 77–90% prediction agreement across prompt types, and IMDB simulator-gold agreement ~95% versus ~50% for the task model.
- Explanation effects grow when the task is hard (GoEmotions subsets, task model F1 0.541), and explicit simulator framing (simulator_consim) does not improve — and usually lowers — agreement with the task model.
- Robust protocols should over-sample task-model errors, include label-leakage baselines, use multiple and stronger simulators, prefer hard or novel tasks, and report effect sizes alongside significance.
Strengths
Rare self-critique of the authors' own evaluation framework
The ConSim authors replicate their own protocol (Sec. 3, Figs. 9–11) and then document its two failure modes rather than defending it, which strongly mitigates adversarial-motivation concerns.
Convergent, quantified shortcut evidence
Class-names-only B1 exceeds 0.7 on several datasets; cross-prompt agreement is 77–90% (Figs. 3/19); on IMDB simulators agree with gold at ~93–98% while the task model agrees with gold at ~50% (App. L); qualitatively opposed concepts (Table 4) score identically (.712).
Cheap, reusable diagnostic baseline
Statistical hygiene above field norm
Reproducibility scaffolding
Weaknesses
Shortcut hypothesis is correlational only
One anomalous simulator breaks the cross-LLM consistency claim
Single representative method per explanation family
Judge-generator family overlap for rationales
No human baseline and no contamination controls
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper is a self-audit: the authors replicate and then undermine ConSim (Poché et al., 2025), their own automated-simulatability protocol in which an LLM simulator replaces the human explainee and explanation quality is scored by how well the simulator predicts a task model's held-out outputs. Three claims are advanced: (1) the protocol's qualitative method ranking replicates across datasets, explanation families, and simulators; (2) in non-anonymized settings, scores are dominated by a task-solving shortcut — a class-names-only baseline (B1) already exceeds 0.7 on several datasets, so high simulatability does not imply explanation use; (3) ConSim's anonymization fix is itself exploitable: a new classes-as-concepts baseline trivially leaks the hidden label mapping and tops every concept method. The framing is carefully hedged — findings are “consistent with” the shortcut hypothesis, not proof of it.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is automated simulatability?
An evaluation protocol for explanations where a human user is replaced by an LLM 'simulator': the simulator is shown a classifier's explanations plus examples, then asked to predict the classifier's outputs on held-out inputs. Prediction accuracy measures how useful the explanation is. ConSim (Poché et al., 2025) applied this to concept-based explanations.
What are the two limitations this paper finds?
(1) The shortcut problem: when class names are meaningful (e.g., 'surgeon' vs 'nurse'), the LLM simulator can score highly by solving the classification task from its own prior knowledge, without reading the explanations — a names-only baseline already exceeds 0.7 on several datasets. (2) The leakage problem: anonymizing class names doesn't help if explanations reveal which anonymous label maps to which real class — a trivial 'classes-as-concepts' baseline that just names the class tops all methods under anonymization.
What is the classes-as-concepts baseline?
A diagnostic introduced by this paper in which each 'concept' in the explanation is simply the class name itself. In normal prompts this is redundant, but in anonymized prompts (classes renamed Class_0, Class_1...) it directly reveals the hidden mapping, e.g., 'Class_0 = surgeon'. It ranked first across all anonymized prompt types (win rates 55–88%), proving anonymized scores can reward label leakage rather than genuine explanation quality.
Does this mean LLM-judged explanation evaluations are useless?
No. The paper's point is that current scores can be misleading without guards. It offers concrete fixes: include a label-leakage baseline, over-sample the task model's errors so model-specific behavior matters, use multiple and stronger simulators, choose tasks that cannot be solved from pretrained knowledge, and report effect sizes with statistical tests rather than rankings alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are embedded in the extraction as flattened numeric matrices; visual claims (violin overlaps, bar magnitudes) were assessed from reported numbers, captions, and appendix text rather than rendered plots.
- The extracted paper text lists Vera Schmitt as an 11th author, who is absent from the supplied metadata's author list — a metadata mismatch, not missing text; the provided text itself appears complete through Appendix M.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.