SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?
Yuqiao Tan, Shizhu He, Jun Zhao, Kang Liu
cs.AI · cs.CL · cs.LG
Quick verdict
Benchmark testing 10 frontier agents on SAE feature discovery in Gemma-2-9B; rigorous metrics, but single-model scope and untested memorization confound.
Summary
SAEScientist-Bench asks whether LLM agents can autonomously run mechanistic-interpretability experiments. Given a target concept, an agent designs contrastive probe texts, searches a 131,072-feature Gemma Scope dictionary on Gemma-2-9B-IT (layers 9 and 20), and submits one feature, scored against a frozen Neuronpedia expert reference on activation rank, AUROC selectivity over held-out positive and contrastive texts, and causal steering rated by a blinded GPT-4o judge against baseline and norm-matched random controls. Across 20 tasks, 10 model-harness configurations and three independent runs each, the best agent (Kimi K3) scores 65.82 Overall versus the expert's 85.56: agents approach expert selectivity (92.91 vs 98.92) but steer far worse (best 31.47 vs 57.75). Case studies document distinct search styles and evidence-misreading failures, such as a selected feature firing harder on English controls than on Portuguese targets. Limitations include a single base model and SAE family across only two layers, one expert feature per task, an LLM judge without human validation, a Cursor-versus-Codex harness confound, no significance tests behind close rankings, and an untested risk that public Neuronpedia features are parametrically memorized. Code and full execution traces are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier agents exhibit genuine but incomplete SAE feature-discovery ability: best Overall 65.82 (Kimi K3) vs expert baseline 85.56 across 20 tasks.
- Agents approach expert-level activation selectivity (92.91 vs 98.92) but fall far short on causal steering (best 31.47 vs 57.75), identifying causal validation as the bottleneck.
- Discovery success depends more on designing informative counterexamples and correctly interpreting measurements than on exploration volume (Sol: 115.1 texts/20.2 candidates vs Opus 5: 77.6/80.2, Fig. 4).
- Benchmark Overall correlates strongly with general capability (Spearman rho=0.800 with AA Intelligence Index v4.2), based on four matched models.
Strengths
Novel, well-targeted evaluation gap
Scores agents on interactive SAE feature discovery with causal steering validation — distinct from SAEBench (which scores SAEs, Sec. 6) and from auditing-agent work; motivated via the post-hoc-monitoring pillar of RSI (Sec. 1).
Rigorous, control-heavy scoring protocol
Three frozen metrics vs an expert reference (Eqs. 3-5); blinded GPT-4o judge over anonymized baseline/steered/random triples; norm-matched random-direction control; degeneration filter and alpha grid calibration on held-out prompts (Sec. 3.3, A.6-A.7).
Unusually deep qualitative forensics
Actionable findings beyond the leaderboard
Weaknesses
External validity restricted to one model, one SAE family, two layers
Single frozen expert feature per task
Judge and statistics thinner than the protocol
Harness confound and untested memorization risk
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SAEScientist-Bench (arXiv 2609.09113v2) introduces a benchmark scoring AI agents on interactive SAE-based mechanistic discovery: given a concept, an agent authors contrastive probe texts, navigates the 131,072-feature Gemma Scope dictionary on Gemma-2-9B-IT (layers 9 and 20), and submits one feature. Submissions are scored against a frozen Neuronpedia expert reference on three axes — Activation Rank (Eq. 3, relative to expert, range [0,200]), Activation Selectivity (AUROC vs pooled hard-negative+neutral controls, Eq. 4), and Causal Steering (net target-relevance gain over max(baseline, norm-matched random control), Eq. 5). Headline claims: frontier agents show real discovery ability but trail the expert (best Overall 65.82 vs 85.56), approach expert selectivity (92.91 vs 98.92) yet fail badly at steering (best 31.47 vs 57.75), and rigorous hypothesis testing — not search volume — predicts success. The RSI framing (post-hoc auditing as the missing pillar of autonomous R&D, Sec. 1) is well argued but aspirational: the benchmark covers post-hoc single-feature auditing only.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAEScientist-Bench?
It is a benchmark that tests whether AI agents can act as interpretability scientists: given a concept, an agent writes probe texts, searches a 131,072-feature sparse-autoencoder dictionary (Gemma Scope on Gemma-2-9B-IT, layers 9 and 20), and submits the single feature that best represents the concept. Submissions are scored on activation rank, selectivity (AUROC on held-out texts), and causal steering versus an expert reference feature.
Which AI model performed best on SAEScientist-Bench?
Kimi K3 achieved the best Overall score (65.82 vs the expert baseline's 85.56). By dimension: Claude Opus 5 led activation rank (75.35), Kimi K3 led selectivity (92.91), and Grok 4.6 led steering (31.47). Claude Opus 5 (65.41) and Claude Sonnet 5 (65.04) followed closely.
What is the paper's main finding?
Frontier agents nearly match an expert at finding features that selectively fire on a concept (92.91 vs 98.92) but achieve only about half the expert's causal steering effect (best 31.47 vs 57.75). A feature that correlates with a concept in activation space often cannot causally change generation, and agents frequently misread their own experimental measurements.
Is the code or data for SAEScientist-Bench available?
Yes. The paper states code is available on GitHub (Trae1ounG/SAEScientist). The evaluation relies on public resources — Gemma Scope SAE checkpoints and Neuronpedia feature documentation — and the authors report archiving all 600 scored episodes with probe calls, feature IDs, and generation records.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided text is a sampled excerpt with two marked gaps: mid Section 4 (protocol details between 4.1 and the Results) and Appendix A.1/A.2 (part of the task inventory table and the expert-curation workflow description); judgments touching those regions are correspondingly limited.
- Figures 1-12 appear as flattened numeric strings; figure-level interpretation relies on printed values and captions only, not visual inspection.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.