arXiv paper2026-09-21· machine learning — mechanistic interpretability of transformers· method paper

Attention as a Routing Graph: Live Circuit Extraction from a Single Forward Pass

Q: What is live circuit extraction from a single forward pass?

It is a cheap way to propose a candidate circuit inside a transformer: run the model once, treat attention links as a routing graph with edges scored by attention weight times value-vector norm, then walk backward from the answer token (top 15 edges per hop, 4 hops) to keep a small edge set. No training or iterative search is involved.

Q: Does the extracted subgraph actually matter causally?

In the paper's tests, yes relative to its control: zeroing those attention edges hurts task scores (induction target logit, IOI name logit-difference) far more than zeroing ten random same-size edge sets, across GPT-2 Small, GPT-2 Medium and Pythia-410M on induction and IOI with n=100 per cell and Wilcoxon p<10⁻¹⁰. The authors caution that part of this advantage comes from removing more total attention mass, not routing alone.

Q: How much cheaper is this than activation patching?

Extraction needs one forward pass and its built-in check about a dozen; a one-pass-per-head patch sweep on GPT-2 Small needs 145 forwards, roughly two orders of magnitude more compute.

Q: Does it recover the known induction circuit?

Only partially. The walk almost always catches early known induction heads (L4H11, L5H5, L5H8) but frequently misses later members of the textbook circuit; the paper positions the output as a cheap causal sketch, not a complete atlas.

Ash Manvi, Samreena Tajreen

View original paper ↗arXiv:2609.25285

cs.AI · cs.LG

Quick verdict

One-forward-pass attention routing sketch shows real causal signal on induction/IOI, but uniform random controls confound attention mass with routing effect.

Summary

The paper proposes extracting candidate circuits from a transformer in a single forward pass: attention maps are read as a routing graph whose edges are scored by attention weight times squared value-vector norm, then a backward walk from the answer token keeps the top-15 incoming edges over four hops. The kept subgraph S is validated causally by zeroing its attention weights (with renormalization) and comparing task-score drops against ten random edge sets of equal size, on induction and IOI across GPT-2 Small, GPT-2 Medium, and Pythia-410M with n=100 prompts per cell. All six cells show positive gaps (e.g., +4.33 GPT-2 Small induction, +5.63 Pythia IOI), 95% bootstrap CIs excluding zero, and Wilcoxon p<10⁻¹⁰; per-example fail rates reach 19/100 on Medium induction. Extraction costs one forward pass versus 145 for a head-by-head patch sweep. The recovered subgraphs reliably contain early known induction heads (L4H11, L5H5, L5H8) but often miss later circuit members. The authors candidly flag the central confound: their uniform random control removes far less attention mass than S, so part of the gap is a mass effect of unmeasured size, and they test only two synthetic tasks on small models without alternative intervention styles.

Review complete

Assessment at a glance

machine learning — mechanistic interpretability of transformers

risk flags identified

● high 0

● medium 2

● low 1

Reproducibility readinesshigh

Code & data availableyes

Review confidencehigh

AI text signal0/12 passages flagged

Deep research viableCode-reproducible

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 12 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

A backward walk over edges scored by attention weight times squared value-norm, computed in one forward pass, yields a small subgraph with real causal signal.
On induction and IOI across GPT-2 Small, GPT-2 Medium, and Pythia-410M (n=100 per cell), ablating the extracted set hurts far more than ablating ten size-matched random sets: all six cells show positive gaps (+1.51 to +5.63), bootstrap CIs excluding zero, and Wilcoxon p<10⁻¹⁰.
Extraction costs one forward pass versus about 145 for a head-by-head patch sweep on GPT-2 Small, roughly two orders of magnitude cheaper.
The extracted subgraphs almost always contain early known induction heads (L4H11, L5H5, L5H8) but often miss later circuit members — partial, not complete, recovery.
Part of each measured gap is an attention-mass effect of unmeasured size because the random control is not mass-matched.

Strengths

Causal check is built into the method, not an afterthought
Every extraction is tested against ten size-matched random edge sets under paired statistics; all six cells in Table 1 report Wilcoxon p<10⁻¹⁰ and 95% bootstrap CIs excluding zero.
Above-average statistical hygiene for a sketch method
n=100 prompts per cell, paired per-example gaps g_i, bootstrap CIs, ±1 SD reported per condition, and per-cell fail counts (§3.1, §3.4) instead of cherry-picked successes.
Honest claim scoping and a self-identified confound
Concrete, conservative cost accounting
Sensitivity probes cover the important knobs

Weaknesses

Headline control conflates ablated mass with routing quality
Single, non-standard intervention style
Narrow external validity
Effects measured only against a near-inert control
Sensitivity analysis limited to one cell

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

The paper (arXiv 2609.25285, cs.AI/cs.LG, v1 of 21 Sep 2026) proposes a deliberately minimal method for candidate circuit discovery. In a single forward pass, attention links are read as a routing graph whose edges are scored by attention weight times squared value-norm (s = A[l,h,i,j] * ||V[l,h,j]||^2, §2) — explicitly positioned as an edge-level cousin of Kobayashi et al.'s vector-norm analysis rather than a new attribution theory. A backward walk from the answer position (top-15 incoming edges, four hops, BOS sink banned, self-loops skipped) extracts a small edge set S. The central claim is modest and clearly scoped: S carries real causal signal on two tasks with textbook circuits — induction and IOI — across GPT-2 Small, GPT-2 Medium, and Pythia-410M, at a fraction of the cost of patch sweeps. The authors explicitly disclaim a complete circuit atlas (abstract, §4). That framing — sketch plus check, not atlas — is the paper's strongest design decision and most of its defense against overclaiming.

Methodology Assessment

The pipeline is transparent. Extraction costs one forward pass. Verification zeroes the attention weights of S (renormalizing what remains), measures the drop in a task metric (target logit for induction, name logit-difference for IOI), and compares against ten uniform random edge sets of size |S|; ten controls per example is a reasonable floor. The method keeps head identity, which induction requires, and its score is grounded in prior work (Kobayashi et al. [8]; the authors correctly invoke Abnar-Zuidema rollout [1] as the lesson that raw attention alone is weak).

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
3 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is live circuit extraction from a single forward pass?

It is a cheap way to propose a candidate circuit inside a transformer: run the model once, treat attention links as a routing graph with edges scored by attention weight times value-vector norm, then walk backward from the answer token (top 15 edges per hop, 4 hops) to keep a small edge set. No training or iterative search is involved.

Does the extracted subgraph actually matter causally?

In the paper's tests, yes relative to its control: zeroing those attention edges hurts task scores (induction target logit, IOI name logit-difference) far more than zeroing ten random same-size edge sets, across GPT-2 Small, GPT-2 Medium and Pythia-410M on induction and IOI with n=100 per cell and Wilcoxon p<10⁻¹⁰. The authors caution that part of this advantage comes from removing more total attention mass, not routing alone.

How much cheaper is this than activation patching?

Extraction needs one forward pass and its built-in check about a dozen; a one-pass-per-head patch sweep on GPT-2 Small needs 145 forwards, roughly two orders of magnitude more compute.

Does it recover the known induction circuit?

Only partially. The walk almost always catches early known induction heads (L4H11, L5H5, L5H8) but frequently misses later members of the textbook circuit; the paper positions the output as a cheap causal sketch, not a complete atlas.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figures 1–4 appear only as captions in the extracted text; the plotted content could not be inspected.
Prompt templates, filtering rules, and per-setting artifacts live in the linked repository, not the paper body; the review relies solely on the embedded text.
Single-version arXiv text (v1, 21 Sep 2026) reviewed offline; no external cross-checks were performed.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper