Encoded Early, Used Late: Where Transformers Begin to Act on an Inferred Partner's Expertise
cs.AI · cs.CL
Quick verdict
Finds partner expertise decodable at layer 8 but causal only past L36 in Llama-3.3-70B; clean patching design, yet one model and 56 instances cap generality.
Summary
The paper asks whether the known gap between where an attribute is linearly decodable and where it causally shapes output extends to attributes a model must infer across a conversation rather than read from the input. The authors build ExpertCollab, 28 synthetic twelve-turn research-planning dialogues between personas at four expertise levels played by Llama-3.3-70B, and analyze Llama-3.3-70B-Instruct activations at the generation boundary. Linear probes find partner expertise most decodable early (0.79 accuracy at layer 8), decaying to near chance by layer 40, while a stated control attribute (the speaker's own level) remains decodable throughout. Counterfactual patching on 48 cross-level pairs with a fixed last-layer readout shows an effect of 0.01 at the peak-decoding layer but near-complete propagation above 0.90 from layer 40; norm-matched random vectors and a probe-free cosine diagnostic mark the same transition. The authors conclude that inferred relational attributes are represented well before they become causally active, aligning with the mid-network transition reported by Lad et al. Main limitations: a single model, a small synthetic same-model corpus (56 pooled instances) whose per-cell estimates the authors themselves call unreliable, and architecture-dependent layer boundaries.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Partner expertise in Llama-3.3-70B is most decodable at layer 8 (0.79 accuracy) and decays to near chance by layer 40, whereas a stated control attribute (own level, peak 1.00 at layer 0) stays decodable throughout.
- Injecting the expertise activation difference at the peak-decoding layer leaves a fixed layer-76 readout essentially unchanged (normalized effect 0.01), while injection past the network midpoint propagates almost completely (>0.90 from layer 40).
- A norm-matched random control and a probe-free cosine diagnostic independently locate the same early-to-mid transition, indicating a structural property rather than a probe artifact.
- The depth gap between peak decodability and causal onset grows with how much an attribute must be inferred rather than stated, proposed as a falsifiable hypothesis (Appendix F).
Strengths
Causal framing with converging controls
Counterfactual patching with a fixed layer-76 readout is triangulated by four checks: calibration control (patch=readout averages 1.01), same-level null (exactly 0), a norm-matched random vector showing no comparable rise, and a probe-free cosine diagnostic that marks the same early-layer transition (figs. 3-4), arguing against a probe artifact.
Disciplined corpus construction
Personas explicitly forbid credential statements, an LLM leakage detector screens all 28 transcripts (Appendix B.5), and a five-feature lexical baseline reaches only 0.32 accuracy versus 0.79 for the neural probe (chance 0.25, Appendix E.1), showing the early signal exceeds surface statistics.
Verbatim reproducibility documentation
Honest scoping and a falsifiable hypothesis
Weaknesses
Very small effective sample
Same-model generation/probing loop
In-sample confusion matrix presented as evidence
20.1% of patching cells masked without sensitivity analysis
Additive single-position patching limits interpretation of the early null
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether the 'decodable-but-unused' phenomenon — established for attributes stated directly in the input (Huang & Chang 2025; Arad et al. 2025; Choi et al. 2025) — extends to attributes a model must infer across an interaction. The target attribute is a dialogue partner's expertise, operationalized in ExpertCollab: 28 twelve-turn research-planning dialogues between personas at four levels (HS/JR/R/PROF) across four scenarios, both roles played by Llama-3.3-70B. Central claims: (1) partner expertise peaks in decodability early (0.79 at layer 8, near chance by layer 40) while a stated control (own level, peak 1.00 at layer 0, >0.60 everywhere) stays decodable throughout; (2) counterfactual patching with a fixed layer-76 readout yields a normalized effect of 0.01 at the peak-decoding layer versus >0.90 from layer 40; (3) a norm-matched random control and a probe-free cosine diagnostic mark the same transition; (4) Appendix F states a falsifiable hypothesis tying decode-to-causal depth to inference demands. Claims are appropriately scoped, and the Scope section is unusually candid.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'encoded early, used late' mean in this paper?
Transformer layers can make information linearly readable (via probes) long before that information causally influences the model's output. The paper shows this for an inferred attribute — a dialogue partner's expertise — in Llama-3.3-70B: it is most decodable at layer 8, but injecting it there barely changes a late-layer readout, while injecting it past the network midpoint propagates almost completely.
What is the ExpertCollab dataset?
A released corpus of 28 synthetic twelve-turn research-planning dialogues between two AI-played personas drawn from four expertise levels (high-school student, junior researcher, researcher, professor) across four research topics. Personas never state credentials; expertise must be inferred from how they write, and an automated leakage check confirmed no explicit credential statements.
Which model was studied and how?
Llama-3.3-70B-Instruct (80 layers), accessed via NDIF and nnsight for activation extraction and patching. The dialogues were generated by the Together AI -Turbo serving variant of the same model, which the authors flag as a limitation since the probed representations may partly reflect the model recognizing its own generation patterns.
Does this mean linear probes are useless for interpretability?
No — the paper's point is that decodability alone does not establish causal function. The most readable layer (8) was causally silent for this attribute, so probes should be paired with interventions such as activation patching before drawing conclusions about where a model uses a representation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only through extracted captions and axis labels; curve values beyond those quoted in the prose (e.g., 0.79 at layer 8, 0.86 at layer 36, cosine saturation ~0.40 by layer 32) could not be verified against the plotted data.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.