arXiv paper2026-09-04· NLP / mechanistic interpretability of large language models· empirical study

Beneath the Surface of Chains-of-Thought: A Mechanistic Interpretation of Reasoning Operations in LLMs

Q: What is the main finding of this paper?

That hidden states of reasoning LLMs (Qwen2.5-7B, Qwen3-8B, Gemma4-31B) encode which reasoning operation — extraction, decomposition, deduction, recall, arithmetic computation, etc. — a chain-of-thought span is performing. Linear probes separate these operations in held-out data with AUROC often above 0.9, peaking in middle layers, and the separation survives controls for surface wording, position, and notation. The paper is diagnostic: it shows representational structure, not that the models' latent computation literally follows the taxonomy.

Q: How were the reasoning-operation labels created and how reliable are they?

GPT-5 segmented and labeled spans using a 28-type taxonomy derived from Pólya's How to Solve It, with the full prompt published in the appendix. Seven human annotators independently relabeled 84 sampled spans: humans reached a majority label for 96.4% of spans (Fleiss' κ=0.666), and GPT-5 matched the human-majority label in 76.2% of cases (Cohen's κ=0.715). Span boundaries were not human-validated, so labels should be read as approximate.

Q: Does this prove the model 'really reasons' the way the text says?

No. The authors explicitly treat the taxonomy as an operational framework for textually expressed functions, not a cognitive ground truth. What they show is that text-annotated operations are linearly separable in hidden states, become context-dependent in middle layers, and depend causally on preceding context via attention-masking — a representational correspondence, not a mechanistic circuit account.

Q: Is code and data available for this paper?

The paper states that code and project materials are available at github.com/naver-ai/beneath-cot, and the appendix publishes annotation prompts, model checkpoints (public Hugging Face models), datasets (DAPO-MATH-17K, TheoremQA, GPQA-Diamond, MATH-500), and generation parameters. Trace generation uses no fixed random seed, so the exact trace corpus is not bitwise reproducible, though the analysis pipeline is seeded.

Seogyeong Jeong, Jaehui Hwang, Dongyoon Han, Geonmo Gu, Alice Oh, Taekyung Kim

View original paper ↗arXiv:2609.04753

cs.CL

Quick verdict

CoT reasoning operations form separable middle-layer geometry, well controlled for confounds; GPT-5 label noise and split leakage temper mechanistic claims.

Summary

The paper asks whether the functional operations visible in chain-of-thought text — extraction, decomposition, deduction, recall, and others from a Pólya-derived taxonomy — have distinct geometric structure inside reasoning LLMs. GPT-5 annotates operation spans in ~500–700 correct traces per model/dataset (DAPO-MATH-17K, TheoremQA; Qwen2.5-7B, Qwen3-8B, Gemma4-31B), validated by seven humans on 84 spans (96.4% majority agreement, Fleiss κ=0.666; GPT-5 exact match 76.2%). Probes (L2-norm, PCA-128, LDA, one-vs-rest centroid directions) separate all eight operations in held-out representations, peaking in middle layers. The result survives text-only, position-only, lexically-matched, competing-vocabulary, digit-density, and no-LDA controls, and transfers to Llama-3-8B (macro AUROC 0.958) and, without retraining, to GPQA-Diamond (0.938) and MATH-500 (0.948). Middle-layer signals become distributed across spans, identical surface tokens separate by operation, and masking preceding attention reduces operation alignment at chunk onset (Δ=−0.436, p<0.0001) while a non-reasoning discourse control is unaffected. Probes trained on correct traces still detect operations in factual-error spans (macro AUROC 0.920–0.955) with modest attenuation. Key limitations: annotation validated on only 84 spans without boundary checks, a train/test split with 14/608 overlapping questions for Qwen3-8B, and a purely diagnostic scope; code is public.

Review complete

Assessment at a glance

NLP / mechanistic interpretability of large language models

risk flags identified

● high 0

● medium 2

● low 3

Reproducibility readinessmedium

Code & data availableyes

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

5 strengths · 5 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Instances of eight Pólya-derived reasoning operations are linearly separable in held-out hidden representations across Qwen2.5-7B, Qwen3-8B, and Gemma4-31B, with separability peaking in middle layers.
This separability is not explained by lexical identity, trace position, digit/formula density, or the supervised LDA projection.
Operation-alignment signals become distributed across spans in middle layers, and identical surface tokens acquire operation-dependent representations from surrounding context.
Masking attention to preceding context causally reduces operation-aligned representations at chunk onset, an effect absent for a non-reasoning discourse control.
Probes trained on correct traces still detect operation identity in factual-error spans (macro AUROC 0.920–0.955), showing a partial dissociation between operation identity and execution correctness.

Strengths

Unusually thorough confound-control battery
Beyond the standard random-label baseline, §3.2.2–3.2.3 add text-only BoW/TF-IDF baselines (Table I: hidden probe +0.041–0.097 macro AUROC), lexically-matched pairs (Table J: CI excludes zero for 5/8 operations), competing-vocabulary subsets (Table K: 0.919/0.848 Qwen3-8B), digit-density restriction (Table L), position-only classifiers (Table G) and position-stratified frozen probes (Table H: 0.916–0.973), plus a no-LDA PCA-only ablation (Table E). Most probing papers run one or two of these.
Multi-model, multi-task generalization with frozen probes
Three model families in the main analysis (Qwen2.5-7B, Qwen3-8B, Gemma4-31B), a replication on Llama-3-8B (macro AUROC/AUPRC 0.958/0.840, §C.1), and cross-task transfer of Qwen3-8B probes without retraining to GPQA-Diamond (0.938/0.764) and MATH-500 (0.948/0.799, §C.2, Table C).
Causal intervention with a genuine specificity control
Transparent annotation validation rather than hidden LLM labeling
Self-flagged exploratory analyses

Weaknesses

Label noise is the binding constraint and is thinly validated
Train/test split is not question-disjoint
Correctness filtering depends on an unvalidated LLM judge
'Mechanistic' framing exceeds the diagnostic evidence
Entropy-centered windowing is an un-ablated selection step

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

This paper asks whether functionally distinct reasoning operations expressed in chain-of-thought text — Extraction, Direct mapping, Decomposition, Recall, Deduction, Algebraic manipulation, Arithmetic computation, Final answer (8 of a 28-type taxonomy built on Pólya's four stages) — correspond to separable geometric structure in the hidden states of reasoning LLMs. Four claims are tested: (1) operations are separable in held-out representations, peaking in middle layers (§3.2); (2) separability is not a lexical or positional artifact (§3.2.2–3.2.3); (3) operation signals become span-distributed in middle layers and contextualize identical surface tokens (§3.3); (4) preceding context causally shapes operation representations at chunk onset (§3.4), and operation geometry persists, attenuated, under factual errors (§3.5). The authors are explicit that the taxonomy operationalizes textually expressed functions, not latent cognition (§2, §6).

Methodology Assessment

The pipeline is deliberately simple: GPT-5 annotates non-overlapping spans (with a published prompt, E.2) in 500–700 correct traces per model/dataset on DAPO-MATH-17K and TheoremQA; span representations (middle-token by default) are L2-normalized, PCA-reduced to 128 dims, and projected by supervised LDA into a 7-dim space where one-vs-rest centroid-difference directions score held-out spans. Simplicity is a virtue here — the claims are about linear separability, not probe capacity. The confound battery is the paper's strongest asset: text-only BoW/TF-IDF baselines los

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 5 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is the main finding of this paper?

That hidden states of reasoning LLMs (Qwen2.5-7B, Qwen3-8B, Gemma4-31B) encode which reasoning operation — extraction, decomposition, deduction, recall, arithmetic computation, etc. — a chain-of-thought span is performing. Linear probes separate these operations in held-out data with AUROC often above 0.9, peaking in middle layers, and the separation survives controls for surface wording, position, and notation. The paper is diagnostic: it shows representational structure, not that the models' latent computation literally follows the taxonomy.

How were the reasoning-operation labels created and how reliable are they?

GPT-5 segmented and labeled spans using a 28-type taxonomy derived from Pólya's How to Solve It, with the full prompt published in the appendix. Seven human annotators independently relabeled 84 sampled spans: humans reached a majority label for 96.4% of spans (Fleiss' κ=0.666), and GPT-5 matched the human-majority label in 76.2% of cases (Cohen's κ=0.715). Span boundaries were not human-validated, so labels should be read as approximate.

Does this prove the model 'really reasons' the way the text says?

No. The authors explicitly treat the taxonomy as an operational framework for textually expressed functions, not a cognitive ground truth. What they show is that text-annotated operations are linearly separable in hidden states, become context-dependent in middle layers, and depend causally on preceding context via attention-masking — a representational correspondence, not a mechanistic circuit account.

Is code and data available for this paper?

The paper states that code and project materials are available at github.com/naver-ai/beneath-cot, and the appendix publishes annotation prompts, model checkpoints (public Hugging Face models), datasets (DAPO-MATH-17K, TheoremQA, GPQA-Diamond, MATH-500), and generation parameters. Trace generation uses no fixed random seed, so the exact trace corpus is not bitwise reproducible, though the analysis pipeline is seeded.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

The text is a sampled excerpt with three marked sampling gaps (part of §3.2.2's opening, a block of references, and parts of Appendix D.1/F.1 narrative); every number cited in this review was verified against tables or in-text summaries that survived sampling, but sections inside the gaps could not be directly inspected.
Figures appear only as captions in the extraction; visual claims (e.g., Figure 4 variance trends, Figure 5 token separation, Figure 6 masking effect) were assessed from captions, surrounding text, and the appendix statistics rather than from the images.
Table C's header row was lost at a sampling-gap boundary; its macro values (GPQA-Diamond 0.938/0.764, MATH-500 0.948/0.799) were cross-verified against the matching in-text figures in §3.2.3.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper