StateSwap: Probing Support-Elimination Hidden States in Multiple-Choice Questions
Chao Gao, Haijiang Liu, Qiyuan Li, Caicai Guo, Frank van Harmelen, Jinguang Gu
cs.CL · cs.AI
Quick verdict
Claims SUP/ELIM framings yield separable, swappable [STATE] activations in LLMs; controls are thorough, but modest MMLU gains lack significance tests.
Summary
The paper asks whether two logically equivalent framings of the same multiple-choice question — asking a model to identify the correct option (SUP) versus to eliminate incorrect options (ELIM) — induce distinguishable internal representations. The authors append an untrained special token, [STATE], to minimally varied paired prompts and treat its residual-stream activation as a read/write interface. Layer-wise separability diagnostics (paired Cohen's d over feature windows, PCA, a random-label control) localize framing-sensitive structure in intermediate layers (Qwen-2.5-7B: layers 11–20; GLM-4-9B: 21–40), using only 50 calibration examples. Swapping [STATE] activations between paired prompts systematically changes predictions and raises both accuracy and cross-framing Jaccard agreement on MMLU-17 (4,607 questions) and MedQA-CH (1,015 questions): e.g., GLM-4-9B MMLU SUP accuracy 66.47→69.21; Qwen MedQA SUP 77.12→80.26. Controls include zeroing/random substitution, a final-content-token comparison, position shifts (>95% generation failures when misaligned), and token-insertion-only checks. A CAA-matched steering extension shows the dual-framing contrast yields more bounded layer-wise responses. Main limitations: the headline accuracy deltas receive no significance testing or confidence intervals (MMLU gains are as small as +0.9 points), the membership-based accuracy metric is lenient, and scope is restricted to greedy decoding, four-option MCQs, and two 7–9B open models. Code is released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SUP and ELIM framings of the same MCQ induce linearly separable [STATE] activations concentrated in intermediate layers (Qwen-2.5-7B: 11–20; GLM-4-9B: 21–40).
- Substituting these activations between paired framings systematically changes predictions and increases both accuracy and cross-framing Jaccard agreement in the evaluated settings.
- The behavioral effects are specific to the [STATE] interface, the substitution content, and the canonical final position, distinguishing them from token insertion, content-token swaps, and unstructured perturbations.
- Mean-difference steering directions from the SUP–ELIM contrast produce more bounded layer-wise responses than matched CAA directions under the evaluated protocol.
Strengths
Intervention design controls decoding and insertion confounds
All runs use greedy decoding (Section 4.2), and the BASE condition already contains the [STATE]-augmented prompt, so token-insertion effects cancel within the BASE→SUB contrast; insertion alone shifts MedQA latent accuracy only from 80.00%→79.80% (GLM) and 86.90%→87.19% (Qwen) in Table E.1.
Control battery is unusually complete
Random-label separability control with 10 permutations and ±2σ bands (Fig 4), zeroing and random substitution (Fig 6b, Table E.2), final-content-token substitution (Fig 6a, Table E.3), position misalignment where >95% of generations fail (Section E.3), and random-[STATE]-initialization robustness across five seed pairs (Table B.2, similarities ≈0.99999999).
Window localization is decoupled from test performance
Claims are deliberately hedged
Cross-model, cross-lingual consistency
Weaknesses
No inferential statistics on headline results
The asymmetry claim is thin in one cell
Bidirectional improvement is left unexplained
Lenient accuracy and set-level agreement metrics
Narrow model scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
StateSwap (arXiv 2609.01081v2) asks whether two logically equivalent framings of the same MCQ — SUP (identify the correct option) and ELIM (eliminate the incorrect options) — induce distinguishable internal representations in LLMs, and whether those representations are causally potent. The method appends an untrained [STATE] token to aligned paired prompts, treats its residual-stream state as a read/write interface, localizes an intermediate-layer window via separability diagnostics, then swaps the full d-dimensional state between framings within that window (Sections 3.1–3.3). Claims are appropriately scoped: separability, behavioral relevance under substitution, and — for the steering extension — bounded layer-wise responses explicitly qualified as 'under the evaluated protocol.'
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is StateSwap?
StateSwap is a training-free interpretability method that appends an untrained special token, [STATE], to multiple-choice prompts and swaps its intermediate-layer residual activations between paired prompts that ask the same question in two framings — find the correct option (SUP) versus eliminate the incorrect options (ELIM) — to test whether the two framings induce distinct, behaviorally relevant hidden states.
Does swapping [STATE] activations improve LLM accuracy?
In the paper's experiments, modestly: both framing-specific accuracy and cross-framing Jaccard agreement rise under substitution, e.g., GLM-4-9B MMLU SUP accuracy 66.47→69.21 and Qwen MedQA SUP 77.12→80.26, though Qwen's MMLU gains are only about +1 point. All results use deterministic greedy decoding on two 7–9B open models, and no statistical significance tests are reported for the deltas.
Which models and datasets were used?
Qwen-2.5-7B-Instruct and GLM-4-9B, evaluated zero-shot on a 17-subject reasoning-focused subset of MMLU (4,607 English questions) and MedQA-CH (1,015 Chinese medical licensing questions), for 5,622 test questions total; 50 training-split examples were used only to localize the intervention layers.
Is code available for StateSwap?
Yes. The paper states that code is available at the GitHub repository Cha0Ga0/SWAPSTATE, and all evaluated models and benchmarks are publicly available, which supports independent reproduction given GPU resources.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal data (PCA point clouds in Figures 3/D.1, separability heatmaps in D.2, transition matrices in Figure 5, steering curves in Figure 7) are not present in the extraction; assessments of those rest on captions and numbers quoted in the text.
- Figure 1's embedded panel text is interleaved with body text in the extraction, so its panel layout could not be reconstructed; its caption is intact and quotable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.