Every Token Leaves a Ripple in the Stream of Thought: Eliciting Model-Internal Token Saliency for Chain-of-Thought Compression
Tianyi Zhao, Yinhan He, Wendy Zheng, Chen Chen
cs.CL
Quick verdict
MIST scores CoT tokens via necessity/sufficiency Taylor linearizations; wins over TokenSkip and heuristics on 4 benchmarks, but no significance tests or code.
Summary
The paper proposes MIST, a token-level chain-of-thought (CoT) compression method that derives token importance from the target model's internal computation rather than external scorers. Each reasoning token is scored along two axes: necessity — the drop in ground-truth answer likelihood when the token's residual-stream state is zeroed — and sufficiency — the gain when that state alone is patched into a no-chain forward pass. Both interventions are approximated with first-order Taylor expansions, so all token scores come from two backward passes per chain regardless of length, are weighted across layers by logit-lens projections, and are combined with a mixing coefficient fixed at 0.6. On GSM8K, MATH-500, MMLU-Pro, and BBH with Qwen2.5-1.5B/7B, Llama-3.1-8B, and Mistral-7B-Instruct, MIST outperforms TokenSkip, GoGI, perplexity, attention-based, and trivial baselines on average: accuracy drops of only 2.4 pp (Qwen2.5-7B, GSM8K) and 1.3 pp (Mistral-7B, GSM8K) versus full CoT, plus gains on MATH-500 (+5.3 pp, Qwen2.5-1.5B) and BBH (+3.7 pp, Llama-3.1-8B). Appendix D provides Taylor remainder bounds and proxy-versus-exact agreement (necessity top-0.5 overlap 0.81). Main limitations: no statistical significance testing or seed variance is reported; accuracy is compared across methods at unmatched realized compression rates; ablations and sensitivity analyses cover single model-dataset pairs; supporting analyses use only 8-100 chains; and no explicit code release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Token importance for CoT compression should be and can be derived from the target model's own answer computation, operationalized as residual-stream interventions along a necessity axis (erasure) and a sufficiency axis (patching).
- First-order Taylor linearization reduces both axes to two backward passes per chain independent of chain length, with high fidelity to exact interventions (necessity top-0.5 agreement 0.81, Table 5).
- Necessity and sufficiency capture largely distinct signals (Spearman ρ=−0.07, top-30% overlap 0.28), and their logit-lens-weighted combination outperforms either axis alone and all single-layer simplifications.
- MIST consistently outperforms TokenSkip, GoGI, perplexity, attention-rollout, H2O, and trivial baselines across GSM8K, MATH-500, MMLU-Pro, and BBH with four instruction-tuned models, with the smallest accuracy loss at comparable or better compression (e.g., 21.3% compression at −2.4 pp on GSM8K/Qwen2.5-7B).
Strengths
Principled, formally grounded saliency definition
Necessity (Eq. 2) and sufficiency (Eq. 3) are operationalized as residual-stream interventions rooted in the activation-patching literature; Appendix D gives Lagrange-form Taylor remainders with Hessian operator-norm bounds (Eqs. 19, 26, 28) rather than hand-waving the linearization.
The authors empirically audit their own approximation
D.5 measures per-layer first-order tightness within ~10% of y=ε, and Table 5 reports proxy-vs-exact intervention top-γ agreement (necessity 0.72/0.81, sufficiency 0.68/0.76 at γ=0.3/0.5, well above chance) — rare honesty in this literature.
Efficiency claim is real and validated
Well-controlled comparison protocol
Analysis depth beyond the main table
Weaknesses
No statistical rigor anywhere
Cross-method accuracy compared at unmatched compression
Supporting evidence rests on tiny samples and single settings
Gains over full CoT are unexplained
No code or artifact link
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MIST (arXiv:2608.31066, UVA) targets token-level CoT compression in the TokenSkip setting: prune full reasoning traces into shorter SFT supervision so the fine-tuned model generates shorter chains at inference. The core reframing is that token importance should come from the target model's own answer computation, operationalized on the residual stream via two interventions — necessity (zero the token's state, measure answer log-likelihood drop, Eq. 2) and sufficiency (patch that state into a prompt-only pass at the final position, measure gain, Eq. 3). The paper claims: (i) the axes are complementary (Spearman ρ=−0.07, top-30% overlap 0.28, Fig. 2c, on 100 MATH chains with Qwen2.5-1.5B); (ii) first-order Taylor linearization collapses the O(T) intervention cost to two backward passes per chain (§3.2, D.4); (iii) the unified score beats the external scorer TokenSkip and internal heuristics across four benchmarks and four models (Tables 1, 6; Figs. 3-5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MIST (Model-Internal Saliency for Token-level CoT compression)?
MIST is a method for shortening chain-of-thought (CoT) reasoning traces. It scores every token of a reasoning chain by how much the language model's own internal computation depends on it: necessity measures how much answer probability drops if the token's internal residual-stream state is erased, and sufficiency measures how much answer probability is recovered if only that token's state is injected into a forward pass without the chain. The top-ranked tokens are kept as shorter training supervision for fine-tuning the model to reason more briefly.
How much shorter does reasoning get, and how much faster is inference?
In the paper's experiments, models fine-tuned on MIST-compressed supervision generate 9.3% to 21.3% fewer tokens than full-chain baselines on GSM8K and MATH-500. Directly measured decoding latency on GSM8K improves by 1.22x (Qwen2.5-1.5B), 1.30x (Llama-3.1-8B), and 1.43x (Mistral-7B) at the most aggressive trained budget, with small accuracy cost on math tasks (e.g., 2.4 points on GSM8K with Qwen2.5-7B).
Does MIST work with closed-source models accessed through an API?
No. MIST requires white-box access to the target model: it reads residual-stream activations at every layer and computes gradients of the answer log-likelihood inside the model. The authors state in the Limitations section that it does not apply to black-box API-only systems, and that it has so far only been tested on 1.5B-8B open instruction-tuned models.
Can compressing chain-of-thought actually improve accuracy?
In this paper's experiments, yes, in some settings: fine-tuning on MIST-compressed chains slightly beat the uncompressed full-chain baseline on MATH-500 (+5.3 points with Qwen2.5-1.5B-Instruct) and BIG-Bench Hard (+3.7 points with Llama-3.1-8B-Instruct), consistent with shorter supervision acting as a regularizer. On MMLU-Pro, however, all compression methods still lost accuracy relative to full chains (MIST by 1.0-2.8 points), so the effect is not universal.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-11 are vector plots whose underlying data values are not recoverable from the text extraction; figure-level judgments (e.g., Fig. 3 curves, Uniform baseline values) rely on the authors' in-text descriptions and axis labels rather than raw numbers.
- Inline mathematics are partially garbled by PDF extraction (e.g., the h→0 erasure notation in Eqs. 1-3, subscripts in Eqs. 6-8); formulas were interpreted from context and from the cleaner Appendix D derivations, which restate both terms.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.