Temperature Fragility and the Conditional Benefits of Truncation Sampling
cs.CL · cs.LG
Quick verdict
关于截断采样（Truncation sampling）仅在高温度导致准确性崩溃时才有帮助的断言；这是一项严谨的 232k 生成研究，尽管每个任务仅有 50 个项目。
Summary
这项实证研究询问，截断采样器（top-p、top-k、min-p、top-nσ）在实际部署系统使用的默认温度（0.6–1.0）下，是否比单纯的温度采样更能提高准确性，因为此前报告的增益是在 1.5–3 的温度下得出的。作者在一个 llama.cpp 流水线中运行了 13 个开放权重模型（1.7B–12B，5 个系列），在 0.7、1.0 和 1.3 的温度下，对固定的 50 个 GSM8K 和 MMLU-Pro 项目进行了测试——共评定了 232,700 个生成结果——其中 10 个模型还在 8 种解码配置和 4 种量化水平下进行了测试。六个模型在 0.7 到 1.3 之间，在 MMLU-Pro 上的准确性下降了 17–38 个百分点；这种损失表现为退化的输出（被封顶或无法解析的生成结果增加了 26–78 个百分点），而不是更多的错误答案。在准确性保持稳定的情况下，在 0.7/1.0 时，没有截断采样器能超过单纯的采样（最大增益为 3.3 个百分点；Westfall–Young 上限为 9.0），但在崩溃的模型上，每个采样器在 1.3 时都能恢复准确性，且截断将崩溃延迟到了温度 2.0。崩溃在 BF16 的 Transformers 下得以复现，排除了引擎本身的问题。关键局限性：每个任务仅有 50 个项目（MMLU-Pro 子集完全是商业问题），脆弱模型是因崩溃而被选出的，每个模型的最佳采样器是事后选择的，且每个采样器仅在一个设置下运行。代码、提示词以及所有记录均已公开。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Temperature fragility is a model-level property: six of thirteen open-weight models lose 17–38 accuracy points on MMLU-Pro between temperatures 0.7 and 1.3 under plain sampling, and the loss consists of capped or unparseable degenerate output rather than more wrong answers.
- At default-range temperatures (0.7 and 1.0), no tested truncation sampler improves on plain temperature sampling for robust models — largest gain 3.3pp, with a Westfall–Young upper bound of 9.0pp on any gain.
- The gains reported for truncation samplers are recoveries from temperature-induced collapse: on fragile models every sampler improves accuracy at 1.3 (best sampler returns accuracy to within 6pp of its 0.7 level) and truncation delays collapse up to temperature 2.0, with top-nσ never collapsing through 2.0.
- The collapse replicates across engines and precisions (llama.cpp Q8 vs Transformers BF16/INT8), and a different post-training of the same base model (Hermes-3 vs Llama-3.1-8B) halves it (19 vs 38 MMLU-Pro points).
Strengths
Single-pipeline controlled design
All 232,700 generations run through one llama.cpp setup (two pinned builds, bridged and cross-checked, Table 3) with fixed items, prompts hashed per record, fixed token budgets, and lm-eval-harness-conformant parsing — directly answering the inference-backend confound raised by Masoudian et al. (2026).
Multiplicity-aware statistics matched to the claim
Item-clustered bootstrap CIs on paired differences (items resampled with repetitions together), plus a Westfall–Young joint-resampling upper bound on the largest sampler gain (Table 2: 9.0pp at 0.7/1.0 on robust models); a GEE sensitivity check reproduces estimates within 0.6pp (median 0.1).
Mechanistic decomposition of the accuracy loss
Engine and precision robustness checks
Unusually complete reproducibility package
Weaknesses
Tiny, skewed item sets underpin the central taxonomy
Recovery result is partially by construction
Capped-rate proxy is contaminated by engine artifacts
Floor effects and single-setting samplers narrow scope
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks one narrowly framed question: at the temperature defaults deployed systems actually use (0.6–1.0), do truncation samplers (top-p, top-k, min-p, top-nσ) improve accuracy over plain temperature sampling? Prior positive results — min-p on GSM8K (Nguyen et al., 2025) and top-nσ holding to T=3 (Tang et al., 2024) — came from single models at temperatures of 1.5–3. The single author answers no for robust models and reframes prior gains as recoveries from collapse, with three claims: (1) temperature fragility is a model property (6 of 13 models lose 17–38 MMLU-Pro points between T=0.7 and 1.3; the loss is capped/unparseable degeneration, not wrong answers); (2) at 0.7/1.0 no sampler beats plain sampling on robust models (largest gain 3.3pp; Westfall–Young upper bound 9.0pp); (3) every sampler improves accuracy at 1.3 on collapsing models and delays collapse to 2.0. Claims are honestly scoped to 1.7–12B open-weight instruct models, English, zero-shot, thinking-off.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do top-p or min-p sampling improve LLM accuracy at default temperatures?
According to this paper, no. On models whose accuracy stays stable between temperatures 0.7 and 1.0, none of the tested truncation samplers (top-p 0.95, top-k 40, min-p 0.05, top-nσ 1.0) reliably beat plain temperature sampling on GSM8K or MMLU-Pro: the largest observed gain was 3.3 points, and a multiple-comparison-aware statistical upper bound on any possible gain was 9.0 points.
Why do some LLMs lose accuracy at higher temperatures?
The paper finds the loss comes from degenerate output rather than worse reasoning: at temperature 1.3, fragile models start producing text that drifts into token noise and hits the token limit or never states an answer — the share of such generations rises by 26–78 points — while answers that do complete stay close to greedy-decoding accuracy on the same items.
What is temperature fragility in this paper?
A model-specific property. Six of thirteen tested open-weight models (including Llama-3.1-8B and Llama-3.2-3B) lost 17–38 accuracy points on MMLU-Pro when temperature rose from 0.7 to 1.3, while the other seven lost at most 10 points. The same base model with different post-training (Hermes-3 vs Llama-3.1-8B) lost half as much, so post-training matters.
When does min-p or top-nσ sampling actually help?
Only above the default temperature range, per this study: on models that collapse, every tested sampler recovered accuracy at temperature 1.3, truncation delayed the collapse up to temperature 2.0, and top-nσ prevented collapse entirely through 2.0. This matches the 1.5–3 temperature range where min-p and top-nσ were originally validated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 and 8 are available only as extracted labels and printed values; visual features (interval overlaps, curves) were read from the printed numbers and appendix tables rather than the rendered figures.
- The paper is a v1 arXiv preprint; peer-review status is unknown from the text.
- The blinded audit's labeling model (Claude Fable 5.1) and its prompt/key exist only in the external repository, so audit quality cannot be verified from the text itself.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.