Greedy Decoding Is Not Precision-Invariant: Cross-Precision Output Divergence in LLM Inference
Gaoyuan Du, Anam Nawaz Khan, Rex Zhou, Xiaoyang Liu, Deepayan Chakrabarti, Fnu Suya, Xueping Li
cs.LG · cs.AI
Quick verdict
Shows BF16 vs FP16 greedy decoding diverges on 49–100% of prompts; margin-gated FP32 lm_head fix adds 22–36pp (A10G). Well-controlled, honestly bounded.
Summary
The paper shows that greedy decoding in LLMs is not precision-invariant: the same model, prompt, and decoding configuration produce different token sequences under BF16 versus FP16 on identical hardware. Across six models (1.1B–7B, four families) and three benchmarks, 49–100% of prompts diverge, and a single token flip cascades into trajectory-level divergence. Through divergence-conditioned tracing, layer grafting, KV-cache grafting, and logit-lens traceback, the authors localize divergence to small top-two logit margins at the lm_head rather than accumulated body error, with the body-error null confirmed by equivalence testing. This mechanism yields five pre-stated predictions (e.g., integer quantization has zero effect; broader FP32 scope hurts), all confirmed. The proposed fix—selective FP32 lm_head recomputation gated at margin threshold τ=10⁻³—raises exact agreement by +22–36pp on A10G (+12–21pp on L4/A100) at <4% latency overhead for batch size ≤4. Limitations: effects vanish at bs≥8 and under end-to-end FP8, magnitudes are hardware-dependent (A10G favorable), the training-time precision-stability hypothesis is unvalidated, interventions are untested beyond 7B, and headline cells use n=100.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Greedy decoding is not precision-invariant: the same model, prompt, and decoding configuration produce different outputs in BF16 versus FP16 on identical hardware, with 49–100% of prompts diverging across six tested models (1.1B–7B) and three benchmarks.
- Divergence is determined primarily by the top-two logit margin at the lm_head relative to the directional perturbation, not by accumulated transformer-body error — a null formally established via equivalence testing.
- Margin-gated selective FP32 lm_head recomputation (τ=10⁻³, firing on 0.7–1.4% of steps) raises exact agreement by +22–36pp on A10G (+12–21pp on L4/A100) at <4% latency overhead for batch size ≤4 greedy decoding.
- The intervention's benefit vanishes when body-originated error dominates: at batch size ≥8, under end-to-end FP8, and on BF16-saturated Qwen models, and effectiveness is hypothesized (not shown) to track training-time precision stability.
Strengths
Convergent causal localization of the divergence mechanism
Four independent experiments — divergence-conditioned trace (Table 7), layer grafting (Table 8: flat ~36% fix rate across body layers vs 65–71% at layer 21/RMSNorm), KV-cache grafting (Table 9: asymmetric 47.5% vs 35.6%), and logit-lens traceback (Table 10: top-2 agreement ≥93% at every layer, top-1 split only at the head) — triangulate the flip to the lm_head; the body-error null is formally established by TOST equivalence (p=0.014 TinyLlama, p=0.006 Qwen), not mere failure to reject.
Prediction-then-test intervention design
The error analysis pre-states five outcomes (Intervention A zero effect; B≈C; top-K sufficient for all K≥2; scope extension degrades EAR 63%→44%; FP32 oracle 100%), and Table 3 plus Appendix IX confirm all five, making the mechanism falsifiable rather than post hoc.
Unusually honest boundary mapping
Statistical reporting above the bar for systems work
Weaknesses
Headline magnitudes come from the most favorable GPU
The organizing hypothesis is unvalidated
Small per-cell samples and underpowered quality tests
Coverage gaps for deployment claims
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper advances four claims: (1) BF16-vs-FP16 greedy decoding diverges on 49–100% of prompts across six models (1.1B–7B, four families, plus a 12.2B divergence-only probe) and three benchmarks, with the FP32 oracle (Table 3, 100% agreement) confirming all divergence is precision-origin; (2) divergence is a top-two logit margin event at the lm_head (σz ≈ 0.026 on TinyLlama), not an error-magnitude event in the 22-layer body; (3) five pre-stated predictions from the error-propagation analysis are all confirmed; (4) a margin-gated FP32 lm_head recomputation (τ=10⁻³, firing on 0.7–1.4% of steps) delivers +22–36pp exact agreement on A10G at <4% latency, bounded to bs≤4 greedy decoding. Scope is honestly circumscribed — the abstract itself calls the method a partial mitigation, not a determinism guarantee.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does the same LLM give different answers in BF16 vs FP16?
Greedy decoding picks the highest-logit token. BF16 (7-bit mantissa) and FP16 (10-bit mantissa) round intermediate arithmetic differently, and when the top two candidate logits are nearly tied, that tiny rounding difference reverses the winner; in autoregressive generation one flipped token cascades into a completely different output. The paper measures this on 49–100% of prompts across six 1.1B–7B models and three benchmarks.
What is the proposed fix and how much does it cost?
Recompute the final lm_head projection in FP32, but only at decoding steps where the top-two logit margin falls below a threshold (about 0.7–1.4% of steps). On an A10G this raises exact output agreement by 22–36 percentage points at under 4% latency overhead for batch sizes up to 4. It does not help at batch size ≥8, with fully FP8-quantized models, or on certain Qwen checkpoints whose divergence originates before the head.
Does forcing cross-precision agreement hurt answer quality?
Matched-pair McNemar tests on four models found no statistically significant accuracy change, and a TOST non-inferiority analysis on Qwen2.5-3B (n=600) bounds any accuracy drop within 5pp (p=0.009). The authors note Mistral-7B point estimates trend slightly negative but cannot be resolved at n=100, so workload-specific validation is advised.
Is greedy decoding deterministic at all?
Within one fixed numerical configuration, yes — with PyTorch deterministic flags and a fixed CUBLAS workspace, repeated runs are bit-identical. The failure this paper documents is cross-precision: each format repeats itself faithfully, but the two formats disagree with each other, so a system can be repeatable while still failing cross-format replay.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several sections were elided in extraction (parts of §5–6, §7.4, and Appendices II, V, XIV, XV); judgments rely on the visible main text and tables.
- Figures are available only as captions/descriptions; visual claims (e.g., Fig. 1 margin bimodality) could not be independently inspected.
- The appendix overview lists code under Appendix XI, but no repository URL was visible in the extracted text.
- Margin statistics differ across protocols (Fig. 1 median ≈10⁻⁵ vs App. I.3 median 0.039); the paper attributes this to different step populations, but the full protocol text was partly truncated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.