Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
Liam Cooper, Shinnung Jeong, Hyeran Jeon, Jeffrey Young, Hyesoon Kim
cs.AR · cs.LG
Quick verdict
Claims bitwise cross-GPU determinism via fixed IEEE-FMA upcast GEMMs, 1.17–3.1× faster than prior fix; well-supported, but NVIDIA-only, linear layers only.
Summary
The paper targets cross-GPU nondeterminism in greedy LLM decoding, caused by floating-point non-associativity combined with architecture-dependent GEMM kernel selection. It replaces the prior state-of-the-art mitigation (Yuan et al. 2025), which materializes a transient FP32 copy of BF16 weights before vendor GEMMs, with fixed-configuration Triton kernels that load BF16 weights, upcast them in registers, accumulate with IEEE-754 FMA on CUDA cores, pin configurations to problem shape, and combine split-K partials in a fixed ascending order without atomics. The reduction order thus becomes a pure function of problem shape: linear-layer outputs are bitwise identical across A100, L40S, and H100 on all seven probe shapes, where the baseline matches on none. In vLLM end-to-end across three models and four math/science benchmarks, cross-GPU divergence falls from 30.81–100% (unmitigated BF16) to zero in 11 of 12 cells; the single residual cell (0.08%, Qwen3-4B GSM8K) is traced to an unpinned attention kernel. The method runs 1.17–1.43× faster than the baseline on A100/H100 and 1.6–3.1× on L40S, saving 1–1.4 GiB of weight memory (4–7% more KV capacity). Limitations: validation on NVIDIA GPUs only, coverage of linear layers only, batch invariance capped at M≤64, and no measured head-to-head against alternative determinism systems.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fixed-configuration fused-upcast Triton GEMM kernels with IEEE-754 FMA, shape-pinned configurations, and deterministic split-K make linear-layer outputs bitwise identical across A100, L40S, and H100 on all seven probe shapes.
- The method is faster than the Yuan et al. (2025) mitigation in all 36 end-to-end configurations (1.17–1.43× on A100/H100, 1.6–3.1× on L40S) while halving decode-phase weight memory traffic.
- End-to-end cross-GPU greedy divergence drops from 30.81–100% (unmitigated BF16) to zero in 11 of 12 cells; the residual 0.08% cell originates in an unpinned attention kernel, not the linear layers.
- Batch invariance holds by construction for M≤64 (row 0 bitwise independent of batch size), yielding exactly zero cross-seed divergence, a property cuBLAS lacks at every tested batch size ≥2.
- A margin-versus-shift analysis over 2.6M decoded positions explains observed flip rates (median predicted/measured ratio 1.5) and shows no safe precision threshold exists below exactness.
Strengths
Reproducibility by construction, not statistically
Fixing the reduction order as a pure function of problem shape reduces cross-architecture agreement to IEEE-754 correctness; verified bitwise on 7/7 probe shapes across A100/L40S/H100 with CPU-seeded inputs and per-node hashing, versus 0/7 for the Yuan et al. cast-then-cuBLAS path (Cross-Architecture Determinism section).
Performance win from the same mechanism, not a tradeoff
Fused register upcast halves decode weight traffic; faster than the baseline in all 36 (GPU, model, benchmark) configurations (1.17–1.43× on A100/H100, 1.6–3.1× on L40S), with Table 2 quantifying 1–1.4 GiB weight savings converted to 4–7% more KV-cache capacity.
Unusually thorough rule-level ablations (Supplement C)
Honest failure accounting and confound control
Community-valuable side findings
Weaknesses
Vendor and pipeline scope limits
Regime boundaries carry real costs
No measured comparison against alternative determinism systems
Narrow evaluation envelope
No uncertainty quantification
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Accelerating the Mitigation of LLM Inference Nondeterminism Across GPU Architectures
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why does the same LLM prompt give different answers on different GPUs even at temperature 0?
Floating-point addition is not associative, so the result of a dot product depends on the order of summation. Different GPUs select different GEMM kernels with different reduction orders and tensor-core arithmetic, producing tiny rounding differences that can flip a near-tie argmax and fork the whole generated sequence.
How much faster is this method than the previous best mitigation, and does it hurt accuracy?
The paper reports 1.17–1.43× faster end-to-end than the prior state-of-the-art (Yuan et al. 2025) on A100 and H100, and 1.6–3.1× on L40S, across all 36 tested configurations. Accuracy differences are numerical-noise level: identical scores on both Qwen3-4B tasks and 1–2 problems difference on Llama-3.2-3B.
Is the LLM output now 100% identical across GPUs?
Not quite. The linear layers are bitwise identical across the three tested NVIDIA GPUs, and 11 of 12 end-to-end cells show zero divergence, but one cell (0.08%, Qwen3-4B on GSM8K) still diverges because the attention kernel — not covered by the method — differs on H100. Full-pipeline determinism would also require pinning attention, normalization, and the sampler.
Does this work on AMD GPUs or consumer cards?
The paper tests only NVIDIA A100, L40S, and H100. The authors argue the approach applies in principle to any hardware with correctly implemented IEEE-754 FMA, but cross-vendor (e.g., AMD) validation is stated as an open problem, and consumer GPUs were not tested.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The system's name is blanked throughout the extracted text (e.g., 'We present , which'); the GitHub URL (github.com/lpc97667/rf) and context identify the artifact as an extraction loss, and the baseline system name is similarly missing.
- Figures 1–5 are present only as captions and surrounding text; plot-level claims (Figure 3 distributions, Figure 4 error bars, Figure 5 correlation) cannot be visually verified.
- The abstract is truncated mid-sentence in the extraction, though the body and supplement cover the same content.
- Judgments rest on the authors' reported numbers; no external execution of the released code was performed for this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.