Silent Failures Beyond the 32-Bit Index Range: A Differential Characterization of Large-Tensor Matrix Multiplication in PyTorch's MPS Backend
cs.DC · cs.LG
Quick verdict
Rigorous black-box study finding silent MPS bmm corruption above 2^32 elements in PyTorch across 11 releases; systematic, reproducible, convincing.
Summary
The paper reports that PyTorch's MPS backend on Apple Silicon silently returns wrong results for torch.bmm once more than 2^32 elements are involved, with relative errors of order one and no exception or warning, in every release tested (2.4.1 to 2.14.0). Using a boundary-targeted differential sweep against a float64 CPU reference, the author varies dtypes, memory layouts, shapes and batch sizes around the 2^31 and 2^32 element boundaries, and adds index-encoded inputs that reveal which input elements each wrong output was computed from. Three ordered rules reproduce all 1376 random-input outcomes on 2.14.0: transposed operands with an output above 2^32 corrupt the whole output (strides ignored); views with at least 2^31 elements raise; contiguous inputs above 2^32 corrupt only later batches (index wraps at 2^32). The rules also account for 592 backward-pass outcomes, where a correct forward pass can yield silently wrong gradients. Results are identical across two chips and two macOS versions (6156 runs) and correct on A100 CUDA (2530 runs). A RoBERTa sentiment classifier case study shows one oversized batch corrupting a third of outputs, with predictions collapsing onto one class. A released guard blocking any MPS tensor at or above 2^32 elements stops all 481 silent runs, at up to 1.73x overhead on small workloads. Limitations: systematic coverage restricted to bmm, M2-generation hardware, fp32/fp16 only, and descriptive rather than causal rules.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 1
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PyTorch's MPS backend silently returns wrong torch.bmm results (relative errors above 1, no exception or warning) for problems exceeding 2^32 elements, in every release from 2.4.1 to 2.14.0.
- Three ordered rules reproduce all 1376 random-input outcomes on 2.14.0 and identify the wrong values as two specific misreadings: strides of a transposed operand ignored, and a flat index wrapping at 2^32 elements.
- A correct forward pass can come with silently wrong gradients in exactly the batches beyond element 2^32 of the upstream gradient; no new rules are needed to predict this.
- The failures are deterministic and backend-specific: identical in all 6156 runs across two chips and two macOS versions, and absent in all 2530 CUDA runs on an A100.
- A single guard rule (never create or read an MPS tensor with 2^32 or more elements) stops all 481 silent failures recorded across 11 versions.
Strengths
Boundary-targeted black-box methodology with calibrated oracles
Section 3 defines tolerances from calibration runs (Eq. 2: 10x max calibration error, floors 1e-6 fp32 / 1e-3 fp16), tests five consecutive batch sizes around each candidate boundary plus a grid with bisection as fallback, confirms switch points with full all-element comparisons under three seeds, and never reads inputs back from the device for the reference — eliminating transfer-fault cancellation.
Index-encoded inputs turn 'wrong' into 'wrong in a specific, explainable way'
Prime-period codes (Eq. 4-6, P=1048573 in fp32) decode which batch/position each wrong output element was read from: decoded shift delta = -65536 batches, exactly 2^32 elements, and transposed-position decoding shows (0,1) vs (1,0) swaps consistent with a 256x64 storage read as 64x256 — the two misreadings are confirmed element-by-element, not just hypothesized.
Rules validated on conditions outside the sweep they were derived from
Determinism and backend-specificity demonstrated to unusual depth
Impact case study quantified with controls, and mitigation evaluated exhaustively
Weaknesses
Systematic coverage is one operation; others sampled at a single shape
Hardware and OS generality untested beyond one chip generation
Rules are descriptive, version-bound, and partly rest on an unobserved assumption
Guard overhead and untested combinations
Case-study generalization limits
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a black-box measurement study of torch.bmm on PyTorch's MPS backend for Apple Silicon, triggered by Issue #197636: after a macOS upgrade, operations whose output exceeds 2^32 elements run to completion but return wrong values where macOS 14 raised an explicit error. The claims are: (1) bmm — and matmul/eager attention built on it — fails silently with order-one relative errors in all eleven releases tested (2.4.1-2.14.0); (2) three ordered rules (Table 3) reproduce all 1376 random-input outcomes on 2.14.0 and identify the wrong values as two specific misreadings (a transposed operand's strides ignored; a flat index wrapping at 2^32); (3) the rules transfer to the backward pass, where a correct forward pass can come with silently wrong gradients; (4) the failures are deterministic, identical in 6156 runs across two chips and two macOS versions, and absent in 2530 CUDA runs on an A100; (5) one guard rule — no MPS tensor with 2^32 or more elements — stops all 481 silent outcomes recorded across versions. The authors are explicit throughout that the rules describe observed behavior, not the closed-source implementation.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is PyTorch on Mac (MPS) giving silently wrong results for very large tensors?
Yes, for this size range. The paper shows torch.bmm — and torch.matmul and eager attention built on it — returns wrong values with relative errors near 1 once a tensor exceeds 2^32 (about 4.29 billion) elements, with no exception or warning, on Apple Silicon across PyTorch 2.4.1 through 2.14.0. Transposed-view operands with an oversized output corrupt the entire output; contiguous oversized inputs corrupt exactly the batches beyond element 2^32.
How can I avoid or detect the MPS silent failure?
The paper's mitigation is one rule: never create or read an MPS tensor with 2^32 or more elements. Split large batches into chunks before moving them to the device (chunking does not change the result for bmm or attention), or install the released guard — a TorchDispatchMode that raises on any operation whose inputs or outputs would have 2^32+ elements, with estimated per-op checks for mm/bmm/addmm/baddbmm/SDPA.
Does this affect training or only inference?
Both. Even when the forward pass is correct, the paper measured silently wrong gradients in exactly the batches beyond element 2^32 of the upstream gradient (20 backward runs, errors 1.16-1.57 matching an index wrapped at 2^32). A forward pass that respects the 2^32 rule never produces such a gradient, so chunking before the device also protects training.
Is NVIDIA CUDA affected by the same bug?
Not for bmm: all 2530 A100 runs in the paper's sweeps, including backward and extensions, were correct. However, torch.arange with more than 2^32 elements was silently wrong on CUDA too (elements from index 2^32 left at zero, Issue #197673, in versions 2.11.0-2.14.0), so CUDA cannot serve as a safe oracle at this scale.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 could not be inspected directly from the extracted text; their content was reconstructed from captions and the surrounding prose, which is internally consistent.
- Tables 3-9 were extracted with some formatting loss; numeric counts were cross-checked against the running text where possible and agree.
- Upstream references (PyTorch issue/PR numbers #197636, #149325, #197673; arXiv v2, Sep 2026) could not be verified offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.