Reproducible AI Requires Reproducible Randomness
Anthony Bertrand, Tom Schmitt, Engelbert Mephu Nguifo, David Hill
cs.AI
Quick verdict
PRNG state transfer does not guarantee identical streams across Python libraries; sound mechanism study, but empirics cover one seed and three KAT vectors.
Summary
This empirical study tests a widespread assumption in computational science: that transferring a PRNG's complete internal state guarantees identical random streams across software libraries. The authors compare Mersenne Twister (MT19937) and Philox implementations in Python's random module, NumPy (legacy and Generator APIs), PyTorch, and TensorFlow against original reference implementations (Matsumoto's OMT; the Random123 OPX with known-answer-test vectors), under pinned versions (Python 3.11.2, NumPy 2.3.2, PyTorch 2.8.0+cu126, TensorFlow 2.20.0). Seed-based initialization fails almost everywhere. Full-state transfer fixes Python-random floats and both NumPy paths, but PyTorch MT integers consume an extra 32-bit output per value (recoverable via int64-draw decomposition), PyTorch floats use 24-bit mask normalization instead of the reference 32-bit scheme, NumPy's Philox pre-increments its counter, TensorFlow's Philox skips outputs, and PyTorch's Philox exposes only 64 of 128 counter bits — deemed unfixable without source changes. The paper contributes a three-class non-portability taxonomy (user-induced, user-resolvable, fundamental), user-level bit-manipulation workarounds, and public notebooks with validated states. Limitations: MT evidence rests on a single seed (42) and Philox on three KAT vectors; one version per library; the claim that CPython's randint is an implementation error is contestable; and several tables contain apparent typos.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Transferring a PRNG's complete internal state does not guarantee identical output streams across Python libraries for either Mersenne Twister or Philox.
- PyTorch's MT float generation applies 24-bit mask normalization instead of the reference 32-bit conversion, producing lower-precision values that cannot be corrected at user level.
- PyTorch's Philox state exposes only 64 of the 128 counter bits, reserving the rest for parallelization, so cross-library Philox stream fidelity is unattainable without source modification.
- User-level bit manipulation recovers reference streams for PyTorch MT integers, TensorFlow Philox, and NumPy Philox, which the authors classify as user-resolvable non-portability.
- Implementation fidelity of PRNGs is a necessary condition for scientific reproducibility, so evaluation must cover initializations and implementations, not just algorithms.
Strengths
Tests against true reference implementations
Compares libraries to Matsumoto's OMT and the Random123 OPX with known-answer-test vectors (Init 0/F/PI, Table 5), not merely library-vs-library — the correct design for a fidelity question. Reference outputs are printed (Table 2: seed 42 genrand_int32 begins 1608637542, 3421126067, 4083286876, 787846414, 3143890026).
Mechanism-level root-causing, not symptom lists
Section 8 reconstructs PyTorch's float path as (int & (2^24-1))/2^24 — 24-bit precision instead of the reference /2^32 — validated against actual torch output (Code 2); Code 1 recovers reference MT integers via int64-draw splitting; NumPy's Philox discrepancy is traced to counter pre-increment, after which Table 10 matches all three 4x64 KATs.
Actionable taxonomy and user-level fixes
Verification-friendly reporting
Weaknesses
Very thin empirical coverage
Contestable normative claim about CPython randint
Table inconsistencies and proofreading
Framing conflates design choices with defects
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow but load-bearing question: does handing a PRNG its complete internal state guarantee identical output streams across software libraries? The authors test MT19937 and Philox (4x32 and 4x64 variants) as implemented in Python's random module, NumPy (legacy random and Generator, 'NumPy V2'), PyTorch, and TensorFlow, against the original references: OMT from Matsumoto's emt19937ar distribution and OPX from the DES Shaw Research Random123 repository, whose kat_vector file supplies three exact initializations (Init 0, Init F, Init PI). Three claims organize the work: (1) seed-based initialization is insufficient across implementations; (2) even full-state transfer fails in several mainstream libraries; (3) discrepancies split into user-induced, user-resolvable, and fundamental non-portability. The headline assertion — 'reproducibility cannot be assumed from PRNG state transfer alone' — is the one that must carry the most evidentiary weight.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does copying a PRNG's full internal state guarantee the same random numbers in a different Python library?
No. This paper shows that even with identical complete initial states, Mersenne Twister and Philox streams diverge across Python's random module, NumPy, PyTorch, and TensorFlow, because libraries make different implementation choices (seeding routines, integer consumption, float normalization, counter layout). Only some combinations match the original reference implementations.
Why do PyTorch and NumPy produce different random numbers with the same seed?
They use different seeding algorithms and generation pathways. With seed 42, NumPy's legacy interface matches the original Mersenne Twister reference, but Python's random module, NumPy's Generator API, and PyTorch do not. PyTorch additionally consumes two 32-bit draws per integer and uses a 24-bit mask for floats, so its outputs differ even after full-state transfer.
Can I make PyTorch reproduce the original Mersenne Twister sequence?
Partly. For 32-bit integers, the authors' workaround draws 64-bit integers and splits each into two uint32 values, recovering the reference sequence. For floats, no user-level fix exists because PyTorch normalizes with a 24-bit mask, (int & 2^24-1)/2^24, instead of the reference division by 2^32; matching requires rebuilding floats manually from curated integer streams.
What is wrong with PyTorch's Philox generator?
PyTorch stores its Philox state as 128 bits total: a 64-bit key plus only 64 bits of the counter, which the reference algorithm allots 128 bits; the remaining space manages parallel substreams. Users therefore cannot set the exact full state. With the all-zero initialization, only the first generated value matched the reference (and it equaled the second reference number), and the paper classifies this as unfixable without modifying PyTorch's source.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Review based on the extracted full text; Figure 1 (Philox counter evolution) is available only as a caption-level description, and the KAT hex values in Tables 5/6/10 could not be independently recomputed here.
- Some table rows appear garbled by extraction (e.g., Table 8's torch integer row), so table-level judgments carry slight transcription uncertainty.
- The abstract in the metadata is truncated mid-sentence, but the full text includes a complete abstract.
- Library behavior is version-specific; findings are pinned to PyTorch 2.8.0+cu126, NumPy 2.3.2, TensorFlow 2.20.0 and may have shifted since.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.