OPEN-1B: A Fully Auditable Training Run
John Donaghy, Brian Wilcox, Oğuzhan Ersoy, Shikhar Rastogi, Adam St Arnaud, Alexey Titov, Jordan Greenberg, Ben Fielding, et al.
cs.LG
Quick verdict
Claims the first bitwise-fully-auditable LLM training run (Open-1B) via RepOps, an invariant data stream, and deterministic collectives; engineering is credible and costs honestly reported, but the audit trust model rests on publisher-generated hashes.
Summary
The paper introduces fully auditable training: a regime in which every operation of an LLM pretraining run is bit-for-bit reproducible across heterogeneous hardware, so any single training step can be replayed and verified on one commodity device. The authors attribute irreproducibility to three ordered sources - kernel reduction order, data-parallel batch ordering, and collective communication - and address each with RepOps kernels (fixed reduction order, unfused arithmetic, uniform subnormal flushing, counter-based RNG), a topology-invariant data stream, and deterministic collectives replayable on one device via a TreeFold algorithm. They release Open-1B (1.61B parameters, 400B tokens on 48xH100) with the full 450.5B-token data mix, checkpoints every 100 steps, per-step hashes, and a cross-hardware audit harness. Training is natively int8 quantization-aware; after diagnosing LSQ step-size drift and gradient creep from zero gradients in the int8 attention backward, they re-pin quantizer scales every step. Reproducibility costs roughly 5x throughput versus stock PyTorch (5% vs 40.5% MFU) plus about 6s/step hashing; Open-1B trails OLMo 2 1B on benchmarks (OLMES 25.4 vs 31.9), attributed to a 10x smaller token budget. Main limitations: the hash chain is publisher-generated with no external anchor, audit cost and third-party coverage are unquantified, and the conclusion calls RepOps proprietary despite promising a full codebase.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Bitwise reproducibility across heterogeneous hardware (x86/ARM CPUs, NVIDIA and Apple GPUs) is achievable by fixing reduction order, arithmetic conventions, and subnormal behavior, and by using a counter-based RNG (RepOps, Section 2.1, Table 1).
- A topology-invariant data stream plus deterministic collectives allow any single training step of a distributed run to be replayed bit-exactly on one commodity device (Sections 2.3 and 6, Algorithm 1).
- Open-1B is the first fully auditable LLM: its 400B-token trajectory is verifiable against published per-step hashes using the released dataset, 100-step checkpoints, code, and audit harness (Sections 3 and 6).
- Collective verification by many independent auditors covering all steps would, for the first time, make undisclosed-data and backdoor injection detectable (Sections 1 and 6).
- Native int8 quantization-aware pretraining is viable at scale after re-pinning quantizer step sizes every step, which cures an LSQ step-size drift failure mode (Section 4.2).
Strengths
Genuinely new transparency tier with complete artifacts
First released model whose whole trajectory is checkable: per-step hashes, checkpoints every 100 steps over 80,957 steps, full 450.5B-token dataset (Table 5), training code, and an audit harness targeting x86/ARM CPUs, NVIDIA GPUs, and Apple Silicon (Sections 1, 3, 6).
Systematic, unusually deep treatment of floating-point hazards
Table 1 maps each break (non-associative reductions, FMA fusion, FTZ/DAZ subnormal handling, per-platform RNG) to a concrete fix; Appendix A.3 even replaces division with precomputed reciprocal multiplies because CUDA and CPU rounding differ by 1 ULP, and TreeFold keeps O(log R) accumulators so single-device replay fits in memory.
Honest cost accounting
Transferable stability findings for low-precision training
Topology-invariant data stream has standalone value
Weaknesses
Trust model for the audit is under-specified
Internal contradiction on openness of RepOps
Audit feasibility never quantified
int8-vs-bf16 comparison stops at loss curves
Single run, single seed, no uncertainty
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Open-1B?
Open-1B is a 1.61B-parameter open-source language model from Gensyn, pretrained on 400B tokens, released together with its full training dataset, intermediate checkpoints every 100 steps, the training code, and an audit harness - so anyone can re-run any single training step on ordinary hardware and check it bit-for-bit against published hashes.
Why are existing open-source LLMs not reproducible even when code and data are released?
Floating-point addition is not associative, and different hardware sums and compiles operations in different orders, so the same recipe yields slightly different weights on different machines. Framework deterministic modes only make repeat runs on one machine identical; they do not carry across CPU, NVIDIA, and Apple hardware, which is the gap this paper targets.
How much slower is bitwise-reproducible training?
In this paper, roughly 5x slower than a well-optimized stock PyTorch bf16 setup (about 5% vs 40% model FLOPs utilization on the same cluster). int8 quantization-aware training recovers part of the gap, and the per-step state hash adds a constant ~6 seconds per step, which is 5-21% of step time depending on cluster size.
Can this scheme actually detect backdoors injected during training?
Only conditionally. Any tampering with data or code would make an honest replay of the affected step mismatch the published hash, so injected data is detectable in principle - but only if auditors actually verify the specific step(s) containing it. That is why the paper proposes collective verification in which many independent auditors together cover all ~81,000 steps.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are present only as extracted text; curve-level values (e.g., the exact int8-vs-bf16 divergence in Figure 7 or drift shape in Figures 5-6) could not be independently read beyond numbers stated in captions and prose.
- The supplied abstract is truncated mid-sentence relative to the full text; the review relies on the full embedded text.
- The 'Code | Models | Audit App' links are placeholders in the extracted text, so the actual availability and licenses of released artifacts (including the RepOps contradiction flagged above) could not be verified offline.
- Benchmark numbers in Table 6 were extracted from flattened text; the headline OLMES averages (25.4 vs 31.9) are used, but per-task column alignment was ambiguous and is not cited here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.