Baseline Shape Decides the Verdict: A Controlled Re-Examination of Ternary Language Models at 60K Parameters
cs.CL
Quick verdict
对 60K ternary-LM 对比的受控重运行：baseline shape 和 LR tuning 会扭转结论；严谨且坦诚，但受限于单一规模/语料库的适用范围。
Summary
本文重新审视了先前发表的一项声明，即 routed ternary (1.58-bit) 混合块在 60K 参数下比参数匹配的 FP32 transformer 在困惑度（perplexity）上提升了 22%。作者在单一固定方案下重新运行了对比实验——在 TinyStories 上以两个训练预算（16M 和 130M 字节）进行了 98 次字节级训练运行，每个单元三个种子（seeds），全部在单台笔记本电脑上完成。Baseline shape 占主导地位：五个参数匹配的 transformer shapes 在 16M 字节下的验证损失跨度达 22.6%，最佳形状与 routed 模型打平，且形状排序随预算变化而反转。在 130M 字节下，routed 模型确实比三个测试的 transformer shapes 提升了 22.2–24.0%，但一个简单的 gated diagonal SSM 比它进一步提升了 9.1%，且 router 诊断显示权重主要集中在 SSM 通路（pathway）上。与架构相关的 ternary 惩罚（+5.3% 对比 +19.5%/+28.1%）被作者标记为存在混淆：transformers 保留了全精度的位置嵌入（full-precision positional embeddings），且 from-scratch LR 未调优。90/10 FP→ternary 调度方案仅在 stage-2 LR 为预训练峰值的约 10 倍时才有效。局限性：仅限于单一参数量、单一语料库、两个预算、部分事后比较以及一个未饱和的 LR sweep。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- At a 16M-byte budget, param-matched transformer depth/width choice spans 22.6% in validation loss — larger than any measured architecture effect — so a single fixed baseline shape can flip an architecture verdict.
- At 130M bytes the routed ternary block genuinely beats the three transformer shapes tested (22.2–24.0%), but a plain gated diagonal SSM beats it by a further 9.1% and its own router concentrates weight on the SSM pathway, so the gain does not require routing.
- The ternary penalty at 130M bytes differs sharply by architecture but cannot be attributed to architecture alone due to unequal quantization (FP positional embeddings), an untuned from-scratch ternary LR, and loss-level covariation.
- A 90/10 FP-then-ternary schedule beats all-ternary training only at a stage-2 LR about 10× the pretraining peak; at a conventional rate it looks 15.3% worse, reversing the conclusion.
Strengths
Pre-specified decision rules and matched design
The 2σ relative noise criterion was fixed before results were inspected, every ternary run has a token-matched FP anchor, budgets are matched within comparisons, and init/data-order seeds are controlled separately (§5).
Explicit claim-evidence mapping
Table 6 marks each claim supported / not supported / confounded; §6.1 labels the 'routed vs. best shape' comparison post-hoc; §6.3 lists three alternative explanations the authors cannot exclude rather than burying them.
Mechanistic corroboration, not just aggregate scores
Exemplary open release at laptop scale
Real methodological yield
Weaknesses
Narrow external validity
Headline ternary-penalty gap is triply confounded
No multiplicity control on the shape sweep
Budget conflates three variables
Imperfect 'no routing needed' contrast
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does the routed ternary model actually beat a transformer at 60K parameters?
It depends on training budget and baseline shape. At a 16M-byte training budget, the best-shaped parameter-matched transformer ties the routed model. At 130M bytes the routed model beats the three transformer shapes tested by 22.2–24.0% — but a simple gated diagonal SSM beats the routed model by a further 9.1% in full precision, so a router is not needed for the gain.
What is the 'baseline shape' problem in small-model comparisons?
At a fixed parameter count, the choice of depth vs. width changes validation loss substantially — in this study, five param-matched transformer shapes span 22.6% in validation loss at a 16M-byte budget (2.835–3.477 nats/byte). That span is larger than the effect of switching architecture entirely, so a paper reporting a single 'param-matched baseline' can get either verdict depending on which shape it picked.
Is the code and data for this re-examination available?
Yes. Training code, parameter-matched configs, a corpus manifest with SHA-256 checksums, and per-run logs are released on GitHub (veldanda/ByteLM, tag p1-v1), and every table and figure regenerates from those logs via experiments/analyze_all.py. Total compute was about 40 hours on a MacBook Air M2, making independent replication realistic.
Does ternary (1.58-bit) quantization hurt all model architectures equally?
Not in this study at the 130M-byte budget: the loss penalty was +5.3% for the best transformer shape versus +19.5% for the routed model and +28.1% for the gated SSM. However, the authors warn this difference is confounded — their transformers keep 11–22% of parameters (including positional embeddings) in full precision while recurrent models quantize ~98%, and the from-scratch ternary training rate was never tuned — so it should not yet be read as an architectural law.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as text in the extraction; Figure 1 values were taken from the caption and body text, not analyzed visually.
- The released GitHub artifacts (tag p1-v1) could not be inspected for this review; reproducibility statements are taken from the paper.
- Table 1's reconstruction of Atome-LM's training conditions is the author's account from the original artifacts and is not independently verified here.
- The abstract in the supplied metadata is truncated mid-sentence; the complete abstract from the paper body was used instead.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.