Scaling Post-Training Ternarisation to Qwen3-8B Capability Retention, Reproduction, Lossless Packing, and Packed Execution
Anirudh Malik, M Sparsh Mehra, Poojith Devan
cs.LG · cs.AI
Quick verdict
Ternary-PTQ scale-up to Qwen3-8B: rare reproduction gate and candor, but single-seed n=500 evals, no competing baselines, and a 7.9% FP16 gap limit strength.
Summary
This company technical report from OneBit AI scales a post-training ternarisation pipeline — KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style error compensation — from Qwen3-4B to Qwen3-8B in a weight-only A16 setting that touches only 252 linear projections, leaving embeddings, LM head, norms, activations and KV cache in FP16 (≈1.64 effective bits/weight, not a whole-model 1.58-bit model). The 8B conversion reproduces the published TWLA reference within 0.05 points on zero-shot average (62.00 vs 62.05), although the local FP16 baseline deviates 7.9% from the published one. Cross-corpus perplexity ratios average 1.361× (WikiText-2 1.318×, C4 1.393×, PTB 1.371×). On eight zero-shot tasks at n=500, mean accuracy is 64.6% versus 72.4% for FP16 — 78.5% chance-corrected retention — and a matched 4B rerun retains 69.6%, an 8.9-point 8B advantage. Lattice-aware packing shrinks the checkpoint from 16.9 GB to 8.24 GiB with perplexity preserved to ~0.0007%, and direct packed execution reaches 15.52 tokens/s in 7.35 GiB, though the prototype packed GEMV is 4.6× slower than FP16 cuBLAS. Key limitations: a single seed, n=500 per task, only two model sizes, no matched comparison against competing ternary PTQ methods, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The TWLA-based ternary PTQ pipeline scales from Qwen3-4B to Qwen3-8B, reproducing the published external reference within 0.05 points on the zero-shot average (62.00 vs 62.05, Table 2).
- Under a matched eight-task protocol at n=500, the 8B conversion retains 78.5% chance-corrected capability versus 69.6% for the 4B run, an 8.9-point size advantage (Tables 4 and 6).
- Cross-corpus perplexity degrades by 1.361× on average (WikiText-2 1.318×, C4 1.393×, PTB 1.371×) at ~1.64 effective bits/weight on linear projections only.
- Lattice-aware packing shrinks the checkpoint from 16.9 GB to 8.24 GiB with perplexity preserved to ~0.0007%, and the packed artifact executes directly at 15.52 tokens/s in 7.35 GiB.
- The prototype packed GEMV remains 4.6× slower than FP16 cuBLAS on one tested shape, so no inference speedup is claimed.
Strengths
External reproduction gate
Table 2 shows the converted 8B model's zero-shot average (62.00) within 0.05 points of the published TWLA reference (62.05), and the authors explicitly refuse to claim ratio superiority given their 7.9% FP16 baseline deviation — a level of gate-keeping rare in PTQ reports.
De-confounded scaling comparison
Section 10 re-benchmarks the 4B checkpoint at the 8B protocol (percdamp=0.01, n=500) after flagging the original comparison as confounded, yielding per-task retentions (Table 6) rather than a bare two-point claim.
Precise representation accounting
Serialization designed into the quantizer
Candor about failures and limits
Weaknesses
No statistical uncertainty anywhere
No competing-method baseline
Internal evaluation inconsistencies
Hardware-constrained experimental design
No code or artifact release stated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a company technical report (OneBit AI) scaling an existing ternary post-training quantization pipeline — TWLA's KOTMS rotation, E2M-ATQ adaptive ternarisation, and GPTQ-style compensation — from Qwen3-4B to Qwen3-8B in a weight-only A16 configuration on a single RTX 5070 (12 GB, Windows). The authors explicitly disclaim algorithmic novelty (Section 16.1) and frame the contribution as an end-to-end scale-up characterization across five questions: external reproduction, capability retention, 4B/8B scaling, lossless packing, and packed execution. The quantified results: (1) zero-shot average 62.00 vs the published TWLA 62.05 (Table 2); (2) 64.6% vs 72.4% FP16 mean accuracy on eight tasks at n=500, i.e., 78.5% chance-corrected retention (Table 4); (3) +8.9 points over a matched 4B rerun (Table 6); (4) 16.9 GB → 8.24 GiB packing with WikiText-2 ΔPPL of 0.000085 (Table 7); (5) direct packed execution at 15.52 tokens/s in 7.35 GiB, with the candid admission that the prototype packed GEMV is 4.6× slower than FP16 cuBLAS (Section 13.1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is the ternary Qwen3-8B really a 1.58-bit model?
No. Only 252 linear projections are ternarised, at roughly 1.64 effective bits per weight (a small salient subset gets a second trit plane); embeddings, LM head, norms, activations and KV cache remain FP16. The packed checkpoint is 8.24 GiB for an 8.2B-parameter model — about 8.6 bits per parameter on disk by simple arithmetic — and the paper is explicit that 'ternary' is engineering shorthand.
How much accuracy does the converted Qwen3-8B lose?
On eight zero-shot tasks with 500 samples each, mean accuracy drops from 72.4% (FP16) to 64.6%, i.e., 78.5% chance-corrected retention. Losses are uneven: knowledge-heavy tasks fall most (MMLU −14.55, ARC-Challenge −10.8) while BoolQ loses only 2.4 points; all six reported math subjects stay above the 25% four-way chance floor.
Is the packed ternary model faster than FP16?
No speedup is claimed. Direct packed execution reaches 15.52 tokens/s using 7.35 GiB — versus roughly 1.48 tokens/s when the model had to be CPU-offloaded because FP16 8B exceeds the 12 GB card — but the prototype packed GEMV kernel is 4.6× slower than FP16 cuBLAS on the single tested shape (4096×2560).
Is code or data available for this paper?
The text announces no code, checkpoint, or format-specification release. It does document unusually complete configs (seed 0, 64 calibration samples of 2048 tokens, percdamp 0.01, lm-evaluation-harness task list), but exact reproduction currently depends on the authors' private pipeline and a Windows/RTX 5070 setup.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-14 are referenced but their visual content is not present in the extracted text; this review relies on the tables and prose, which carry all key numbers.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.