Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment
Anirudh Malik, M Sparsh Mehra, Poojith Devan
cs.AI · cs.LG
Quick verdict
Transparent PTQ study ternarizing Qwen3-4B to 1.641 bits/weight (-9.8 pts acc., 8.29→3.96 GiB), honestly reported but lacking external baselines.
Summary
This industrial research report from OneBit AI applies weight-only post-training ternarization to the instruction-tuned Qwen3-4B using the TWLA pipeline — KOTMS rotation, E2M-ATQ ternarization, and GPTQ-style compensation — quantizing 252 linear tensors (81.62% of parameters) while activations stay FP16 and embeddings/LM head remain BF16. The measured effective budget is 1.641 bits/weight rather than the nominal 1.585. Across ten zero-shot tasks evaluated with paired bootstrap confidence intervals, mean accuracy falls from 64.5% to 54.7% (-9.8 points); degradation is stratified: BoolQ retains 84.6% chance-corrected teacher performance, ARC-Challenge only 43.8%. Perplexity rises from 13.639 to 18.748 on WikiText-2, with the worst ratio on C4 (1.46×). A lattice-aware packing run shrinks the checkpoint from 8.29 to 3.96 GiB with unchanged perplexity, while a lossy third-party packing attempt is explicitly excluded. Execution results are negative: the reconstruction path decodes about 22% slower than FP16, and a preliminary Triton GEMV microbenchmark is 4.6× slower than cuBLAS. Limitations include a single seed, no external baselines (int4 GPTQ/AWQ or competing ternary PTQ), an unexplained perplexity mismatch between the headline run (18.748) and the packing run (18.344), no end-to-end benchmark of the packed artifact, and no stated code release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A weight-only PTQ pipeline (KOTMS rotation + E2M-ATQ + GPTQ compensation) converts Qwen3-4B to an adaptive ternary representation at 1.641 effective bits/weight covering 81.62% of parameters, with embeddings and LM head kept BF16.
- The conversion costs 9.8 accuracy points on average across ten zero-shot comparisons (64.5% to 54.7%), with stratified degradation: BoolQ retains 84.6% chance-corrected teacher performance versus 43.8% for ARC-Challenge.
- Lattice-aware packing reduces the checkpoint from 8.29 GiB to 3.96 GiB with perplexity preserved to reported precision, while a lossy third-party packing attempt is explicitly excluded from claims.
- The current execution path provides no speedup — reconstruction-based decode is 0.78× FP16 and a preliminary Triton GEMV is 4.6× slower than FP16 cuBLAS — so compression is demonstrated but acceleration remains open.
Strengths
Honest effective-bit accounting
The paper separates the nominal 1.585-bit ternary symbol from the measured 1.641 bits/weight (§7.1, Eq. 12-13), gives the salience-mask breakdown (Table 5: ~3.6% of weights on a second plane), and keeps representation bits distinct from the 3.96 GiB checkpoint (§7.3), correcting a common abuse of the '1.58-bit' label.
Methodologically careful capability evaluation
Chance-corrected retention (Eq. 14) instead of raw ratios, paired per-item records with 10,000-resample bootstrap CIs, dual MMLU scoring protocols (§6.4: letter 68.5→51.0 vs continuation 45.1→35.0), and a consistent audit trail (§11.5). Table 4 numbers are internally consistent — the 64.5/54.7 means recompute exactly.
Candor about negative results
Above-average reproducibility detail and failure documentation
Weaknesses
No external baselines at matched or higher bits
Unexplained inconsistency between headline and packing-run perplexities
Statistical fragility
The actual deliverable is under-validated and not released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an industrial technical report (OneBit AI) characterizing an end-to-end, weight-only post-training ternarization of instruction-tuned Qwen3-4B using prior TWLA components: KOTMS rotation (§4.2), E2M-ATQ order-2 asymmetric residual ternarization (§4.3), and GPTQ-style Hessian-based compensation (§4.5). The paper explicitly disclaims algorithmic novelty (§13.2) and frames four empirical questions: capability, representation, storage, execution. Central claims: (1) a 1.641 effective bits/weight quantized representation over 81.62% of parameters (Table 2: 3.633B ternarized weights across 252 linear tensors; embeddings/LM head stay BF16); (2) mean zero-shot accuracy 64.5% → 54.7% across ten comparisons (Table 4) with stratified loss (BoolQ 84.6% vs ARC-Challenge 43.8% chance-corrected retention); (3) lattice-aware packing shrinks the checkpoint 8.29 → 3.96 GiB at unchanged perplexity (Table 6), with a lossy third-party packing explicitly excluded (§9.2); (4) no speedup — reconstruction decode is 0.78× FP16 (Table 7) and a Triton GEMV is 4.6× slower than cuBLAS (Table 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How much smaller is the ternarized Qwen3-4B model than the original?
After a lattice-aware packing run, the checkpoint drops from 8.29 GiB to 3.96 GiB (about 2.1× smaller) while reproducing the same perplexity to reported precision. The quantized linear weights use 1.641 effective bits/weight, but embeddings and the language-model head stay BF16 and dominate the remaining size.
Is the 1.58-bit Qwen3-4B faster than the FP16 original?
No. The paper reports that its reconstruction-based execution path decodes at 21.28 tok/s versus 27.41 tok/s for FP16 (0.78×), and a preliminary Triton GEMV kernel is 4.6× slower than FP16 cuBLAS on the tested shape. The authors explicitly do not claim any inference speedup.
How much accuracy does the ternarized Qwen3-4B lose?
Mean zero-shot accuracy across ten tasks falls from 64.5% to 54.7% (−9.8 points). Per-task losses range from −3.3 points on BoolQ to −17.5 points on MMLU under letter scoring; chance-corrected retention spans 84.6% (BoolQ) down to 43.8% (ARC-Challenge), with knowledge-heavy tasks degrading more.
What does '1.641 effective bits per weight' mean?
A single ternary symbol (−1/0/+1) carries log2(3)≈1.585 bits of information. This model assigns a Hessian-salient ~3.6% of weights a second ternary plane, so the measured average information budget is 1.641 bits/weight — distinct from both the ideal 1.585 figure and the full 3.96 GiB checkpoint size, which includes BF16 embeddings and metadata.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.