QTEA: Ternary LLMs with Sparse Residual Salient Weight and By-Column Optimization
Yipin Guo, Arun M George, Jie Fu, Tareq Mahmoud, Sixue Xing, Siddharth Joshi
cs.LG · cs.AI
Quick verdict
1.7-bit ternary PTQ with sparse FP8 residuals and error decay; strong gains over ternary baselines look credible, but key baselines are cited, not re-run.
Summary
QTEA is a post-training quantization (PTQ) framework that pushes large language models below 2 bits per weight. It quantizes all weights to a ternary base ({−1,0,+1}) and then compensates ternary rounding error with FP8 residuals placed only in salient columns — 5% of columns with 1:4 semi-structured sparsity within each column, i.e. 1.25% of weights — keeping the format regular for GPU kernels. Two additions target GPTQ-style quantization: a column-wise rescale factor alternately refined with ternary reassignments during column-by-column quantization, and an error-decay factor that exponentially damps error propagation as the remaining column suffix shrinks. A LUT-based CUDA kernel (five ternary values per byte) delivers 7.2× per-token speedup over FP16 with CUDA Graphs on Llama2-70B (3.6× on Qwen3-14B) and 13.3× over FP16 without graphs. On Qwen3-14B at 1.7 bpw, QTEA reaches 52.65% average zero-shot accuracy versus 45.11% for the best ternary baseline (PT2-LLM), with WikiText2/C4 perplexity 11.78/26.14 versus 16.48/68.13; Llama3-8B gains are smaller (40.29% vs 37.79%). An iso-memory control (Table 10) shows the gain is not just from extra FP8 storage. Limitations: baselines are cited from the PT2-LLM paper rather than re-run; no variance or significance reporting; custom-hardware results are architecture-level estimates; only decoder-only models are evaluated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- QTEA quantizes all weights to a ternary base at an effective 1.7 bpw (1.6 ternary + 0.1 FP8 residual) and uses column-semi sparse salient residuals as error compensators rather than unquantized bypasses.
- On Qwen3-14B, QTEA improves average zero-shot accuracy 16.7% relative over the strongest ternary PTQ baseline (52.65% vs 45.11%) and lowers WikiText2/C4 perplexity 1.40×/2.61×.
- Column-wise rescale refinement inside GPTQ and error-decay damping each contribute measurably (no-rescale 50.63%; removing decay costs 0.46pp on Qwen3-14B and 2.74pp on Llama3-8B).
- The LUT-based CUDA kernel achieves 7.22× per-token speedup over FP16 with CUDA Graphs on Llama2-70B (3.62× on Qwen3-14B) with negligible residual-path overhead.
- Accuracy gains persist under an iso-memory comparison where PT2-LLM receives the same FP8 salient budget (Table 10: QTEA still ahead by 6.08/4.87 points).
Strengths
Regular, hardware-friendly salient-residual representation
Column-semi sparse FP8 residuals (5% columns × 1:4 = 1.25% of weights) keep a predictable access pattern; Table 4 quantifies the accuracy cost of this regularity at only 0.31/0.29pp vs the dense Column-5% variant and 0.91/0.57pp vs unstructured Global 1.5%, and Fig 7 shows the residual path adds just 0.01–0.03 ms/token over the ternary-only kernel on H200.
Iso-memory control isolates the contribution
Table 10 gives PT2-LLM the same FP8 salient-weight budget, improving it only 1.90/2.67 points on Qwen3-8B/14B, while QTEA stays 6.08/4.87 points ahead — the gain is not merely extra precision storage.
Unusually complete ablations
Real kernels plus honest cost disclosure
Internally consistent headline numbers
Weaknesses
Baselines cited, not re-run
Missing strongest modern competitors
No statistical rigor
Best-case speedup in headline
Absolute quality gap and narrow evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
QTEA (arXiv 2609.00224v2, University of Notre Dame) is a weight-only post-training quantization (PTQ) framework targeting sub-2-bit LLM inference. It makes four coupled claims: (1) all weights first go to a ternary base ({−1,0,+1}, 1.6 bpw, packed 5-per-byte), and salient-weight information is re-introduced not as a high-precision bypass but as FP8 residual corrections confined to 5% of columns with 1:4 semi-structured sparsity inside each column (1.25% of weights, 0.1 bpw), for 1.7 bpw total; (2) a per-column rescale factor vj is alternately refined with ternary reassignment during GPTQ-style column-by-column quantization (§3.3); (3) an error-decay factor γi = exp(−λdecay·[H⁻¹]ii/mean_k[H⁻¹]kk·i/B) damps error propagation as the remaining column suffix shrinks (§3.4); (4) a LUT-based CUDA kernel yields per-token speedups of 7.22×/13.3× vs FP16 with/without CUDA Graphs on Llama2-70B. Headline accuracy: Qwen3-14B at 1.7 bpw reaches 52.65% average zero-shot vs 45.11% for the strongest ternary baseline (PT2-LLM), with WikiText2/C4 perplexity 11.78/26.14 vs 16.48/68.13 (Table 1).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is QTEA?
QTEA is a post-training quantization method for LLMs that compresses all weights to an effective 1.7 bits per weight: a ternary ({−1,0,+1}) base plus sparse FP8 corrections placed in 5% of columns with 1:4 sparsity (1.25% of weights). It adds column-wise scale refinement and error decay to GPTQ-style quantization and ships a lookup-table CUDA kernel for fast inference.
How much faster is QTEA than FP16 inference?
With CUDA Graphs, 7.2× per-token on Llama2-70B (41.13→5.70 ms/token) and 3.62× on Qwen3-14B (23.75→6.56 ms), measured on an H200 with the authors' LUT kernel; versus FP16 without CUDA Graphs the speedup reaches 13.3× on Llama2-70B.
Does QTEA's accuracy gain come just from keeping some weights in FP8?
The authors tested this: giving the PT2-LLM baseline the same FP8 salient-weight budget raises it by only 1.90/2.67 points on Qwen3-8B/14B (Table 10), while QTEA stays 6.08/4.87 points ahead — so the gain is not just extra precision storage.
What are QTEA's main limitations?
It is weight-only (no activation/KV-cache quantization), evaluated only on decoder-only models with single runs and no variance reporting; the speedups require the authors' custom kernels (H200); and the custom-chip numbers (3.83× latency, 69.4% less energy) are architecture-level estimates, not post-layout silicon measurements.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded results (Figs 1, 6, 7) are available only via captions and in-text numbers; plotted data points could not be independently verified from the extraction.
- Table 1 is the densest element of the extraction; headline QTEA and PT2-LLM cells were cross-checked against the §4.2 prose and match, but individual baseline cells carry minor misalignment risk.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.