Accuracy is Not Enough: A Divergence-Based Approach to Evaluate Fidelity Loss in Quantized LLMs
Shahzeb Qamar, Lorenz Sparrenberg, Christian Bauckhage, Baha Rababah, Carson Leung, Murat Kantarcioglu, Cuneyt Gurcan Akcora, Rafet Sifa
cs.LG · cs.AI
Quick verdict
TVD/JSD fidelity test for quantized LLMs across 120 runs; sensible diagnostic, but near-chance task baselines and no significance testing weaken evidence.
Summary
The paper argues that zero-shot top-1 accuracy is a poor yardstick for post-training quantization of LLMs, because it depends only on the argmax token and can stay flat while the underlying next-token distribution drifts far from the BFloat16 reference. The authors propose computing Total Variation Distance and Jensen-Shannon Divergence between the full-vocabulary softmax distributions of the uncompressed and quantized models, and prove (Theorem 1) that coarse-graining to answer-level distributions cannot increase these distances. A 120-configuration matrix spans five models (LLaMA-3.1-8B, LLaMA-3.2-1B, Mistral-7B-v0.1, Teuken-7B, Gemma-2B), four benchmarks (MMLU, BoolQ, PIQA, CausalBench), and llama.cpp schemes from BF16 to Q2_K. Reported results include Teuken-7B on PIQA, where Q4_0 loses only 0.2% accuracy yet reaches TVD 0.560, and LLaMA-3.1-8B on MMLU, where Q4_K attains TVD 0.055 versus 0.221 for Q4_0 at similar memory, supporting Q4_K as the better fidelity trade-off. Limitations are significant: all task accuracies come from a single completion-style template under which MMLU scores sit near the 25% chance level and LLaMA-3.1-8B scores below the BoolQ majority class; no confidence intervals, significance tests, or multi-seed variance are reported; perplexity baselines and a code release are absent.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Zero-shot top-1 accuracy is insensitive to quantization-induced distributional change and can remain nearly stable while fidelity loss is severe (e.g., Teuken-7B on PIQA Q4_0: accuracy 68.9%→68.7% but TVD 0.560).
- Full-vocabulary TVD and JSD between BF16 and quantized next-token distributions form a bounded, interpretable fidelity diagnostic, and coarse-graining to answer level cannot increase them (Theorem 1).
- Divergence metrics generally increase under stronger quantization while accuracy behaves non-monotonically in the evaluated settings.
- Across the tested llama.cpp schemes, mixed-precision Q4_K yields substantially lower divergence than uniform Q4_0 at similar memory and accuracy (e.g., LLaMA-3.1-8B MMLU TVD 0.055 vs 0.221).
Strengths
Well-justified, interpretable metric choice
Section III-C selects bounded, symmetric TVD and JSD over unbounded KL (which blows up on near-zero probabilities common at low bits), and TVD carries a direct mass-reassignment interpretation that the paper consistently uses (e.g., TVD 0.560 = 56% of probability mass must move).
Correct and relevant coarse-graining theorem
Theorem 1 (proofs in Supplementary Section VII via log-sum inequality) shows answer-level TVD/JSD are upper-bounded by full-vocabulary distances — the data processing inequality grounded in the quantization-evaluation setting, connecting the prompt-level diagnostic to answer-level measures.
Large, fully reported experimental matrix with honest degenerate-case reporting
Decision-relevant Q4_K vs Q4_0 separation invisible to accuracy
Unusually careful scoping of claims
Weaknesses
Near-chance and below-majority task baselines undermine the headline claim's external validity
No uncertainty quantification whatsoever
Unresolved reference-runtime confound
Compared position and competing distributional metric underspecified
Modest novelty and redundant dual reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets evaluation of post-training quantization (PTQ) for LLMs on memory-constrained devices. Its central claim: zero-shot top-1 accuracy depends only on the argmax token and can stay flat while the next-token distribution drifts far from the BFloat16 reference. The remedy is a distribution-sensitive framework comparing full-vocabulary softmax distributions of BF16 and quantized models with bounded, symmetric divergences — Total Variation Distance (TVD, interpretable as the fraction of probability mass needing reassignment) and Jensen-Shannon Divergence. The authors are notably careful about scope: abstract, Section II and conclusion all state these metrics measure reference fidelity only, not correctness, calibration, safety, or user-perceived quality. Four contributions are enumerated: (1) accuracy masks distributional differences; (2) the TVD/JSD framework plus a coarse-graining theorem; (3) a 120-configuration matrix (5 models × 4 benchmarks × 6 quantization states); (4) Q4_K beats Q4_0 on divergence at similar memory.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper propose instead of accuracy for evaluating quantized LLMs?
It compares the full next-token probability distributions of the uncompressed BFloat16 model and each quantized model using Total Variation Distance (TVD) and Jensen-Shannon Divergence. TVD has a direct interpretation: it equals the fraction of probability mass that must be reassigned to transform one distribution into the other, so a TVD of 0.25 means 25% of the mass moved.
What is the main practical finding of the paper?
Across five models and four benchmarks, mixed-precision Q4_K quantization showed much lower divergence from the BF16 reference than uniform Q4_0 at nearly identical memory footprint and task accuracy — for example TVD 0.055 vs 0.221 for LLaMA-3.1-8B on MMLU — a difference that accuracy metrics cannot detect.
Does a high TVD mean the quantized model performs worse on tasks?
Not by itself. The authors stress that TVD measures distance from the BF16 reference distribution, not correctness, calibration, safety, or user-perceived quality; it is meant as a complementary fidelity diagnostic alongside task accuracy, not a replacement or a deployment threshold.
Is code available for this paper?
The paper contains no code-release statement. It does pin the llama.cpp commit (ded446b), names all public models and datasets, and gives example prompt templates, so an independent reproduction would require reimplementation of the evaluation pipeline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear in the extracted text only as captions and axis labels; judgments about figure internals rest on the captions and the complete appendix tables, not on the plotted values.
- The abstract supplied in the task metadata is cut off mid-sentence, but the complete abstract is present in the paper body itself, so no content was lost for this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.