Train Where the Quantized Model Goes: On-Policy Distillation for Low-Bit Reasoning
Yuanteng Chen, Zhilei Liu, Peisong Wang, Yuantian Shao, Chuangyi Li, Weining Wang, Shuang Qiu, Gang Li, et al.
cs.LG · cs.AI
Quick verdict
On-policy distillation after QAD restores sub-3-bit reasoning (MATH retention 35%→70%); well-controlled but single-run evidence and partial ablations.
Summary
The paper identifies a failure mode of quantization-aware distillation (QAD) below three bits: short-form QA recovers (86% BF16 retention averaged over six Qwen3 settings) but long-form reasoning collapses into repetitive loops — for Qwen3-0.6B at 2.79 bits, 95% of MATH-500 generations exhaust the decoding budget and 70% end in repeated 8-grams. The authors attribute this to quantization-amplified exposure bias: teacher-forced training never supervises the trajectories a quantized student actually generates. Their fix is a second stage, on-policy distillation (OPD): the student rolls out through its deployment quantized forward path, a frozen BF16 teacher supplies token-level reverse-KL feedback on student prefixes, and group-relative verifier rewards (math answer checks, code tests) shape sequence-level quality. Across Qwen3-0.6B/1.7B/4B and Falcon3-1B at 2.79 and 1.88 effective bits, OPD raises average MATH-500 BF16 retention from 35.3% to 69.7% and HumanEval from 66% to 91%, while QA9 retention stays flat-to-better (88.1%→92.1%). A matched-budget control on Qwen3-0.6B/1.7B shows OPD beating continued teacher-forced QAD in all 16 comparisons, at far lower cost (57 vs 820 GPU-hours on Qwen3-4B at 1.88 bits). Limitations: single runs with no variance estimates (re-evaluated baselines drift about 1 point across tables), the matched control omits the 4B and Falcon3 models, and no ablation separates the KL and verifier-reward terms of the objective.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Teacher-forced QAD below three bits recovers short-form QA but fails long-form reasoning, with failures manifesting as decoding-budget exhaustion and repetition loops (Sec 3, Figs 2–3).
- The gap is caused by quantization-amplified exposure bias: supervision on fixed corpus prefixes never covers the trajectories the quantized student generates (Sec 3.2).
- Adding an OPD stage raises average BF16 retention across four models at 2.79/1.88 bits from 35.3% to 69.7% on MATH-500 and from 66% to 91% on HumanEval, while QA9 retention rises from 88.1% to 92.1% (Table 2, Sec 5.2).
- Under matched budgets, OPD beats continued teacher-forced QAD in all 16 controlled comparisons, and the gap persists when QAD is extended to 3× budget (Table 4, Appendix G).
- A student's own BF16 weights are a sufficient teacher for ≥1.7B models; only Qwen3-0.6B requires a larger (1.7B) teacher (Appendix F).
Strengths
Concrete behavioral diagnosis, not just accuracy deltas
Section 3/Figure 3 quantifies the failure mode directly: Qwen3-0.6B W2.79 QAD shows 95% budget exhaustion and 70% repeated-8-gram endings on MATH-500 (BF16: 27%/12%), and Figure 2 shows retention falling monotonically with required generation length (86% QA9 → 56% GSM8K → 27% MATH-500 → 16% AMC23).
The right controlled experiment was run
Table 4 compares OPD against continued teacher-forced QAD with identical starting checkpoints, corpora, schedules, steps and learning rate — OPD wins all 16 comparisons (e.g., Qwen3-1.7B W2.79 MATH-500: 45.60 vs 23.20), and Appendix G shows extending QAD to 3× budget still trails by 8.9–18.1 GSM8K points, ruling out a pure compute-budget explanation.
Unusual transparency about deviations and baseline alignment
Strong cost story with per-run accounting
Central aggregates verify against the raw table
Weaknesses
No variance reporting anywhere
Matched-budget control covers only half the model zoo
The objective's two components are never ablated
Mechanism claim is correlational
Reporting inconsistencies and an artifact asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets sub-3-bit weight quantization of small open LLMs (Qwen3-0.6B/1.7B/4B, Falcon3-1B) and asks why quantization-aware distillation (QAD) recovers short-form QA but not long-form math/code reasoning. The diagnosis (Sec 3): retention falls monotonically with required generation length — 86% on likelihood-scored QA9, 56–59% on GSM8K/MBPP, 27% on MATH-500, 16% on AMC23 across six Qwen3 settings — and failures are behavioral: for Qwen3-0.6B W2.79, 95% of MATH-500 generations exhaust the decoding budget (BF16: 27%) and 70% end in repeated 8-grams (BF16: 12%). The proposed cause is quantization-amplified exposure bias: teacher-forced supervision never covers the student's own quantized trajectories. The fix (Sec 4) is a second OPD stage optimizing Eq. (3): sampled reverse-KL on student prefixes plus a GRPO-style group-relative verifier reward, β=1 throughout. Headline claims: MATH-500 average BF16 retention 35.3%→69.7%, HumanEval 66%→91%, QA9 preserved (88.1%→92.1%).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does 'on-policy distillation for low-bit reasoning' mean in this paper?
After a quantized LLM is repaired with standard teacher-forced distillation (QAD), the authors add a second stage in which the quantized model generates its own reasoning trajectories through the exact quantized forward path used at deployment, and a frozen full-precision teacher scores every token on those trajectories (reverse-KL), plus a reward for correct final answers and passing code tests. This supervises the model on the states it actually reaches, fixing repetitive loops in long math and code generation at 2.79 and 1.88 effective bits.
How large are the improvements reported?
Averaged over four small models (Qwen3-0.6B/1.7B/4B and Falcon3-1B) and two bit widths, accuracy retention versus the unquantized BF16 model rises from 35.3% to 69.7% on MATH-500 and from about 66% to 91% on HumanEval, while nine short-answer benchmarks stay roughly unchanged (88.1% to 92.1% retention). The largest single gain is Qwen3-4B at 1.88 bits: GSM8K jumps from 17.36 to 64.59.
Does the method need a bigger teacher model?
Mostly no. For students of 1.7B parameters and up, the model's own BF16 weights serve as the teacher and match or beat 4B/8B teachers (Appendix F). The exception is Qwen3-0.6B, whose BF16 copy is too weak (teacher pass@4 of 62% versus 83–89% for larger models); the paper uses Qwen3-1.7B as its teacher and reports that larger teachers add nothing beyond that.
How expensive is the OPD stage compared to the initial quantization-aware distillation?
Much cheaper: a few hundred optimizer steps. The paper reports 57 versus 820 GPU-hours for Qwen3-4B at 1.88 bits and 11 versus 253 for Falcon3-1B at 2.79 bits — roughly 14–23× less compute than the QAD initialization stage, though that QAD stage itself (up to 820 GPU-hours) is still required first.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–6 are available only as extracted text fragments; the loop/termination curves and Figure 4/5 baseline bars could not be visually verified beyond numbers quoted in the prose.
- The promised GitHub link is absent from the extracted text, so code availability is a claim rather than a verified fact.
- Several cited works are 2026 preprints/ICML 2026 papers (EdgeRazor, Lv et al. 2026, Lee et al. 2026) whose content could not be checked from the provided text.
- Appendices appear complete through G (tables included), but any material beyond G would not have been reviewed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.