When Quantization Preserves Accuracy but Not Evidence: Explanation-Aware Post-Training Quantization for Medical LLMs
Yeji Kim, Mi-Young Kim, Randy Goebel
cs.CL · cs.AI
Quick verdict
PTQ keeps answers but erodes medical rationales; paper adds explanation-aware losses to OSTQuant. Gains are plausible but hinge on small sets and LLM judges.
Summary
The paper argues that post-training quantization (PTQ) of medical LLMs is typically judged by answer accuracy while the rationales users rely on may silently degrade. A diagnostic shows OSTQuant's token-level KL fidelity signal is dominated by generic discourse tokens (83.1% of signal vs a 75.3% token share) rather than clinical cues (15.6%). The authors add an explanation-aware objective to transformation-based PTQ: a two-pass calibration first trains a quantized probe and builds an offline 'faithfulness cache' of teacher-answer-supporting evidence spans (proposed by Qwen2.5-72B, filtered by forced-choice margin tests), then adds a rationale-token loss and a rationale-conditioned recovery loss to the OSTQuant objective, keeping the W4A4KV4 format and inference cost unchanged. Across four 7-8B models on MedExQA, MedExpQA, and ChallengeClinicalQA, the calibration-matched OSTQuant baseline preserves accuracy but drops MedExpQA PredRec from 90.40 (full precision) to 56.54; their method restores 88.47 and raises full-precision agreement from 48.66 to 79.47 (overall 68.44 to 79.86). FER/UCR paired differences on same-prediction MedExpQA examples favor the method (+5.34 and -4.65 points, CIs excluding zero), though most model-level CIs cross zero and BioMistral's FER is negative. An OmniQuant check shows smaller but consistent gains. Limitations: small test sets (125-940 items), reliance on GPT-5.4 judging with moderate human agreement (kappa 0.63), roughly fourfold calibration cost, and metrics partially aligned with the training objective, mitigated by control experiments.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A calibration-matched OSTQuant W4A4KV4 baseline can preserve answer accuracy while sharply degrading rationale-to-answer support (MedExpQA PredRec 90.40 to 56.54).
- Adding a teacher-rationale faithfulness cache plus rationale-token and recovery losses restores MedExpQA PredRec to 88.47 and raises full-precision agreement from 48.66 to 79.47 (overall 68.44 to 79.86).
- On MedExpQA same-prediction examples the method improves FER (+5.34, 95% CI [1.50, 9.18]) and lowers UCR (-4.65, 95% CI [-9.10, -0.28]) relative to OSTQuant.
- The objective transfers to OmniQuant with smaller gains, supporting its use as a preservation regularizer across transformation-based PTQ pipelines.
Strengths
Novel problem framing backed by a quantitative diagnostic
Table 1 measures where OSTQuant's token-level KL fidelity signal lands on teacher-forced rationales: generic/discourse tokens take 83.1% of signal vs a 75.3% token share, while clinical cues get 15.6% vs 22.2% share — a concrete, reproducible motivation for concentrating preservation pressure on evidence tokens.
Evaluation protocol goes well beyond accuracy
Section 5.2 combines answer accuracy, full-precision agreement, PredRec simulatability, FER (FActScore-style claim retention, Eq. 7), UCR (Eq. 8), and pairwise LLM judging; Appendix F adds content controls (question-only, shuffled, rationale-only, external Gemma-4-12B-IT reconstructor) that specifically guard against the PredRec/Lrec scoring overlap.
Deployment-neutral, calibration-matched comparison
Behavior-grounded cache rather than pure LLM judgment
Thorough ablations and honest negative results
Weaknesses
Small test sets with no significance testing on headline metrics
Metric-objective alignment creates a circularity risk
LLM-judge dependence with a small, potentially biased human validation
Narrow scope and a non-equivalent generalization check
Roughly fourfold calibration cost and pipeline complexity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a gap in how post-training quantization (PTQ) is evaluated: standard objectives and metrics (reconstruction, perplexity, answer accuracy) do not test whether quantized models preserve the rationales users inspect in explanation-critical domains. Claims: (1) a calibration-matched OSTQuant W4A4KV4 baseline preserves medical QA accuracy while degrading rationale-to-answer support (MedExpQA PredRec 90.40 full-precision to 56.54, Table 2); (2) an explanation-aware objective — a teacher-rationale faithfulness cache plus rationale-token (Eq. 4) and rationale-conditioned recovery (Eq. 5) losses — restores PredRec (88.47) and full-precision agreement (48.66 to 79.47 on MedExpQA, 68.44 to 79.86 overall); (3) FER/UCR paired differences confirm reduced rationale drift; (4) the objective transfers to OmniQuant with smaller gains. Scope is deliberately narrow: four 7-8B models, three medical MCQA benchmarks, one primary bit format (W4A4KV4; OmniQuant check is W4A4 fake-quant).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper mean by 'preserves accuracy but not evidence'?
After aggressive 4-bit quantization (W4A4KV4), models often keep picking the same answers — sometimes even slightly more benchmark-accurate — but the text explanations they generate can drop or invent the clinical facts that justify those answers. In the paper, the OSTQuant baseline on MedExpQA kept accuracy while its rationale-to-answer consistency metric (PredRec) fell from 90.40 in the full-precision model to 56.54.
What is the 'faithfulness cache' in EAQuant?
It is an offline table built during quantization calibration. For calibration questions where the full-precision teacher answered correctly, it stores evidence spans from the teacher's rationale. A Qwen2.5-72B model proposes candidate spans, and behavioral tests (how much each span shifts the forced-choice answer margin) decide which spans truly support the answer and which were weakened by quantization. Those weakened spans get extra weight in two auxiliary losses during quantization optimization.
Does EAQuant improve medical QA accuracy?
No, and that is not the goal. The method aims to preserve the full-precision model's answers and rationales. Accuracy stays close to full precision (e.g., 31.54 vs 31.80 average on MedExpQA), while agreement with the full-precision model's answers rises from 48.66 to 79.47. On MedExpQA the plain OSTQuant baseline even scores higher accuracy (36.80), which the authors attribute to quantization-induced answer drift rather than better behavior.
Is the code and evaluation setup available for this paper?
Yes. The paper states code and evaluation scripts are at github.com/dut0817/EAQuant, and the appendices give complete quantization settings, cache parameters, and all prompts. Reproduction requires roughly four L40S GPUs (about 12.6 GPU-hours for one Llama-3-8B calibration run) plus access to a GPT-5.4-class API for the FER/UCR judge steps.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are available only as caption/diagram text; visual layout could not be verified.
- The extracted text interleaves main body and appendices; table numbering was cross-checked but some placements may differ from the PDF.
- Several cited works are future-dated preprints (Wang et al. 2026, Gemma 4, Wee et al. 2025) that could not be verified offline.
- The GPT-5.4 judge is a proprietary, time-varying API ('as of 2026-05'); exact judge behavior at review time may differ.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.