The Effect of Quantization on Clinical Benchmarks: Accuracy and Safety Across Model Families
Leonard Twagirayezu, Prasenjit Mitra
cs.LG · cs.CL
Quick verdict
Claims INT8 GPTQ is clinically safe but INT4 degrades model-dependently; sound factorial design, yet headline risk findings rest on n=19 and self-judging.
Summary
This empirical study runs a fully factorial evaluation of post-training quantization on clinical LLMs: five 7-8B parameter models (two general-purpose, three clinically fine-tuned) at FP16, GPTQ-INT8, and GPTQ-INT4 precision, across MedQA, MedMCQA, Med-HALT, a 150-item risk-stratified HealthBench sample, and MedSafetyBench, using greedy decoding on H100 GPUs. INT8 degradation never exceeds -1.9% on any benchmark, supporting INT8 as a deployment default. INT4 degradation is strongly model-dependent: clinically fine-tuned BioMistral-7B loses 19.7% on MedMCQA - worse than any general-purpose model - while Meditron-7B shows 0%. On the HealthBench emergency-risk subgroup (n=19), Qwen2.5-7B degrades 26.8% under INT4 versus 3.0% overall. On MedSafetyBench, model family dominates precision (FP16 refusal rates span 10.2%-74.9%), yet Qwen (-17.8%) and Meditron (-28.3%) show substantial INT4 refusal losses, indicating accuracy and safety robustness are independent. Two recovery methods, MedQA-calibrated GPTQ and QLoRA fine-tuning, recover MedMCQA while further degrading MedQA in most models. Main limitations: HealthBench relies on self-judged rubric scoring, the n=19 emergency subgroup lacks statistical testing, refusal classification is keyword-based with acknowledged limitations, and only a single GPTQ configuration is evaluated.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- INT8 GPTQ is safe across all five models and benchmarks, with degradation never exceeding -1.9% relative to FP16.
- INT4 GPTQ degradation is model- and task-dependent: clinically fine-tuned BioMistral-7B loses 19.7% on MedMCQA while Meditron-7B shows 0%, so clinical fine-tuning does not confer compression robustness.
- High-risk clinical scenarios are disproportionately vulnerable: Qwen2.5-7B degrades 26.8% on the n=19 emergency-referral subgroup under INT4 versus 3.0% overall.
- Accuracy robustness and safety robustness are independent: the most accuracy-robust model (Qwen2.5-7B) shows the largest INT4 safety degradation on MedSafetyBench (-17.8% refusal rate).
- Both recovery methods (clinical-calibration GPTQ substitution and QLoRA) recover MedMCQA while further degrading MedQA, so recovery requires task-specific validation.
Strengths
Genuine gap addressed with a fully factorial design
Fills the 7-8B edge-scale hole left by Zhan et al. (65-72B models, BitsAndBytes NF4) and the clinical-task hole in Chen et al./Kharinaev et al. general-safety work: 5 models x 3 precisions x 4 accuracy benchmarks (60 conditions) plus 15 MedSafetyBench conditions, all under one fixed PTQ method (GPTQ, group size 128).
Risk stratification surfaces effects invisible to aggregate scores
Qwen2.5-7B's emergency subgroup drop (-26.8%, Table 1 risk-wt-5) versus -3.0% overall is the paper's most clinically actionable insight, reinforced by its -9.4% subgroup delta even under INT8 - the largest INT8 effect observed anywhere in the study.
Anomaly transparency
Convergent recovery evidence from two independent methods
Deterministic, version-pinned protocol
Weaknesses
Self-judged HealthBench confounds all open-ended results
Headline claims outrun statistical power
Refusal classifier opaque at reporting level
Single quantization recipe, single run
Recovery experiments conflate training and evaluation distribution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether post-training quantization uniformly degrades clinical LLM accuracy and safety at the 7-8B scale relevant to edge deployment, and whether clinical fine-tuning confers compression robustness. Five models - Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct (general-purpose), BioMistral-7B, Meditron-7B and Llama3-Med42-8B (clinical) - are evaluated at FP16, GPTQ-INT8 and GPTQ-INT4 across MedQA (n=1,273), MedMCQA (n=1,000 stratified), Med-HALT reasoning_FCT (n=1,000), a 150-item risk-stratified HealthBench hard-split sample, and MedSafetyBench (n=450, nine AMA categories), yielding 60 accuracy conditions plus 15 safety conditions. Five central claims are advanced: (1) INT8 is universally safe (max -1.9%); (2) INT4 degradation is model- and task-dependent, with clinical fine-tuning conferring no robustness (BioMistral -19.7% MedMCQA vs Meditron 0.0%); (3) high-risk scenarios are disproportionately vulnerable (Qwen -26.8% on the n=19 emergency subgroup vs -3.0% overall); (4) accuracy and safety robustness are independent (Qwen most accuracy-robust yet -17.8% refusal loss); (5) two recovery methods trade MedMCQA gains for MedQA losses.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Is 8-bit quantization safe for clinical LLMs?
In this study, GPTQ-INT8 quantization of five 7-8B medical and general-purpose LLMs degraded accuracy on MedQA, MedMCQA, and Med-HALT by no more than 1.9% relative to FP16, and the authors recommend INT8 as a deployment default for clinical edge devices. Caveats: only GPTQ was tested, results are point estimates without formal statistical equivalence testing, and one subgroup showed a -9.4% INT8 dip.
How much accuracy does 4-bit (INT4) quantization lose on medical benchmarks?
It depends heavily on the model. Across five 7-8B models, INT4 GPTQ degradation on MedMCQA ranged from 0% (Meditron-7B) to -19.7% (BioMistral-7B), and on the open-ended HealthBench evaluation Llama-3.1-8B lost 23.0%. Notably, the clinically fine-tuned model degraded the most, so clinical fine-tuning did not confer compression robustness.
Does quantization make medical AI models less safe?
On MedSafetyBench (harmful medical requests), INT4 quantization reduced refusal rates substantially for two of five models: Qwen2.5-7B fell 17.8% and Meditron-7B fell 28.3% relative to FP16. Baseline safety was driven mainly by instruction tuning (FP16 refusal rates spanned 10.2%-74.9% across models), with precision effects secondary but non-trivial for some models.
Can the accuracy lost to INT4 quantization be recovered?
Partially, and with trade-offs. The paper tested MedQA-sample GPTQ calibration and QLoRA fine-tuning on 1,000 MedQA examples; both recovered MedMCQA performance in most models while further degrading MedQA (e.g., Llama-3.1-8B fell to -49.6% on MedQA under Med-calibrated INT4). Recovery must therefore be validated per task, not assumed.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 and Figure 2 are referenced but their rendered values are not present in the extracted text; every number cited in this review comes from Tables 1-4 or the body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.