Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs
Saurav Bhandari, Benjamin Wade
cs.LG
Quick verdict
Preregistered study: sentiment steering survives INT8/NF4 quantization at matched effect; careful, honest, but only descriptive at k=3-4 models.
Summary
This paper empirically studies how weight-only quantization (INT8, NF4) affects activation steering in four open-weight 7-9B instruction-tuned models (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B) across two behavioral targets: RoBERTa-judged sentiment and judge-free reasoning-trace length. Using a preregistered iso-effect framework that compares capability cost (generative GSM8K, N=200) at matched behavioral effect, with five resamples per cell and REML/Knapp-Hartung pooling, the authors report that sentiment steering survives quantization: the pooled INT8 contrast is -0.010 (90% CI [-0.026, +0.007]), descriptively Equivalent under a TOST three-label rule (δ=0.03), while NF4 (-0.017, CI [-0.067, +0.033]) stays Inconclusive with high heterogeneity (I²=0.81). Reasoning length shows asymmetric dose-response: lengthening is graded until cap-runaway and collapse, whereas shortening is a step function collapsing to empty output after only 12-30% reduction. The paper contributes a methods finding: the naive E* ladder anchors on the collapse floor for floor-bounded targets; a censored construction (excluding alphas with >10% failure) restores interpretable crossings. Mistral-NF4 shows an eighteen-point unsteered GSM8K drop (0.545→0.365) replicated across two independent campaigns, reframing its nominally significant iso-effect contrast as a floor effect. Steering vectors remain highly collinear across precisions (cosine 0.989-0.998 INT8, 0.945-0.990 NF4). Main limitations: results are descriptive at k=3-4 models (the confirmatory K=12 pool is deferred), only bitsandbytes schemes and one steering method (CAA) at one layer are tested, and length capability costs are interpolation-limited.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- At matched behavioral effect, INT8 sentiment-steering capability cost is equivalent to FP16 (pooled contrast -0.010, 90% CI [-0.026, +0.007], within the δ=0.03 TOST margin), while NF4 remains Inconclusive.
- Reasoning-length steering is asymmetric: lengthening is graded but terminates in cap-runaway and collapse, whereas shortening is a step function that collapses to empty output after only 12-30% reduction.
- The naive iso-effect E* ladder anchors on the collapse floor for floor-bounded targets; a censored construction excluding alphas with >10% failure restores interpretable crossings.
- Quantization can dominate steering: Mistral-NF4 loses 18 GSM8K points unsteered (0.545→0.365), replicated across two independently collected campaigns.
- Steering directions survive quantization with high cosine similarity to FP16 siblings (0.989-0.998 INT8; 0.945-0.990 NF4), with a consistent similarity ordering across all 21 comparisons.
Strengths
Exemplary preregistration discipline and honest scoping
Appendix A's ten dated deviation entries (e.g., the MMLU→GSM8K probe switch backed by a dose-blindness experiment: flat MMLU under site=last vs 0.80→0.36 under site=all positive control), GATE C parser rescore audit (1,644 flips, 0 missing/extra, Appendix B), and explicit refusal to claim the deferred confirmatory K=12 result anywhere in the paper
A methods failure reported as a quantified contribution
§4.3/Figure 4 quantifies the contamination (worst bracketing failure rate predicts IECC, Spearman ρ=0.395; mean IECC +0.080 clean vs +0.529 contaminated) and shows the 10% censoring rule recovers FP16 acceptance from 0-2/5 to 3-5/5 while collapsing Qwen's α* sd from 18 to 8 — a reusable fix for any floor-bounded efficacy metric
Baseline shift isolated, replicated, and used against the paper's own significant result
Mechanistic corroboration of the behavioral null
Full artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether post-training weight-only quantization (bitsandbytes INT8/W8A16 and NF4/W4A16) changes the behavior of inference-time activation steering (CAA vectors, one layer per model) in four 7–9B instruction-tuned models (Qwen2.5-7B, Llama-3.1-8B, Mistral-7B-v0.3, Gemma-2-9B-it), on two targets: RoBERTa-judged sentiment (3 models) and judge-free reasoning-trace length (4 models). Scoping is unusually disciplined: all results here are descriptive at k=3/k=4; the preregistered confirmatory K=12 pool is explicitly deferred (§1, §3.7). Central claims: (i) sentiment-steering capability cost at matched effect survives quantization, with the pooled INT8 contrast at −0.010 (90% CI [−0.026, +0.007]) inside the δ=0.03 TOST margin (Table 2); (ii) length dose–response is asymmetric — graded lengthening into cap-runaway and collapse, step-function shortening collapsing after 12–30% (§4.2); (iii) the naive E* ladder fails on floor-bounded targets and a censored repair restores crossings (§4.3, Fig. 4); (iv) Mistral-NF4 loses 18 GSM8K points unsteered (0.545→0.365), replicated across campaigns (§4.4); (v) steering directions survive quantization (cosine 0.989–0.998 INT8, 0.945–0.990 NF4, §4.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does quantization break activation steering in LLMs?
According to this paper, largely no for weight-only quantization at 7-9B scale. Steering vectors extracted under INT8/NF4 remain nearly collinear with FP16 versions (cosine similarity 0.989-0.998 for INT8, 0.945-0.990 for NF4), and the capability cost of sentiment steering at matched behavioral effect is indistinguishable from full precision for INT8 (pooled contrast -0.010, 90% CI [-0.026, +0.007], inside the ±0.03 equivalence margin). NF4 remains inconclusive. Caveats: the result is descriptive over three sentiment models, and reasoning-length steering has a hard failure cliff regardless of precision.
What is iso-effect (E*) evaluation in steering studies?
Instead of comparing models at the same steering coefficient alpha, iso-effect evaluation first locates the alpha at which each system reaches the same fraction of its maximal behavioral effect, then compares downstream capability (here GSM8K accuracy) at those matched operating points. This separates shifts in steering calibration (dose needed to achieve an effect) from changes in capability cost (what steering costs once the effect is achieved). The paper shows the naive version of this ladder fails on targets with a hard failure floor and proposes excluding high-failure steering strengths before deriving levels.
Why did Mistral-7B lose 18 GSM8K points under NF4 quantization?
The paper measures unsteered GSM8K accuracy dropping from 0.545 (FP16) to 0.480 (INT8) to 0.365 (NF4) for Mistral-7B-v0.3, while the other three models stay within four points across schemes. The drop replicates in two independently collected experiment matrices. The paper does not diagnose the root cause (quantization configuration versus model-specific sensitivity) and presents it as a deployment warning: compression can dominate the steering intervention, and iso-effect contrasts must be read jointly with the unsteered baseline.
Can activation steering make an LLM reason more briefly?
Only within a narrow, risky window. Across four models, shortening steering yields at most 12-30% shorter outputs (model-dependent: Llama ~30%, Mistral ~27%, Qwen ~15%, Gemma ~12%) before discontinuous collapse to empty output — a step function with no graceful degradation. Lengthening is graded but eventually hits cap-runaway and termination failure. Clean operating windows vary by nearly two orders of magnitude across models (e.g., roughly [-6, +8] for Llama versus ±200 for Gemma), so per-alpha failure rate must be monitored directly in any production use.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction renders figure panels (Figures 2-5) as scattered axis labels and caption fragments; figure-level judgments here rest on captions and in-text numbers only.
- Appendix C/D per-cell E* crossing tables reside in the external supplementary repository, not in the provided text, so they could not be checked.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.