OBC-Prune: Outcome-Based Calibration for Large Reasoning Model Pruning
Ha Lan Nguyen, Huy Hoang Tran, Trac-Duy Tran, Dung D. Le
cs.AI
Quick verdict
Outcome-stratified causal calibration improves one-shot LRM pruning across three solvers; extensive results but single-seed, no significance tests.
Summary
The paper proposes OBC-Prune, a calibration framework for one-shot pruning of large reasoning models (LRMs). It samples 16 rollouts per problem from the dense model, keeps 128 problems where at least one rollout is correct and one incorrect, and pairs a median-length correct rollout with the longest near-miss incorrect rollout. An attention-suppression intervention measures each sentence's causal effect on subsequent predictions (mean KL divergence, with a correction for earlier sentences' positional advantage); softmax-normalized effects become per-token weights that upweight correct-rollout tokens (1+beta*e) and downweight wrong-rollout tokens (max(eps,1-gamma*e)). These weights rescale calibration activations (via sqrt(w)) entering the second-order statistics of SparseGPT, Wanda, and ALPS, leaving the solvers unmodified. On DeepSeek-R1-Distill-Qwen 1.5B/7B/14B at 40-50% sparsity, OBC-Prune matches or beats C4, RAC, and SSGR in nearly all cells on MATH500, LiveCodeBench, and AIME-25 (e.g., 14B LiveCodeBench 50%: 0.481 vs SSGR 0.358 under SparseGPT), with gains growing with sparsity and generalizing to Qwen3 and R1-Distill-Llama models. Pruned models also track dense hidden states better, collapse (non-termination) less often, and generate shorter completions. Main limitations: only binomial SEs are reported (no significance tests, single calibration seed), many margins and all ablation deltas fall within one SE, calibration is math-only (N=128) despite code evaluation, and pair-construction heuristics are not ablated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Stratifying one-shot pruning calibration by outcome via difficulty-matched correct/incorrect rollout pairs improves accuracy over uniform on-policy calibration (RAC, SSGR), with gains widening as sparsity increases.
- An attention-suppression intervention yields sentence-level causal scores that, converted to per-token activation rescalings, improve SparseGPT, Wanda, and ALPS without modifying any solver.
- OBC-pruned models track dense hidden states more faithfully, fail to terminate their chain of thought less often, and generate shorter completions than baseline-calibrated models.
- The advantage generalizes beyond R1-Distill-Qwen to Qwen3 and R1-Distill-Llama-8B families and is robust to beta/gamma/epsilon choices and to the softmax/min-max normalization design.
Strengths
Solver-agnostic integration mechanism
Stage 4 (Sec 4.4, App A) injects outcome weights purely by scaling activations with sqrt(w_t) before the second-order statistic, so SparseGPT's Hessian, Wanda's per-channel energy, and ALPS's Gram matrix are all re-weighted with zero solver changes — a clean, transferable design validated empirically on all three backends (Tables 1-3).
Unusually broad evaluation for a calibration paper
Seven models across three families (R1-Distill-Qwen, Qwen3, R1-Distill-Llama-8B), two sparsity levels, three benchmarks, a 10-90% sparsity sweep (Fig 4), a hyperparameter sweep (Table 5), and a component ablation (Table 6) — most calibration papers stop at Table 1.
Mechanistic evidence beyond accuracy (RQ3)
First routing of interventional sentence scoring into a pruning objective
Weaknesses
Statistical support is thin for the headline claim of consistency
Ablation deltas cannot be distinguished from noise
Math-only calibration vs code evaluation, no decontamination discussion
Pair-construction heuristics are never ablated
Runtime accounting omits calibration cost; SE computation is unclear
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is OBC-Prune and how does it work?
OBC-Prune is a way to choose calibration data for one-shot pruning of large reasoning models (models that think in long chain-of-thought before answering). It finds 128 problems the model sometimes solves and sometimes fails, pairs a correct reasoning trace with an incorrect one, uses attention-suppression experiments to measure which sentences causally matter to the outcome, and then upweights activations behind correct reasoning and downweights those behind failed reasoning inside standard pruners (SparseGPT, Wanda, ALPS) — with no retraining.
How much accuracy do the pruned models retain with OBC-Prune?
At 50% sparsity on MATH500 with the SparseGPT backend, accuracy is 0.706 for the 1.5B model, 0.772 for 7B, and 0.866 for 14B, versus dense-model accuracies of 0.802, 0.886, and 0.914 respectively — consistently above the C4, RAC, and SSGR calibration baselines, with the gap widening as sparsity increases.
Does OBC-Prune require retraining or fine-tuning the model?
No. It is a training-free, post-training method: it needs only the model itself to sample 16 rollouts for each of 128 calibration problems plus per-sentence suppression passes (done on a single H100), and it modifies only the statistics the pruning solver consumes, not the solver or the weights beyond pruning.
Which models and benchmarks was OBC-Prune evaluated on?
DeepSeek-R1-Distill-Qwen at 1.5B/7B/14B as the main testbed, plus Qwen3 (1.7B/8B/14B) and DeepSeek-R1-Distill-Llama-8B for generalization, evaluated on MATH500 (accuracy), LiveCodeBench (pass@16), and AIME 2025 (accuracy) at 40% and 50% sparsity.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 were extracted as text only: Figure 1's actual error values are not recoverable, and Figure 4's sweep values are known only from prose approximations (e.g., 'roughly 6% at 70%' at 1.5B).
- The abstract in the extraction is truncated mid-sentence; all quantitative judgments are based on the complete tables and appendix text, which extracted cleanly.
- No code or artifact release is mentioned anywhere in the extracted text, so codeAvailable=false reflects absence of a statement rather than an explicit refusal.
- Equation formatting is mangled by extraction (norms, subscripts), but Equations 1-8 were interpretable from context.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.