HSRM: Hidden-State Reward Models for Test-Time Verification
cs.AI · cs.CL
Quick verdict
Test-time verifier (2M params) reads generator hidden states, matching a 55M text verifier in 15/16 settings; thorough ablations, no significance tests.
Summary
The paper introduces HSRM, a ~2M-parameter reward model that verifies candidate solutions for best-of-N reasoning by reading the generator's internal hidden states instead of re-encoding the generated text. A frozen LLM (Qwen3 1.7B–14B, plus Llama 3.2/3.1) samples candidates; HSRM extracts last-layer hidden states at reasoning-step boundaries (fixed delimiters such as '\n\n'), runs them through a 2-layer Transformer encoder, and scores each candidate. It is trained with a tie-safe pairwise ranking loss on self-generated trajectories labeled only by final-answer correctness, requiring no human process supervision. Across GSM8K, MATH-500, AIME, and OlympiadBench, HSRM matches or beats a reproduced 55M-parameter text-only energy verifier (EORM) in 15 of 16 generator–dataset settings, and approaches a 7B process reward model on GSM8K while using roughly five orders of magnitude fewer verification FLOPs per candidate. Ablations attribute the gain to the hidden-state input rather than capacity, show top-4-layer concatenation works best, and find that post-'</think>' extraction beats full-trace reading in thinking mode. Limitations: math-only evaluation, white-box requirement, a stronger 7B PRM still wins on harder benchmarks, modest AUROC, no statistical significance tests, and one internal numerical inconsistency (Table 9 vs Table 10).
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 0 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A ~2M-parameter verifier reading frozen-generator hidden states at reasoning-step boundaries matches or outperforms a 55M-parameter text-only energy verifier in 15 of 16 generator–dataset settings across four math benchmarks.
- The advantage comes from the hidden-state input representation rather than verifier capacity (Table 1 text-only 2.56M: 82.23%/0.519 AUROC vs hidden-state 2.12M: 86.11%/0.691; Table 10 shows encoder choice matters little).
- HSRM approaches Qwen2.5-Math-PRM-7B on GSM8K while using ~3,500× fewer parameters and roughly five orders of magnitude fewer verification FLOPs per candidate (Sec. 6.3, Table 11).
- In thinking mode, restricting extraction to post-</think> representations improves within-problem AUROC from 0.669 to 0.736 (Qwen3-1.7B) and 0.672 to 0.884 (Qwen3-4B) on MATH-500 (Table 2).
- The verification signal transfers zero-shot across math datasets (Fig. 6) and across generator families, with HSRM beating matched EORM in all six Llama settings (Table 3, e.g., GSM8K/Llama-3.2-1B: 48.4 vs 40.3).
Strengths
Structurally grounded efficiency claim with quantified cost analysis
Sec. 6.3 and Table 11 ground the efficiency claim in mechanism: HSRM reads cached step-boundary states (0 extra generator passes, S≤100 inputs) vs re-encoding T text tokens with a 53.4M encoder or a 7.6B PRM; Fig. 4 plots the accuracy–FLOPs frontier across four generator scales.
Capacity-vs-modality confound is explicitly controlled
Table 1 (GSM8K/Qwen3-1.7B): a 2.56M from-scratch text encoder reaches 82.23%/0.519 AUROC vs 86.11%/0.691 for the 2.12M hidden-state model; Table 10 shows DeepSet (0.40M) ≈ default Transformer (2.12M), isolating modality as the causal factor.
Unusually thorough ablation coverage
Honest cheap-baseline envelope
Transparent labeling and split documentation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 0 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HSRM and how does it verify LLM reasoning?
HSRM is a ~2M-parameter reward model for best-of-N reasoning. Instead of a large language model re-reading each sampled solution as text, HSRM takes the generator's own internal hidden states, sampled at reasoning-step boundaries during decoding, and runs them through a small 2-layer Transformer encoder that outputs a score per candidate. The highest-scoring candidate is returned. It is trained only from self-generated solutions labeled by final-answer correctness.
How much cheaper is HSRM than a 7B process reward model?
Per the paper's Section 6.3 and Table 11, HSRM uses about 3,500× fewer parameters than Qwen2.5-Math-PRM-7B (≈2M vs ≈7.6B) and roughly five orders of magnitude fewer verification FLOPs per candidate, because it reads cached hidden states already computed during generation rather than re-encoding each full solution with a large model. On GSM8K it roughly matches the 7B PRM's best-of-8 accuracy, though the PRM stays ahead on harder benchmarks like AIME.
Can HSRM be used with API-only LLMs like GPT or Claude?
No. HSRM requires white-box access to the generator's internal hidden states during decoding. It works with open-weight models (the paper uses Qwen3 1.7B–14B and Llama 3.2/3.1), but it cannot be applied to closed API-only models that do not expose internal activations; text-based verifiers remain the option in that setting.
Does HSRM work outside mathematics?
Not in this paper. All four benchmarks (GSM8K, MATH-500, AIME, OlympiadBench) are mathematical reasoning tasks. The authors' Limitations section states that extending the approach to other domains and a broader range of architectures is future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Headline per-cell results for Figure 2 (and FLOPs points in Figure 4) exist only as plotted curves with axis labels in the extraction, so the exact 15-of-16 per-cell outcomes cannot be independently re-verified from tabulated numbers; all other cited figures come from tables or in-text values.
- The extracted text is complete through Appendix F with references present; no truncation detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.