Gradients Know What Outcomes Don't: Unlocking Reinforcement Learning for LLM Reasoning with Gradient-Aligned Rewards
Leqi Zheng, Jinbo Su, Fang Niu, Chaokun Wang, Weiping Wang, Jiajun Zhang, Shannan Yan, Jie Wu, et al.
cs.LG
Quick verdict
Paper claims expert-CoT gradient-cosine rewards boost RLVR math pass@k at <9% overhead; well-motivated with theory, but gains are small on tiny benchmarks.
Summary
The paper proposes Gradient-Aligned Reward (GAR), a dense process reward for RLVR that differentiates among correct rollouts. For each verifier-passing rollout, GAR runs a truncated backward pass through only the output projection layer, forms a gradient-activation vector (Eq. 3-4), and adds a bonus equal to its cosine with a cached anchor gradient computed from the expert chain-of-thought solution already shipped with NuminaMath-CoT (Eq. 5-6); the bonus is group-centered, clipped at zero, and scaled by beta=0.5. Theoretically, the cosine decomposes multiplicatively into prediction-error and activation-pattern factors (Theorem 2) and is bi-Lipschitz equivalent to output-layer NTK similarity up to conditioning (Theorem 1), with safe-shaping and variance propositions attached. Training Qwen3-4B/8B-Base without SFT warmup on ~10k NuminaMath problems (400 steps, batch 128, K=16), GAR-GRPO and GAR-REINFORCE++ beat GRPO, REINFORCE++, MASPO, Grad2Reward and G2RL on IMO-AnswerBench, HMMT 2025/2026 and AIME 2026 (10 independent runs per configuration, paired t-tests), with relative pass@1 gains up to 52.4% but absolute gains often below one problem on 30-item benchmarks. Zero-shot transfer to GPQA Diamond and MMLU-Pro is positive (+2.57/+2.41 points at 4B). Measured overhead is 8.3%. Main limitations: benchmark granularity, absence of an SFT-on-expert-CoT baseline, NTK assumptions strained by 400 RL steps, and reward bias toward expert-style solutions (alternative correct methods score 3.5x lower cosine).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Cosine similarity of truncated LM-head gradient-activation vectors against a cached expert-CoT anchor yields a dense process reward for RLVR at under 9% wall-clock overhead (8.3% measured, Fig. 5).
- The alignment cosine decomposes multiplicatively into prediction-error and activation-pattern factors (Theorem 2) and is bi-Lipschitz equivalent to output-layer NTK similarity up to conditioning and dispersion terms (Theorem 1).
- GAR-GRPO and GAR-REINFORCE++ improve over GRPO, REINFORCE++, MASPO, Grad2Reward and G2RL on every benchmark-model combination in Table 2, with the largest relative pass@1 gain of 52.4% (4B, HMMT 2025, 3.17 to 4.83).
- Rewards trained only on math transfer zero-shot to GPQA Diamond and MMLU-Pro (e.g., 4B +2.57 GPQA P@1, +2.41 MMLU-Pro over GRPO; 8B +2.28 MMLU-Pro).
- The verifier gate plus non-negative clip makes shaping safe: correct responses never fall below the outcome-only reward and strictly outrank incorrect ones (Theorem 3).
Strengths
Turns a free, already-present resource into process supervision
GAR repurposes the expert CoT solutions shipped with NuminaMath-CoT as gradient anchors (Sec. 2.3, D.2), requiring no annotation or extra model; Table 1 positions this against heuristic shaping and offline PRMs.
Well-engineered online implementation with honest cost accounting
Verifier gating confines gradient computation to correct rollouts, anchor caching reduces O(K) anchor passes to O(1) (App. C.3, D.3); measured overhead 8.3% (185s to 200s/step, Qwen3-8B, 4xA100, Fig. 5) matches the predicted 5-12% envelope (App. J) and is far below G2RL (+37.1%) and Grad2Reward (+43.7%).
Theory that characterizes the signal, corroborated empirically
Informative ablations across two optimizers
Unusually disciplined experimental protocol
Weaknesses
Headline relative gains rest on very small absolute bases
Missing natural baseline: direct use of the same expert CoT
Theory-practice gap on the NTK grounding
Reward bias toward expert-style solutions is acknowledged but only partially mitigated
Under-specified components and unreported compute
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets flat-reward credit assignment in RLVR: once multiple rollouts answer correctly, binary verification gives them identical reward and GRPO's group-relative advantage carries no signal within the correct subset (Sec. 2). GAR adds a dense bonus: for each verifier-passing rollout, hidden states are detached at the output-projection boundary and a truncated backward pass through W_o produces a gradient-activation vector (Eq. 3-4); its cosine with a cached anchor vector derived from the dataset's expert CoT solution (NuminaMath-CoT) becomes a group-centered, zero-clipped, beta=0.5-scaled bonus (Eq. 5-6). Central claims: (1) expert-CoT gradient cosine is a meaningful process signal grounded in output-layer NTK similarity (Thm. 1-2); (2) the shaping is safe and variance-increasing without bias (Thm. 3, Props. 2-4); (3) consistent pass@k gains on four math benchmarks at 4B/8B with under 9% overhead; (4) zero-shot transfer to GPQA Diamond and MMLU-Pro.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Gradient-Aligned Reward (GAR)?
GAR is a reward augmentation for RL-based LLM training (e.g., GRPO). For each model answer that passes the correctness verifier, it computes a compact gradient vector by backpropagating only through the output projection layer, and rewards how closely this vector points in the same direction as a reference vector computed from the expert worked solution already provided with the training dataset. Higher cosine similarity means the model's reasoning traces the expert's approach, so correct answers get differentiated rewards instead of identical ones.
How much extra compute does GAR add during RL training?
The paper measures 8.3% wall-clock overhead per training step over standard GRPO for Qwen3-8B on 4xA100 GPUs (185s to 200s per step), within its analytically predicted 5-12% range. This is much cheaper than the gradient-based competitors it compares against: G2RL adds 37.1% and Grad2Reward 43.7%, because GAR only backpropagates through the LM head, skips incorrect answers via the verifier gate, and caches each expert anchor per prompt group.
Does GAR's preference for expert-style solutions hurt reasoning diversity?
Partially addressed. Pass@16 — a coverage proxy — stays above baselines on every dataset and optimizer combination in Table 2, suggesting exploration is not visibly reduced. However, the paper's own analysis (Appendix M) shows rollouts using a different correct solution method than the expert receive about 3.5x lower alignment scores (mean 0.13 vs 0.45), so valid non-expert derivations systematically earn less bonus; a zero-floor clip keeps their reward non-negative, but the relative advantage still favors expert-style derivations.
What models, data, and benchmarks were used?
Qwen3-4B-Base and Qwen3-8B-Base trained from base checkpoints without SFT warmup on roughly 10,000 NuminaMath-CoT problems (400 steps, batch 128 prompts, 16 rollouts per prompt), with each problem's expert solution serving as the anchor. Evaluation is pass@1/4/16 on IMO-AnswerBench, HMMT February 2025 and 2026, and AIME 2026, plus zero-shot transfer to GPQA Diamond and MMLU-Pro, averaged over 10 independent training runs per method.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1's cell-to-column alignment is garbled in the PDF extraction (baseline rows run together), so no specific cell claims were cited from it; only its caption text was used.
- Figure 3's numeric labels interleave with body text in the extraction, so ablation conclusions were drawn from the surrounding prose rather than independently read axis values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.