Learning to Solve Hard Problems in RL for LLMs by Never Giving Up
Michael Noukhovitch, Hamish Ivison, Nathan Lambert, Aaron Courville
cs.LG · cs.AI
Quick verdict
声称 RL 在简单 prompt 上浪费了算力（马太效应）；NGU 自适应采样解决了这个问题——消融实验充分，计算资源匹配，但核心收益有限。
Summary
该论文记录了 LLM 强化学习后训练中的“马太效应”：增益集中在基础模型已经能解决的问题上，其中简单 prompt 的提升远高于困难 prompt，这一规律在数学 (AIME)、代码 (LCBv5) 和智能体编程 (SWE-Bench) 模型对中均成立 (Figure 1)。在使用 Qwen2.5-0.5B GSM8k Platinum 测试平台时，作者表明在固定批次大小下增加每个 prompt 的补全数 K 并不能解决问题——K=4 的表现甚至优于 K=32，即使在困难子集上也是如此 (Figure 2)——并提出了“信号效率”的观点：固定 K 的 GRPO 在简单 prompt 的过度采样上浪费了计算资源。他们的补救措施 Never Give Up (NGU) 会持续为某个 prompt 采样 K 个补全，直到出现正确的补全，并以概率 p_NGU 放弃；通过异步强化学习 (RL)，这能低成本地过滤掉简单 prompt，并将计算资源重新分配给困难 prompt。消融实验建立了最佳实践：从损失中过滤掉早于 T≈4 步的 rollout，但将其保留在基线中，并通过“锚定积极样本”进行重缩放，而不是像 Reinforce-Ada 那样进行降采样 (Figure 6, Table 1)。在 Qwen3-4B 的 Deepscaler 上，计算资源匹配的 NGU 将困难子集 pass@1 的增益从 +2.5 提高到了 +4.3，平均值从 25.3 提高到了 26.5 (Tables 2-3)。在 Manufactoria 编码任务中，NGU 配合每个测试的奖励学会了通过所有测试，而 GRPO 则陷入停滞 (Figure 9)。局限性：模型相关的难度定义、总体增益适中、规模仅为 0.5-4B、仅限二元奖励。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RL post-training of LLMs exhibits a Matthew Effect: performance gains scale with the base model's initial competence, across math, code, and agentic domains (Figure 1).
- At fixed total batch size, increasing completions-per-prompt K does not improve hard-problem performance — K=4 beats K=32 — so signal loss is not the primary driver; compute misallocation (signal inefficiency) is (Figure 2).
- Never Give Up, an adaptive sampler that keeps drawing K more completions until one is correct (giving up with probability 1−p), combined with asynchronous RL, staleness filtering (T≈4), and anchoring-the-positives baselining, improves performance per compute especially on hard problems (Figures 4-6, Table 1).
- On Deepscaler with Qwen3-4B at matched compute, NGU improves hard-subset pass@1 gains (+4.3 vs +2.5) and the AIME+BRUMO average (26.5 vs 25.3) over fixed-K GRPO (Tables 2-3).
- On Manufactoria, per-test reward + NGU learns to pass all tests where standard GRPO plateaus, and matches the efficiency of the all-tests-reward two-stage recipe (Figures 9-10).
Strengths
Clean cross-domain demonstration of the phenomenon
Section 2 / Figure 1 compares RL-trained vs base models in three domains (Olmo 3.1 RL-Zero on AIME, DeepCoder on LCBv5, DeepSWE on SWEBench), showing gains concentrated on easy buckets and near-zero on the hardest — establishing the Matthew Effect before proposing anything.
Genuinely counterintuitive and useful negative result on K
Section 3 / Figure 2: at fixed N×K=256, K=4 outperforms K=8/16/32 even on hard subsets, directly undermining the 'just raise K' (signal loss) remedy and motivating the signal-efficiency reframing.
Ablations yield actionable best practices, not just a method
Compute-matched scaling comparison with curriculum baselines
Honest scope conditions
Weaknesses
Difficulty construct is model-relative and partly circular
Headline gains at scale are modest and subset-driven
Statistical reporting is thin
Narrow empirical scope
Internal inconsistency and infra dependence hurt replication
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes one observational claim and one prescriptive claim. Observationally, RL post-training exhibits a 'Matthew Effect': gains scale with the base model's initial competence (Section 2, Figure 1 — Olmo 3.1 RL-Zero gains +0.63 on the easiest AIME bucket vs +0.03 on the hardest; similar gradients for DeepCoder/LCBv5 and DeepSWE/SWE-Bench). Prescriptively, the cause is not 'signal loss' (zero-gradient groups) but compute misallocation, reframed as the signal efficiency hypothesis (Section 3), and the Never Give Up (NGU) sampler — keep drawing K more completions until one is correct, give up with probability 1−p — fixes it (Section 4). Three claims carry the paper: (1) at fixed N×K, larger K does not help hard problems (K=4 beats K=32, Figure 2); (2) NGU + async RL + anchoring positives is the best configuration (Figure 6, Table 1); (3) NGU transfers to 4B scale (Deepscaler, Section 5) and to a new difficulty axis — tests within a prompt (Manufactoria, Section 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Matthew Effect in RL for LLMs?
It is the pattern documented in this paper: RL post-training improves an LLM in proportion to its initial competence — problems the base model already solves improve a lot, while the hardest problems barely improve. The authors show it across math (AIME), code (LiveCodeBench v5), and agentic coding (SWE-Bench) pairs of base and RL-trained open models.
How does the Never Give Up (NGU) sampling method work?
NGU starts with a small group of completions (e.g., K=4) per prompt. If all are correct, the prompt is filtered as too easy; if at least one is correct, it trains normally; if all are wrong, it samples K more completions instead of discarding the prompt — continuing with probability p (e.g., 0.95) and giving up otherwise. With asynchronous RL, this spends fewer samples on easy prompts and more on hard ones.
Does NGU require special infrastructure?
Yes. The paper's own ablation shows NGU with synchronous RL underperforms plain GRPO (extra-hard pass@1 of 14.8 vs 47.1 on their GSM8k testbed), because quickly filtering easy prompts only pays off if the freed compute immediately serves other prompts. The benefits rely on an asynchronous training stack with in-flight weight updates.
What results does NGU achieve?
On Deepscaler math with Qwen3-4B at matched compute (~120 H100-hours per setting), NGU raises the hard-subset pass@1 improvement from +2.5 (best fixed-K baseline) to +4.3 and the AIME+BRUMO average from 25.3 to 26.5. On Manufactoria coding with Qwen3-4B-Instruct, GRPO with per-test rewards never passes all tests of a single problem, while NGU learns to fully solve problems.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels are images: only captions and garbled axis-label fragments were extracted, so curve-level judgments rest on captions and Tables 1-3 rather than plotted values.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.