Reinforcement Learning with Verifiable Rewards for Small Search Agents
Gaurisankar Jayadas, Aske Plaat, Álvaro Serra-Gómez, Sandheep P
cs.AI · cs.IR
Quick verdict
GRPO+search lifts Qwen3.5-0.8B EM 0.092→0.352; sparse EM reward ranks worst at matched horizon. Careful, honest ablation; one model, one dataset, three seeds.
Summary
The paper tests whether the reason-over-search recipe — RLVR over an interleaved search tool, previously shown at 3B-32B scale or via distillation — transfers to a sub-1B model trained from its own initialization. The authors train Qwen3.5-0.8B with GRPO on MuSiQue, varying only the reward shape (EM-only, token-F1-only, token-F1 with a 0.1 format floor) across three seeds each, and evaluate every checkpoint on a held-out seven-benchmark suite (51,713 rows; six of seven strictly out-of-distribution). All nine runs clear the 0.092 untrained floor; the best reaches 0.352 average EM, a 3.8-fold gain with no distillation in the training loop. At the matched horizon (step ≤180), the Search-R1-faithful EM-only reward ranks last at every seed and even on the exact-match metric it optimizes (0.271 vs 0.305/0.307); the exact rank test gives p≈0.037 one-sided. Between the two F1 shapes the ordering is seed-dependent. The authors propose but do not establish a mechanism (sparse rewards zero out group-relative advantages; per-group variance was not logged) and report emergent rollout compression under F1 rewards. Limitations: one model family and size, one training dataset, three seeds, checkpoint selection on the reported suite without a validation split, and a matched horizon of only ~0.58 epochs under which a slow-start reading of EM-only cannot be excluded.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The reason-over-search RLVR recipe transfers below 1B parameters: GRPO-trained Qwen3.5-0.8B reaches 0.352 held-out average EM vs a 0.092 untrained floor (3.8x) on a seven-benchmark suite, with no distillation in the training loop.
- At the matched horizon (step ≤180), the Search-R1-faithful EM-only reward is the worst of three shapes at all three seeds, even on the exact-match metric it directly optimizes (0.271 vs 0.305/0.307).
- A proposed but unestablished mechanism: sparse 0/1 rewards collapse within-group reward variance, zeroing GRPO advantages; supporting evidence (lowest gradient norm at every seed, entropy failing to sharpen at 2/3 seeds) is indirect.
- F1 rewards induce emergent rollout compression, hypothesized to arise from the 8192-token cap acting as an implicit length penalty; compression is a deployment-cost win, not an accuracy lever (held-out EM correlates positively with tool-call count in 7/9 runs).
- Between F1+format and F1-only the ordering is seed-dependent (+0.9, +0.9, −1.0 points at matched horizon), reported as a trade-off: F1+format for seed-robustness, F1-only for the ceiling.
Strengths
Strictly controlled single-variable design
Only the reward shape varies across nine runs; the prompt was frozen by a separate two-step ablation (6.6-fold, then ~1.7-fold lift) before any reward run, and training/evaluation render byte-identical prompts from one shared code object (Sections 4.1, 5).
Exemplary statistical honesty at n=3
Exact exchangeability test reported in three variants (pre-specified 1/27≈0.037; two-sided 0.074; unspecified 0.111), with the key admission that EM-only was pre-specified as the test arm but the direction was not — the 0.6B pilot predicted a different loser (Section 6.2).
Metric-robustness check defeats the obvious objection
Evaluation pipeline validated against published numbers
Informative behavioral decomposition of the deficit
Weaknesses
Thin external validity for a general conclusion
Checkpoint selection on the reporting suite
Horizon-conditional headline
Mechanism untested and confounded with group size
Retrieval stack deviates from reference recipes on an unlogged basis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests the reason-over-search recipe — RLVR over an interleaved search tool with an answer-match reward — at sub-1B scale, where the only prior evidence ([14], DGPO) required distillation from a larger teacher. Two claims: **C1 feasibility** (GRPO trains Qwen3.5-0.8B from its own initialization to clear the untrained floor) and **C2 reward shape** (the Search-R1-faithful EM-only reward is the worst of three shapes at every seed at the matched horizon, worst even on the metric it optimizes). The design is deliberately minimal — three reward shapes × three seeds, everything else fixed (Section 5) — in the JustRL spirit of attributing effects to a single lever. The framing is honest about scope: single-model, single-dataset findings, with the conclusion explicitly downgraded to small-model RLVR needing its own reward-design study rather than a scaled-down large-model recipe.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does RLVR work for sub-1B language models used as search agents?
According to this paper, yes. Qwen3.5-0.8B trained with GRPO and an interleaved Wikipedia search tool on MuSiQue lifts held-out average exact match from 0.092 (untrained) to 0.352 in the best run across seven QA benchmarks, with no distillation in the training loop — against earlier evidence that sub-1B search agents need to be distilled from a larger teacher.
Is an exact-match reward or a token-F1 reward better for training small search agents?
At 0.8B parameters, this paper finds the sparse exact-match-only reward is the worst of three tested reward shapes at every seed at the matched training horizon — even on the exact-match metric it directly optimizes. The two token-F1-based rewards (with or without a 0.1 format floor) win; between those two F1 variants, the ordering flips across seeds, so the paper reports a stability-vs-ceiling trade-off rather than a winner.
Why might exact-match rewards fail for small models?
The authors' hypothesis (not proven): at this scale exact matches are rare, so most rollouts in a GRPO group score 0, the within-group reward variance collapses, and the group contributes almost no learning signal even though nearly all rollouts were wrong. Their behavioral data also locates the concrete deficit in failure to terminate rollouts (22.8% abort rate for EM-only vs 5.0% for the floored F1 reward), not in worse answers once an answer is produced.
How much compute does the training require?
Roughly $120-140 of rented compute per full-epoch run on a single four-GPU RTX PRO 6000 Blackwell node (~312 optimizer steps over ~1 epoch of MuSiQue, 19,968 prompts), using the NeMo-RL framework with a FlashRAG/E5 retrieval service over Wikipedia-2018.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figs. 2-13) are image content; this review relies on their captions and the in-text/tabulated numbers rather than visual inspection of curves.
- Full text including appendices A-J was extracted, but the extraction contains OCR garbling in figure panels; no figure-only data appears to be lost since key values are tabulated.
- arXiv v1 (23 Sep 2026) single-version review; the paper's claimed reproduction of Search-R1 published numbers was not independently verified.
- Qwen3.5 is a 2026-generation model whose training provenance is not fully documented (the authors note this themselves), which limits cross-paper comparability of the 0.8B floor.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.