One Policy, Any Budget: Internalizing Budget-Aware Search via Reinforcement Learning
Xiaowei Sun, Jin Li, Yili Hong, Yikun Fu, Yanghua Xiao
cs.AI
Quick verdict
AnySearch 训练单一策略以适应任意搜索预算，通过脚手架课程强化学习实现；基准测试和消融实验表现强劲，但性能提升有限且未经过充分测试。
Summary
本文提出了 AnySearch，这是一个训练单一 LLM 搜索代理的框架，使其能够在任意工具调用预算下工作而无需重新训练。该方法结合了训练脚手架（显式预算状态注入和结构化的双因素推理提示）、两阶段 GRPO 课程学习——第一阶段在脚手架激活时将预算从 5 线性退火至 1，第二阶段移除脚手架，并通过滑动窗口准确率跟踪器自适应地采样预算——以及一个复合奖励，将精确匹配准确率与正确性门控的工具效率项相结合，后者的权重随组准确率缩放。在三个主干模型（Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Qwen3-4B）的七个 QA 基准测试上，通过精确匹配进行评估，AnySearch 在 Qwen2.5-7B 上达到了 0.431 的平均 EM，而最强的 RL 基线 (StepSearch) 为 0.403；在预算 1–8 范围内（包括训练中未见过的 6–8）保持了单调递增的准确率；并且在所比较的方法中，实现了最高的工具生产率和最低的总 token 消耗。消融实验支持了脚手架内化、λ=0.6 的自适应采样以及各个奖励组件的有效性。主要局限性：预算是离散的搜索调用次数，是在静态的 2018 Wikipedia 语料库上评估的；报告了三次运行的平均值，但没有方差或显著性检验；一些单数据集的差距较小或完全持平（例如，Llama-3.1-8B 上的 NQ 与 ZeroSearch 持平，均为 0.468）。
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single policy trained with the two-phase scaffolded curriculum adapts to any tool-call budget at inference, requiring only the total budget as input.
- Budget-aware behavior is internalized: removing the scaffold in Phase II and at inference causes no loss, and re-adding it at inference gives no gain (Fig. 5a).
- AnySearch outperforms prompt-based, RAG-based, and RL-based baselines across seven QA benchmarks and three backbones, including unseen budgets 6-8 beyond the training maximum of 5.
- The adaptive efficiency weight makes early stopping rare (41/12.6k-scale cases, 0.33%) and net-positive: +417 correct answers vs 77 lost, a net +2.70% accuracy gain (Table 4).
- One AnySearch run costs about the same as one fixed-budget baseline run (~328 vs 304-336 GPU-hours, Table 7) while amortizing across all tested budgets.
Strengths
Genuinely underexplored problem with clear positioning
Section 1-2 sharply distinguish internalized budget awareness from BATS's persistent external tracker and from reward-shaping methods (OTC-PO, IKEA) that lack a budget interface; the task formulation (Sec 3.1) makes the constraint explicit via bt+1 = bt - I(at in Atool).
Complete ablation chain directly testing the internalization hypothesis
Fig. 5a covers all four scaffold schedules including Ours(+Scaffold) (no inference gain) and Only Scaffold (no training); Fig. 5b isolates λ (0/0.6/1); Fig. 5c isolates curriculum phases; Table 3 isolates each reward term; Fig. 6 shows format reward preventing reward/length/entropy collapse near step 175; Appendix D.1 consolidates three behavioral signals.
Evaluation beyond the training budget range
Honest counterfactual on the efficiency reward's risks
Cost grounding and reproducibility documentation
Weaknesses
No variance or significance testing despite three-run means
Internal inconsistency in the tool-productivity numbers
Baseline comparison partly conflates internalization with training distribution
Universal outperformance claim contradicted by exact ties on small sets
Small GRPO groups and narrow λ sensitivity check
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AnySearch (arXiv 2609.00813) targets a real deployment gap: RL-trained search agents are trained at one fixed tool-call budget and cannot adapt when that budget changes. The authors claim a single policy that, given only a total budget B at inference, allocates searches proportionally; that budget awareness is internalized into the policy (no external tracker, in contrast to BATS); and that accuracy, tool productivity, and token cost improve simultaneously. The Limitations section scopes the claims honestly: discrete search-call budgets, a static 2018 Wikipedia corpus, EM on seven QA benchmarks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AnySearch?
AnySearch is a reinforcement-learning framework that trains a single LLM search agent to work under any tool-call (search) budget: at inference the user only specifies a total budget and the policy decides how to allocate searches. It was published as arXiv 2609.00813 (2026) by researchers from Fudan University, Southeast University, and Shanghai Jiao Tong University.
How much does AnySearch improve over prior methods?
On seven QA benchmarks with Qwen2.5-7B-Instruct it averages 0.431 exact match versus 0.403 for the strongest RL baseline (StepSearch). Gains hold on Llama-3.1-8B (0.448 vs 0.424) and Qwen3-4B (0.407 vs 0.383), and accuracy keeps rising on budgets 6-8 that were never seen during training (training capped at 5), though some per-dataset results are ties.
Does AnySearch need a budget tracker at inference time?
No. Unlike BATS, which relies on an external budget-state tracker, AnySearch internalizes budget awareness during training: a scaffold (explicit budget injection plus structured reasoning prompts) is used in Phase I and removed in Phase II, and the paper's ablations show that re-adding the scaffold at inference yields no measurable gain.
Is code or training detail available for AnySearch?
The paper states code is available at github.com/xwsun01/AnySearch, and the appendix documents full hyperparameters (Table 5), both prompt templates (Appendix C), and the retrieval setup (E5-base-v2, FAISS flat index, 2018 Wikipedia dump, top-3 passages), trained with GRPO on 8×H800 GPUs.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure data (Figs. 3-8) is extracted as sparse axis-label strings, so curve values are cited from the authors' own in-text approximations (e.g., about 0.40 at B=5) rather than exact plotted points.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.