Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Yiwei Yang, Haoxiang Zhang, Bingbing Wen, Yao Lu, Yuchen Wu, Lei Zhang, Julian McAuley, Pan Lu, et al.
cs.CL
Quick verdict
Controlled study: RL agents learn cue-driven spurious tool use (to +39.2%); judge-based necessity reward fixes it, but evidence is single-seed, small-scale.
Summary
This paper identifies a shortcut-learning failure mode in RL-trained LLM agents: spurious tool selection driven by superficial prompt cues rather than task requirements. The authors build controlled synthetic environments from Natural Questions (factual QA aided by web search) and DeepMath-103k (math aided by Python), injecting cues correlated with a specific tool during training but causally irrelevant to tool necessity, and measure cue-driven tool-call inflation via counterfactual evaluation (cue present, tool not needed). Training Qwen2.5-7B-Instruct with GRPO in VerlTool, search-semantic cues induce spurious search calls on math tasks up to +39.2% on GSM8K, while code-semantic cues produce almost no spurious Python use. The asymmetry tracks learning progress — NQ accuracy rises while DeepMath stays flat — and swapped-cue experiments show task competence enables shortcut formation while cue-tool semantic alignment modulates its strength; shortcuts largely vanish below ~80% cue-task correlation. As mitigation, a dense decision-level reward — a GPT-5 Nano judge penalizing each unnecessary call by 0.5 — eliminates spurious use while preserving or improving accuracy. Main limitations: single seed per run, one 7B base model, 1,000 training examples, artificial cues, an unvalidated LLM judge, and unexplained variance among repeated no-training baselines.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- RL-trained agents form cue-driven spurious tool-selection policies, raising spurious tool invocation on tasks that do not need the tool by up to +39.2% (GSM8K, [WEB INDEX] cue, Table 1).
- Across the tested conditions, shortcuts form only for tools the agent has already learned to use reliably, linking task competence to shortcut vulnerability (Section 5.3, Figure 2).
- Swapped-cue experiments show semantic alignment between cue and tool modulates shortcut strength: ≤ +3.5% for misaligned cues versus up to +39.2% for aligned ones (Tables 3-4).
- Spurious tool use scales with cue-task correlation and largely disappears below ~80% (Table 5).
- A dense tool-necessity reward from an LLM judge eliminates cue-driven tool use (∆ ≤ +0.3) while preserving or improving task accuracy (Table 1).
Strengths
Genuinely causal, counterfactual design
Counterfactual evaluation groups hold each question fixed and vary only cue presence, so ∆Tool_Y-N (Section 2.2) isolates the causal effect of the cue on tool selection rather than a correlational association; Definition 1 gives a precise, testable criterion including the no-utility condition.
Confound-disentangling controls
The swapped-cue experiments (Tables 3-4) cleanly separate task competence from semantic alignment: misaligned cues on the well-learned NQ task yield only ∆Search ≤ +3.5%, versus up to +39.2% for aligned cues on the same evaluation sets (GSM8K, Table 1), and search cues on the unlearned math task yield ∆Py ≤ +3.2%.
Dose-response ablation with a threshold finding
Mitigation validated inside the training loop
Unusually complete reproducibility inputs
Weaknesses
Single seed, no variance quantification
Unexplained baseline instability
Competence claim rests on one learned and one failed task
Unvalidated LLM judge as sole reward and evaluator
Limited external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper studies a specific failure mode in RL-trained LLM agents: cue-driven spurious tool selection, where a superficial prompt feature (a `[WEB INDEX 742]` prefix, an `<answer>` format instruction) becomes spuriously associated with a tool (web search, Python) during GRPO training, so the agent invokes the tool on tasks that do not need it. Unlike efficiency-oriented work on excessive tool calls, the target is *why* a tool is selected. Definition 1 formalizes this: cue presence inflates selection probability (π(Ts|xc,a) ≫ π(Ts|xc)) while the tool confers no task advantage. The central claims are: (1) shortcuts arise reliably under outcome-only reward, up to +39.2% spurious search on GSM8K (Table 1); (2) they form only for tools the agent has already mastered (Section 5.3); (3) cue-tool semantic alignment modulates strength (Tables 3-4); (4) a per-decision tool-necessity reward from an LLM judge suppresses them without accuracy loss. Claims are scoped honestly to the synthetic setting.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'spurious tool use' in RL-trained LLM agents?
It is when an agent calls a tool (e.g., web search) because a superficial prompt cue — such as a [WEB INDEX 742] tag or an <answer> format instruction — became correlated with that tool during reinforcement-learning training, not because the task actually needs it. The unnecessary call often does not hurt final accuracy, so standard answer-only evaluations miss it. In this paper's experiments, such shortcuts inflated unnecessary search calls on math tasks by up to +39.2%.
Does the proposed tool-necessity reward reduce spurious tool calls, and does it hurt accuracy?
According to Table 1, adding the penalty from a GPT-5 Nano judge (−0.5 per unnecessary call) drops cue-driven search-call inflation to roughly zero (for example, ∆Search goes from +39.2% to +0.0% on GSM8K for the [WEB INDEX] cue) while accuracy matches or exceeds the no-cue baseline (e.g., 91.8% vs 89.2% on GSM8K in the <answer> condition).
Why did no code-tool (Python) shortcut form in these experiments?
The agent never learned to use Python effectively on DeepMath — validation accuracy stayed flat around 45-50% over 30 training steps — whereas it did learn to use search on NQ (rising from ~45% to ~75%). The authors conclude that shortcut formation requires existing task competence, and swapped-cue experiments further show that semantic alignment between the cue and the tool modulates how strong the shortcut becomes.
Can this paper's results be reproduced?
Partially. The paper fully specifies the base model (Qwen2.5-7B-Instruct), GRPO hyperparameters (Table 6), cue designs, and both judge prompts (Appendix A.3), all built on public datasets with the open-source VerlTool framework. However, each condition used a single seed on 4×H100 GPUs and code is promised only upon acceptance, so the qualitative effects should replicate but exact numbers may not without multiple seeds.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 are images and could not be inspected; accuracy-curve claims (~45→75% NQ, flat ~45-50% DeepMath) are taken from the text description only.
- This is an arXiv v1 preprint (2609.16268, Sep 2026), not yet peer reviewed.
- Tables were extracted from plain text; a minor column-alignment risk exists, though values appear internally consistent across sections.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.