When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering
Saanvi Paturi, Arsen Kenzhebayev, Arham Sethi, Vyas Raina, Ivaxi Sheth, Vatsal Raina
cs.CL
Quick verdict
Benchmark of 500 query pairs shows unnecessary tool availability cuts LLM answer rates 98.2%→63.5%; design is clean but lacks CIs and released code.
Summary
The paper asks whether making a related but unnecessary tool available changes whether LLMs answer questions they can answer from their own knowledge. The authors build 500 query pairs across 10 domains — each pairing a tool query that needs the domain's tool with a topically adjacent closed-domain query that does not — and evaluate six models under four conditions: tool query with tool, closed-domain query without tool (baseline), with tool, and with tool after a prior tool call. The pooled baseline answer rate is 98.2%; with the unnecessary tool available it falls to 63.5%. GPT-OSS 20B is nearly unaffected (95.8→94.6) while Gemini 2.5 Flash-Lite collapses from 99.4 to 23.4 despite calling the tool in only 7.8% of trials, showing suppression is not explained by tool invocation. Item-level trajectories show a preceding tool interaction recovers 410 of 1,056 lost answers but newly loses 232. A one-sentence scope-aware system instruction recovers up to 45.6 percentage points (Llama 3.3 70B) but reduces tool use where it is needed (GPT-OSS C1 tool use 94.4→72.4). Limitations: ten single-tool domains, partially repeated items, one run per item with no confidence intervals, a same-family judge, and data/code released only on acceptance.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Merely making a related but unnecessary tool available cuts the pooled answer rate of six LLMs on otherwise-answerable questions from 98.2% to 63.5%.
- The suppression is not explained by unnecessary tool invocation: Gemini 2.5 Flash-Lite falls 76.0 points while calling the tool in only 7.8% of trials.
- A preceding successful tool interaction has mixed item-level effects: 410 of 1,056 lost answers recover, but 232 previously correct answers newly fail.
- A one-sentence scope-aware system instruction recovers most of the loss (up to +45.6 percentage points for Llama 3.3 70B) at the cost of reduced tool use where the tool is genuinely needed.
Strengths
Matched within-item control design
Each closed-domain query is identical across C2–C4 (Section 3.2), and C2 establishes per-question answerability — a control the authors correctly note BFCL and MetaTool lack (Section 2) — so the 98.2%→63.5% drop is attributable to tool context rather than item difficulty.
Decouples availability from invocation
Gemini 2.5 Flash-Lite drops 76.0 points at 7.8% tool use while Grok 4.1 Fast stays at 100% with 16.0% invocation (Table 2), cleanly separating two behaviors prior tool-use benchmarks conflate; Section 6.1's anemometer example shows refusal occurring without any tool call.
Item-level trajectory analysis exposes cancellations
Mitigation reported with its utility costs
Weaknesses
No statistical inference
Same-family LLM judge, unvalidated
Narrow, partially non-independent benchmark
Artifacts and exact strings unavailable
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a narrow, well-posed question: does the presence of a related-but-unnecessary tool change whether an LLM answers a question it can answer from parametric knowledge, and does a prior tool interaction modulate this? The stated contributions (Section 1) are: (i) a 500-pair benchmark across 10 domains with matched no-tool controls; (ii) evidence that the pooled answer rate falls from 98.2% to 63.5% when an unnecessary tool is available; (iii) an item-level trajectory analysis; (iv) a one-sentence scope-aware prompt mitigation worth up to +45.6 points. Claims are scoped to six models and ten single-tool domains, and the authors mostly respect those bounds.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What happens when an LLM is given a tool it doesn't need?
According to this paper, merely making a related but unnecessary tool available lowers the pooled answer rate of six models on otherwise-answerable questions from 98.2% to 63.5%. Gemini 2.5 Flash-Lite falls from 99.4% to 23.4% even though it calls the tool in only 7.8% of trials, showing the harm comes from the tool's presence in context, not from misusing it.
How can the answer suppression be fixed?
The authors add one sentence to the system prompt stating that tools are optional and do not limit the model's knowledge. This recovers up to 45.6 percentage points (Llama 3.3 70B), but it can also reduce tool use where the tool is genuinely needed: GPT-OSS 20B's tool-use rate on tool-relevant questions fell from 94.4% to 72.4% with the instruction.
Does a previous tool call make the effect worse or better?
Both, depending on the item. Across the benchmark, a preceding successful tool call in the same conversation recovered 410 of the 1,056 answers lost when the tool became available, but caused 232 answers that had been correct to fail. The net effect differs by model: Llama and Flash-Lite recover most, while Gemini Flash and Pro show more new failures than recoveries.
Is the benchmark or code public?
Not yet. The paper states that the data and evaluation code will be released upon acceptance, and that the full prompts, tool definitions and judge prompt are contained in the unreleased code. At the preprint stage, no artifact is independently available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures assessed from extracted text labels only: Figure 2's five-number summaries and means are present and internally consistent, but Figure 1 is purely illustrative and not analyzable.
- Prompts, tool schemas and judge prompt are deferred to unreleased code (footnote 3), so the exact experimental strings could not be audited.
- Review covers arXiv v1 preprint (Sep 2026); the paper is not yet peer reviewed and several cited comparisons (e.g., Zeng et al. 2026) are themselves preprints.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.