Closed-World Resolution Against Tool Hallucination in LLM Agents
cs.AI · cs.CR · cs.SE
Quick verdict
Measurement/benchmark study: closed-world resolver must precede gates to block tool hallucination; candid, but core checks are by construction and simple.
Summary
This paper measures and benchmarks a failure mode of tool-augmented LLM agents that the authors argue existing tool-selection and gating defenses structurally miss: calls to tools that do not exist, or with arguments no schema declares. It contributes a five-class taxonomy (H1-H5) split by where each error is detectable, and a deliberately minimal 'Resolution Rung' - a training-free closed-world check (registry membership plus signature type-check) that the authors argue must sit before any causal gate. Across ten hosted Bedrock models under a schema-enforced and a raw-JSON surface, they measure 322 genuine hallucinations; fabricated-tool calls concentrate on the unconstrained surface (34 vs 3), and scale does not help - a 675B model hallucinates at rates similar to an 8B one, while Claude models' low schema-surface rates are largely abstention. An MCP extension adds a second taxonomy (M1-M5) and measures 154 hallucinations on a merged multi-server surface, including from frontier models that were clean on the single-registry surface. The versioned Hallucinated-Tools Benchmark (HTB) is released. Main limitations, acknowledged by the authors: the zero-over-rejection result is a construction-consistency check on the resolver's own honest generator rather than independent evidence; rates are point estimates at 60 probes/model/surface without confidence intervals; and live validation on real catalogs remains future work.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A hallucinated tool call is a structural blind spot for any gating or contract-verification scheme, because a gate only decides about tools it deliberately exposed (Prop. 1) and a fabricated tool has no trusted contract to verify against.
- Hallucination defense must be placed strictly before any causal gate; the training-free closed-world Resolution Rung (registry membership plus signature type-check, Algorithm 1) is that placement (Props. 2-3).
- Ten hosted Bedrock models emit 322 genuine hallucinations across two invocation surfaces; fabricated-tool calls concentrate on the unconstrained raw-JSON surface (34 vs 3), and model scale does not reduce hallucination (675B comparable to 8B).
- Merging MCP servers into one flat namespace creates collision and shadowing hallucination surfaces (taxonomy M1-M5) that persist in frontier models clean on the single-registry surface, with 154 live hallucinations measured and 0 rejected-none executed under the MCP resolver.
- The released versioned HTB benchmark scores any resolver on H1-H5 and M1-M5, with baselines showing a JSON-schema validator alone leaks all fabricated-tool calls (H1 leak 1.00).
Strengths
Detectability-indexed taxonomy is genuinely useful
The H1-H5 classes (§III) are organized by where in the stack each error becomes decidable - schema errors (H1-H3, detectable part of H5) vs gating errors (H4) vs an irreducible semantic residue (H5) - which cleanly predicts which layer owns which failure and drives the whole paper's structure.
Unusual candor about what is measured vs definitional
§VI-A finding 3 explicitly tells readers to 'treat 322 as the datum and the execution rates as definitional', and Table III labels its 1.00/0.00 cells as by-construction; §X pre-emptively concedes the circular over-rejection measurement, the thin statistics, and the undemonstrated H5 handoff.
Abstention-aware rate reporting catches a real confound
MCP merge-hazard result is a clean structural demonstration
Baseline table empirically lands the thesis
Weaknesses
The headline 'no honest over-rejection' claim is circular
No inferential statistics at all
Formal results are elementary and the mechanism is trivial by design
Live evidence confined to one provider and one synthetic registry
H5 residue resolution is asserted, not demonstrated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement and benchmark paper about tool hallucination in tool-augmented LLM agents: emitted calls naming unregistered tools or carrying undeclared arguments. The central conceptual claim is a placement argument: because a causal gate only decides about tools it deliberately exposed (Definition 1, Proposition 1), a fabricated call is never a gating decision, so hallucination defense must sit at a 'rung 0' before any gate (Proposition 2). The reference mechanism - the Resolution Rung (Algorithm 1) - is registry membership plus signature type-checking; the authors explicitly disclaim its novelty ('no more than disciplined input validation', §I) and stake the contribution on taxonomy (H1-H5, §III; M1-M5, §VII), measurement, and placement. Supporting empirical claims: 322 genuine hallucinations across ten hosted Bedrock models under two invocation surfaces; fabricated tools concentrated on the raw-JSON surface (34 vs 3 H1); scale invariance; and 154 MCP hallucinations including frontier models clean on the single-registry surface.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is tool hallucination in LLM agents?
It is when a tool-calling LLM agent emits a call to a tool that was never registered, or calls a real tool with arguments its schema does not declare (for example inventing a 'wipe_disk' tool or an 'override' flag). Because such calls never went through any permission decision, a fail-open runtime simply executes them, making hallucination an action-safety problem rather than a benign parse error.
What is the 'Resolution Rung' proposed in this paper?
A training-free, closed-world check placed before any permission gate: it rejects any emitted call whose tool name is not in the registry, or whose arguments fail the tool's declared signature (undeclared or missing fields, wrong types/enums/ranges). It uses only the tool registry an agent already maintains, needs no model access, and composes with existing gating or contract-verification layers.
Does making LLMs bigger reduce tool hallucination?
According to this paper's measurements, no. On an unconstrained raw-JSON surface the 675B Mistral-Large-3 hallucinated at 0.55, essentially the same as the 8B Ministral-3 at 0.45, and the Llama-3.1 family went 0.55 (8B) to 0.57 (70B). Low hallucination rates of some frontier models under a schema-enforced API were largely refusals to answer, not safe compliance.
What is the Hallucinated-Tools Benchmark (HTB)?
A released, versioned, deterministic benchmark from this paper that scores any resolver (any allow/reject check) on the five single-registry hallucination classes H1-H5 and the five MCP multi-server classes M1-M5 with a single aggregate score. It includes external baselines (name allowlist, fuzzy-name router, JSON-schema validator, first-provider MCP hosts) and adapters for real catalogs such as API-Bank-style registries and published MCP server manifests.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction omits parts of §II (registry/signature formal definitions), the §VI-A probe-construction details, the full contents of Table IV, and the end of the conclusion; judgments rest on the remaining text.
- Figures 1-2 are known only through captions and surrounding prose; the visual data was not analyzable.
- The absence of confidence intervals and seed variation is a property of the source itself (acknowledged in §X-iii), not an extraction artifact.
- Model names/versions and Bedrock availability are as stated by the paper and were not independently verifiable from the provided text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.