The Menu Is an Execution Prior: State-Path Tool Menus for Online Agents
cs.AI
Quick verdict
Paper claims state-path tool menus lift ToolBench success 0.737→0.898; convincing controlled gains, but single-run LLM-judged scores lack variance estimates.
Summary
The paper addresses how LLM agents can act reliably when tool libraries contain thousands of APIs. It defines the 'tool menu' — a short, ordered, request-specific subset of tools fixed before execution — and argues that relevance-ranked menus surface the final action while omitting prerequisite 'bridge' tools. The authors propose the state path, a pre-execution route from the observable request state to the outcome, and State-Path Tool Menu: an encoder producing state- and relation-aware tool representations, a retriever selecting 32 tools covering entry, bridge, target, and terminal roles, and a reranker ordering producers before consumers. Trained on successful ToolBench trajectories, it raises ToolBench online success from 0.737 (official menu) to 0.898 with the same Qwen2.5-72B-AWQ executor and 8-call budget, winning 50 of 51 disagreements. Ablations attribute +3.6 points to reordering the official tools and a further +12.5 to learned coverage plus order; gains grow with reference-path length and persist across three executor families, four additional benchmarks, and three independent evaluators. Chain-coverage diagnostics connect the gain to the mechanism. Limitations: single deterministic runs without variance estimates; reliance on an LLM success judge; all published baselines re-engineered into a static 32-tool interface (each scoring below the official menu); a fixed menu that cannot react to observations; and dependence on informative schemas and recurring training paths.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Relevance-ranked menus surface the final action while omitting or mis-ordering producer tools, so route completeness — not individual relevance — is the binding objective for bounded menus.
- Learning state-path membership and producer-before-consumer order raises ToolBench online success from 0.737 to 0.898 with agent, prompt, 8-call budget, and evaluator unchanged.
- A 32-tool State-Path menu achieves higher complete-chain coverage than the official 128-tool list (0.704 vs 0.691).
- The gain persists across three executor families (+16.1/+12.8/+6.9 points), the primary measures of four additional benchmarks, and three independent LLM evaluators.
- Structured input/output fields, not prose descriptions, carry the dependency signal (Table 9: +4.8 edge accuracy and +5.0 replay success over names alone).
Strengths
Mechanism-linked ablations
Table 6 cleanly separates order from coverage: reordering only the official 32 candidates gives +3.6 (0.737→0.773), a single fixed path template also reaches 0.773, full learned method 0.898. Fig 3 shows the coverage gap widening with reference-path length; Table 7 gives a monotone dose-response (0.510→0.704 chain coverage) with a corruption control losing only 1.0 point at 50% corruption.
Controlled paired online evaluation
Same executor (Qwen2.5-72B-AWQ), 8-call budget, menu-neutral prompt, timeout handling, and official ToolEval across all rows (§A.2). Of 51 paired disagreements, State-Path wins 50 and loses 1 (Table 4); three independent judges (GPT-3.5-turbo-16k, GPT-4o-mini, GPT-5.5) each rank it first over 400 traces (Table 5).
Diagnostics tie the gain to the proposed mechanism
Practical cost profile and honest nuance
Weaknesses
No variance estimates or significance testing
Baseline adaptation fidelity unverified
Key design knobs never swept
Figure 3 caption contradicts its own data
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets one decision point in LLM agent systems: the pre-execution tool menu — which subset of a large API library is exposed to the executor, and in what order. The authors define the 'tool menu' (a request-specific, ordered subset fixed before the first call; the agent may only call menu tools), argue that relevance-ranked menus surface the final action while omitting producer tools for its inputs, and propose the *state path*: a pre-execution route ENTRY → BRIDGE* → TARGET [→ TERMINAL] starting from the observable request state. Their method, State-Path Tool Menu, learns route membership and displayed order jointly. Central claims: (1) state-path completeness, not relevance, is the binding objective for bounded menus; (2) the learned menu lifts ToolBench online success from 0.737 to 0.898 with agent, prompt, 8-call budget, and evaluator unchanged; (3) 32 state-path-selected tools beat the official 128-tool list at full-chain coverage (0.704 vs 0.691); (4) gains persist across evaluators, executor families, and four additional benchmarks. Scope is honest: the menu is explicitly a one-shot pre-execution decision, with online updates left open (§6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'tool menu' in this paper?
A short, ordered subset of an agent's available tools — 32 out of a library of thousands — that is constructed and fixed before execution begins; the agent may only call tools that appear on the menu. The paper argues the menu acts as an execution prior.
How much does State-Path Tool Menu improve results?
On 304 ToolBench tasks, with the same executor (Qwen2.5-72B-AWQ) and 8-call budget, online success rises from 0.737 with the official menu to 0.898 with the State-Path menu. The method also leads the primary metric on AppWorld (chain 0.465), TRAJECT-Bench (full chain 0.732), UniToolCall (ordered prefix 0.635), and ToolHop (first action 0.683).
Is code or data available?
The paper links a GitHub repository (github.com/Met2348/State-Path) and releases extensive implementation detail: architecture sizes, training seeds, decoder constants, the agent prompt, and a taxonomy of baseline adapters.
What are the paper's main limitations?
The menu is fixed once before execution and cannot react to observations; results are single deterministic runs under an LLM-based success judge with no variance estimates; every published baseline was re-engineered into a static 32-tool interface; and the method depends on informative tool schemas and recurring training paths.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 3's bar values (83.2/90.4/36.4/65.7/18.8/45.0) and axis labels bleed into the extracted text as garbled inline content near the Table 7 discussion; group pairing follows the legend and caption, and all other numbers cited in this review come from clean table text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.