Toollery: Scaling LLM Agents to Thousands of Skills and Tools
cs.LG · cs.AI · cs.CL
Quick verdict
Toollery uses generated intent queries for tool retrieval; big recall gains at 79K scale, honest cost analysis, but key end-to-end wins rest on private data.
Summary
Toollery is a training-free candidate-compression framework addressing the cost and accuracy degradation of full-library skill/tool prompting in LLM agents. Offline, a generator LLM (DeepSeek-V4-Pro) produces three user-intent queries per capability specification, filtered by a source-capability recovery check against eight distractors; these form a retrieval 'manual'. Online, a standard retriever (BM25, RAGAnything/dense Qwen3 embeddings, or RRF hybrid) maps each user request to a compact top-k candidate set that the final LLM reasons over. On the ~79K-capability SkillRouter benchmark (75 queries per pool), manual+specification BM25 raises Hit@1 from 0.553 to 0.920 and FullCoverage@10 from 0.413 to 0.853 over raw specification retrieval, beating the dedicated SkillRouter reference retrievers. On BFCL-V4, raw top-10 retrieval is already near saturation: end-to-end AST accuracy improves only from 0.825 to 0.830 (non-live) and 0.834 to 0.837 (live), neither significant. The largest end-to-end gain comes from a proprietary, unreleased dataset of 3,396 smart-cockpit requests (exact-set accuracy 0.163 to 0.268). Strengths include cache-aware cost replay showing no billed-cost advantage under natural cache hits, an overlap audit supporting test isolation, and detailed ablations. Limitations: the headline accuracy claim rests on private data, manual-only retrieval underperforms specification retrieval on smaller tool pools, and code release is promised only upon acceptance.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Offline LLM-generated user-intent queries (a 'manual') plus online retrieval compress capability libraries from hundreds to ~79K entries into a fixed top-k candidate set before final LLM selection, at $0.22 offline cost per 100 capabilities.
- On the ~79K-capability SkillRouter benchmark, intent-augmented BM25 raises Hit@1 from 0.553 to 0.920 and FullCoverage@10 from 0.413 to 0.853 versus raw specification retrieval, beating the dedicated SkillRouter reference retrievers.
- On BFCL-V4, where raw top-10 retrieval is near saturation, Toollery maintains comparable AST accuracy (.830 non-live, .837 live vs .825/.834) while cutting prompt context from 57.0K-84.1K to 1.3K-2.1K tokens and latency from 8.4-11.8s to 1.5-1.6s.
- On a proprietary 3,396-request smart-cockpit dataset over 220 tools, Toollery raises exact-set accuracy from 0.163 to 0.268 and FullCoverage@10 from 0.224 to 0.381 at a fixed top-10 budget.
- Monetary savings from candidate compression depend on provider cache state: no billed-input-cost advantage under natural replay (98.9% cache hits), but 97.5% billed savings when caching is suppressed.
Strengths
Unusually honest cost and null-result reporting
Table 15 shows no billed-input-cost advantage under natural replay (delta $0.000070, 95% CI [-0.000524, 0.000462], 98.9% full-library cache hits), and Section 4.3 states plainly that BFCL end-to-end gains are not significant.
Large-scale retrieval gains against a strong reference
Table 1: at ~79K capabilities, manual+specification BM25 reaches Hit@1 0.920 / Recall@10 0.942 / FullCoverage@10 0.853 vs raw BM25 0.553/0.571/0.413 and the dedicated SkillRouter† retrievers 0.740/0.704/0.520, in 623 ms/query.
Test-isolation discipline
Thorough ablations
Actionable failure analysis
Weaknesses
Headline accuracy rests on private data
Cross-model main comparison
Manual-only retrieval contradicts the intent-mismatch narrative
Latency comparisons conflate representation length with method value
Baseline and artifact availability
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Toollery (arXiv 2609.22218; Huawei 2012 Laboratories) targets a real production pain point: prompting an LLM agent with an entire skill/tool library costs tokens and latency linear in library size and adds distractors. The paper unifies skills, MCP services, atomic tools, and API functions as 'capabilities' and adds a training-free candidate-compression layer: offline, a generator (DeepSeek-V4-Pro) writes N=3 user-intent queries per specification, optionally filtered by a recovery check (a verifier must recover the source capability among 8 random distractors); online, a standard retriever maps each request to a compact top-k candidate set over which the final LLM decides. The claims, as hedged in the abstract: bounded O(k) online cost, higher recall than specification retrieval, significant end-to-end gains on a cockpit dataset, parity on BFCL-V4, and cache-dependent monetary savings. The abstract matches the body; there is no abstract-level overclaim.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Toollery?
Toollery is a training-free framework that compresses large LLM agent skill/tool libraries into a small retrieved candidate set before the LLM makes its final choice. It works by indexing LLM-generated user-intent queries (three per capability) alongside the original tool specifications, then retrieving only a top-10 candidate set for each user request.
How much prompt cost does Toollery save?
Reported reductions are large in context size: 57K-84K tokens drop to 1.3K-2.1K on BFCL-V4, and a ~79K-skill library drops from roughly 43M tokens per request to about 27.7K. However, actual billed-cost savings depend on provider caching: the paper found no significant billed-cost advantage when cache hit rates were naturally high, but 97.5% savings with caching suppressed.
Does Toollery improve accuracy?
Retrieval recall improves over raw specification retrieval in all three evaluated settings. End-to-end selection accuracy improves significantly only on a proprietary smart-cockpit dataset (0.163 to 0.268 exact-set accuracy); on the public BFCL-V4 benchmark, gains of 0.003-0.005 AST accuracy were reported as not statistically significant.
Is Toollery code or data available?
A GitHub repository is referenced in the paper, and the authors promise to release code, prompts, run configurations, and normalized manual artifacts upon acceptance. The 3,396-request smart-cockpit dataset that carries the main accuracy gain is proprietary and will not be released.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All tables arrive flattened into running text by PDF extraction; values were cross-checked against captions and inline prose, but multi-column layouts (e.g., Tables 9 and 15-17) were reconstructed from the flattened stream.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.