Selection Is Retrieval, Abstention Is Not: On-Device Tool Routing over 70 Korean-English Actions
cs.CL
Quick verdict
Finds BM25 suffices for on-device tool selection but abstention needs a frozen encoder; a careful, candidly limited synthetic study whose cost rejection of neural ranking is post-hoc.
Summary
The paper decomposes on-device tool routing into two decisions - selecting which local action to invoke, and abstaining when the catalog holds no valid action - and measures what a non-neural retriever loses on each. Over 600 author-written Korean/English requests and a 70-action catalog (64 skills, six file tools, plus chat/ask/delegate labels), character 3-gram BM25 with an alias-expanded index reaches 0.988 top-1 on the 164 lexically matched requests (17 KB index, 0.140 ms CPU) versus 85 of 166 on paraphrases, rising to a mean 0.825 over five trials when the candidate set is restricted to seven. No classifier over BM25 score features exceeds 0.697 AUC for in- versus out-of-catalog, where frozen multilingual-e5-base reaches 0.806; the separate gate cuts delegate-to-skill errors from 134 of 150 to 12 of 150. The retained pipeline (BM25 rank + e5 gate, 13.9 ms, 278 MB) beats single-pool baselines; BGE ranking (0.660 accuracy) and Qwen3-4B native calling improve quality but are rejected on 14x and ~80x latency grounds. Key limitations: fully synthetic single-author data, the cost budget rejecting neural ranking was set after measurement, and neither code nor data is released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Selection under matched vocabulary is effectively solved by a 17 KB character 3-gram BM25 index (0.988 top-1 on 164 matched requests, 0.140 ms CPU median).
- No signal derived from BM25 scores separates in-catalog from out-of-catalog requests above 0.697 AUC, while frozen multilingual-e5-base reaches 0.806, so abstention - not selection - is the justification for a neural component on device.
- Separating an in-catalog gate from nearest-item ranking reduces delegate-to-skill errors from 134/150 to 12/150 and raises end-to-end accuracy from 0.508 to 0.592 over single-pool baselines.
- A neural ranker (P3, 0.660 accuracy) and native tool calling (Qwen3-4B) beat the retained pipeline on quality but are rejected on measured latency (14x and ~80x) and memory grounds, a trade the authors concede was framed after measurement.
Strengths
Clean decision decomposition with direct causal evidence
Table 1 isolates the gate's effect: delegate-to-skill errors fall from 134/150 (standalone BM25) and 54/150 (single-pool ladder) to 12/150, mutating false positives from 269/600 to 43/600, and accuracy rises 0.508 to 0.592 - the abstract's asymmetry claim is actually substantiated, not just asserted.
Unusually thorough confirmatory controls
Section 4's alias claims are backed by a shuffled-alias control (0.127), two length-padding controls bracketing E1 (0.349 and 0.361 vs E1's 0.512), BM25 b=0 invariance, and an example-expanded E2 rejected when its length-matched filler control scored equal or better (0.518 vs 0.512).
Statistical care beyond the norm for engineering papers
Calibration analysis that preempts an obvious objection
Radical transparency in Limitations
Weaknesses
Fully synthetic, single-author evaluation data
Central architectural conclusion rests on a post-hoc cost budget
No uncertainty on the tiny margins that decide pipeline selection
No code or data release; self-referential evidence chain
SLM comparison incomplete by the authors' own admission
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper in plain terms?
For on-device assistants choosing among tools, a tiny keyword-based search (BM25, 17 KB, 0.14 ms) picks the right tool almost perfectly when the user's wording resembles the tool catalog, but it cannot tell when NO tool fits the request. Deciding whether to abstain - and instead chat, ask a follow-up, or delegate to a server - requires a small neural embedding model. Hence the title: selection is retrieval, abstention is not.
Why not just run a language model on the device to route tools?
The paper measures the cost: a 4-billion-parameter model needed about 1,111 ms per routing decision on a GPU versus 13.9 ms for the retained BM25-plus-encoder pipeline, and roughly 80x the latency of the retained pipeline in the full-catalog condition. On a phone-like device with tight latency and memory budgets, the authors argue the language model buys quality the deployment cannot pay for, though they concede no cost limit was fixed in advance.
Is BM25 better than embedding models for tool selection?
It depends on vocabulary match and how many candidates are active. BM25 wins when requests reuse catalog wording (0.988 vs 0.951 top-1); embeddings win on paraphrases (0.608 vs 0.512 at 70 candidates). With seven candidates the two are nearly tied (0.825 vs 0.835), and the vocabulary-match share at which BM25 overtakes embeddings shifts with candidate count (0.664 to 0.798 in the paper's Table 8), so there is no single winner.
What are the biggest limitations of this study?
All 600 test utterances were written by the single author (no production or third-party data), the latency budget used to reject the better neural ranker was set after seeing results, the test catalogs and data are not released, and two of four small-language-model checkpoints could not be run at all for runtime reasons. The paper is unusually explicit about all of this in its Limitations section.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are only available as captions in the extracted text; visual claims (e.g., coverage-vs-danger curves in Figure 2, Jaccard distributions in Figure 4) could not be independently checked.
- The external comparisons (Mobile Actions, Needle 2 C engine) are vendor/repository reports the authors themselves flag as not rerun; this review inherits that uncertainty.
- Table formatting is mangled in extraction, but all quoted numbers were cross-checkable against surrounding prose; no figure-dependent conclusion is asserted here.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.