From Fixed Keys to Readable Schemas: Small Language Models for Vehicle Agent Function Calls
Hamed Jafarzadeh Asl, Yuanhao Yu, Vahid Partovi Nia
cs.LG · cs.AI · cs.CL
Quick verdict
Controlled FT-vs-SIP study on 9,822-example vehicle benchmark: schemas beat tokens on unseen functions and refusal; FT's refusal gap is partly by construction.
Summary
The paper compares two ways of exposing a vehicle's available functions to small language models: Functional Tokens (FT), which assign each function a dedicated vocabulary entry, and Schema-in-Prompt (SIP), which serializes function schemas into the input context. Using a new benchmark of 9,822 single-turn examples over 79 Android Automotive functions, split into Seen (5,521), Unseen (2,146), and Out-of-Scope (2,155) examples, the authors fine-tune Gemma 3-270M, FunctionGemma-270M, Qwen3-0.6B, and Qwen3-1.7B under matched recipes with three seeds. On Seen functions all configurations reach 95.5-98.7% exact accuracy, with the 0.6B model best and no significant FunctionGemma-vs-Qwen3-1.7B gap. On Unseen functions FT scores zero by construction, while SIP reaches 84.1% (Qwen3-1.7B), 59.3% (0.6B), and 15.6-18.6% at 270M. SIP refuses Out-of-Scope requests more reliably (95.7-98.4% vs 81.4-83.1%), and a decomposition shows the gap concentrates on requests whose unavailable function has a trained token (FT refusal 2.9-5.1% vs SIP 81.2-93.5%). SIP costs 26-98x time-to-first-call and 2.4-4.2x peak memory on a four-thread CPU proxy. Limitations: the benchmark is synthetic (requests written by Llama-3.3-70B), English-only, single-turn; the main grid uses LoRA only for the 1.7B model; latency is not measured on vehicle hardware; no code or data release is stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- On Seen functions, both representations perform nearly identically (95.5-98.7%) and additional scale gives limited benefit, with Qwen3-0.6B strongest in the main grid and no detected FunctionGemma-270M vs Qwen3-1.7B difference in any seed.
- Functional Tokens achieve zero exact accuracy on Unseen functions by construction of the output space, while Schema-in-Prompt generalizes to withheld functions, reaching 84.1% with Qwen3-1.7B and improving with scale.
- SIP's Out-of-Scope refusal advantage (95.7-98.4% vs FT's 81.4-83.1%) is concentrated in requests whose unavailable function has a trained token, where FT refuses only 2.9-5.1% of the time versus SIP's 81.2-93.5% (Table 5).
- SIP's flexibility costs 26-98x time-to-first-call, 2.4-4.2x peak resident memory, and roughly 1,800 additional prompt tokens on a four-thread CPU device proxy.
- Function-surface representation, rather than model scale alone, determines the capabilities and failure modes of SLM-based vehicle function calling.
Strengths
Genuinely controlled comparison
FT and SIP use the same corpus, optimization recipe (3 epochs, LR 2e-5, batch 16, 4096 ctx), identical evaluation examples, and three seeds per configuration; comparisons are backed by paired example-level exact McNemar tests per seed (Appendix C.4, Table 4) rather than aggregate accuracy alone.
Benchmark construction with anti-shortcut safeguards
Label-first generation validates the structured call programmatically before Llama-3.3-70B writes the request (A.3); leakage screening applied at both training and scoring; offered sets of 4-24 functions (median 14) deliberately mix Seen and Unseen schemas so neither split identity nor the target is identifiable by elimination (A.4); partitioning is by normalized request text and by underlying unsupported request type (A.6).
The refusal decomposition is the strongest evidence
Honest reporting culture
Confounding choices are ablated
Weaknesses
The FT refusal failure is partly baked into the evaluated input format
Main grid mixes adaptation methods and model families
Synthetic corpus with a small verified slice
No non-fine-tuned baseline
Cost claims rest on one CPU proxy configuration
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks how the *representation of the available function surface* — dedicated Functional Tokens (FT) versus schemas serialized into the prompt (Schema-in-Prompt, SIP) — affects small language models (270M–1.7B) doing single-turn vehicle function calling. Five claims are made: (1) on Seen functions, representation and scale barely matter (95.5–98.7% exact accuracy, Table 3; best at Qwen3-0.6B, and no detected FunctionGemma-270M vs Qwen3-1.7B difference in any seed, Table 4); (2) FT scores zero on Unseen functions by construction while SIP generalizes (84.1% at Qwen3-1.7B, 59.3% at 0.6B, 15.6–18.6% at 270M); (3) SIP refuses Out-of-Scope requests more reliably (95.7–98.4% vs 81.4–83.1%); (4) SIP costs 26–98× time-to-first-call, 2.4–4.2× peak memory, and ~1,800 extra prompt tokens on a CPU proxy (Table 7); (5) therefore function-surface representation, not scale, is the decisive deployment choice. Appendix B formalizes (2)–(3) via task-level output support: Y_FT = T ∪ {⊥} versus row-specific Y_SIP,i = O_i ∪ {⊥}. Claims (1) and (2) are solid; claim (3) is directionally credible but partly an input-format artifact; claim (4) is proxy-only.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the difference between Functional Tokens and Schema-in-Prompt for function calling?
Functional Tokens (FT) give each supported function its own vocabulary token learned during fine-tuning, so the model's possible outputs are fixed at training time and inference prompts stay short. Schema-in-Prompt (SIP) instead lists the schemas of the currently available functions in the prompt, so the model can select functions it never saw as training targets, at the cost of much longer prompts (about 1,800 extra tokens in this paper) and slower inference.
Why does the Functional Token model score zero on unseen vehicle functions?
Because an unseen function has no dedicated token in the model's learned output space, exact-match accuracy is zero by construction: the model can only refuse, emit a different known function, or produce invalid output. This is a property of the representation, shown empirically in Section 5 and formalized via task-level output support in Appendix B of the paper.
How much slower and heavier is Schema-in-Prompt than Functional Tokens?
On a four-thread Xeon 8580 CPU with Q4_K_M quantization (a proxy, not vehicle hardware), SIP needed 26-98 times more time to produce a first parseable call, 2.4-4.2 times more peak memory, and about 1,785-1,905 additional prompt tokens compared to FT across the matched model pairs (Table 7). Absolute SIP latency reached about 10.2 seconds for the 1.7B model.
Is the vehicle function-calling benchmark based on real driver data?
No. The 9,822 requests are synthetically generated: structured calls are sampled and validated against Android Automotive schemas first, then Llama-3.3-70B writes natural-language requests, which are screened for label leakage, semantically screened by a different model family (Mistral-Small-3.2-24B), and manually spot-checked on a 217-example slice (100% faithful, 99.1% unambiguous). The authors state that natural driver requests, other languages, and multi-turn interaction remain future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text ends naturally after Appendix C.7/Table 7 with no visible truncation; the absence of a code or data release statement is verifiable from the full text, not a truncation artifact.
- Figures 2-4 could only be assessed via their captions, axis ranges, and the corresponding numeric tables (Tables 3, 5, 7); per-figure point estimates beyond those tables were not independently checkable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.