Extending FunctionGemma for Practical On-Device Mobile Function Calling
Ali Rezagholizadeh, Soheila Samiee
cs.LG · cs.SE
Quick verdict
270M on-device function caller lifted 29.3%→76.5% E2E via 9.5K synthetic examples; credible open work, but eval is synthetic-only with no significance testing.
Summary
The paper extends FunctionGemma 270M-it, a compact instruction-tuned model, from Google's seven-category Mobile Actions catalog to fifteen practical Android device-control categories (calls, SMS, brightness, camera, screenshots, app management). The authors generate MOBILEACTIONSEXTENDED, ~9,500 synthetic, schema-validated conversations (a teacher LLM prompted with tool schemas; ~300 single-tool and ~5,000 multi-tool prompts), then run full TRL supervised fine-tuning with completion-only loss, releasing an extended specialist and a combined model trained jointly on the merged Google corpus. On the 965-example held-out extended split, end-to-end exact-match accuracy rises from 29.3% (base) and 17.2% (Google's specialist) to 76.5% for both new models; on the 961-example Google split, the combined model reaches 82.3% versus the specialist's 90.3% — an 8.0-point cost for doubling coverage. Under an execution-aware metric that defaults two optional booleans, the extended specialist scores 100%. The main limitations: evaluation is confined to synthetic, self-generated held-out data with no real-user or ASR-noise testing; all comparisons use a single 270M backbone; no variance estimates or significance tests are reported; and on-device latency (~13 s per turn on a Pixel 8 Pro CPU) sits awkwardly beside the abstract's low-latency claim. Artifacts (datasets, models, pipeline, Android demo) are publicly released, supporting reproducibility.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fine-tuning FunctionGemma 270M-it on ~9,500 schema-validated synthetic conversations raises end-to-end exact-match accuracy on the extended 15-category set from 29.3% (base) to 76.5%, with 100% function-name accuracy.
- The combined model trained on merged data retains 76.5% on the extended set and reaches 82.3% on Google's Mobile Actions set — an 8.0-point cost versus the 90.3% specialist in exchange for doubled category coverage.
- Google's narrow Mobile-Actions specialist actively underperforms the base model out of distribution (17.2% vs 29.3% on the extended set), evidencing prior over-concentration from narrow fine-tuning.
- Under an execution-aware criterion with safe defaults for two optional booleans, the extended specialist achieves 100% on the extended eval set, implying all residual errors are recoverable formatting issues.
- Compact on-device function calling is practically deployable, demonstrated by a 285.6 MB quantized bundle completing a function-call turn in ~13 s on a Pixel 8 Pro CPU.
Strengths
Complete open artifact chain
Both datasets (UGrowAI/Mobile-Actions-Extended, -Combined), both fine-tuned checkpoints, the three-package reproduction pipeline (Appendix D.4) and a Buildozer Android demo APK are released with enumerated Hugging Face IDs — unusual completeness for this paper scale.
Dual-metric honesty about errors
Section 6.3/Table 3 separates strict exact match from an execution-aware re-score, quantifying that all residual MA-Extended errors are optional-boolean omissions (phone_call, send_email at 0% strict in Table 5) rather than wrong tool selection; per-category Tables 5-8 expose failures instead of hiding them.
Clean three-question design with a decisive merge ablation
Quantization/deployment separation
Fully specified training recipe
Weaknesses
Synthetic-only, self-generated evaluation loop
No statistical rigor
Execution-aware metric is author-defined post hoc
Latency undercuts the headline claim
Single backbone, unnamed teacher
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow but practical niche: extending Google's FunctionGemma 270M-it from its seven-category Mobile Actions catalog to fifteen Android device-control categories (calls, SMS, brightness, camera, screenshots, app management; Table 1). Three research questions structure the work — Q1 coverage gap, Q2 targeted fine-tuning, Q3 cross-domain transfer. Headline claims: (i) end-to-end exact-match accuracy on the new extended set rises from 29.3% (base) and 17.2% (Google specialist) to 76.5% (Table 2); (ii) the combined model costs 8.0 points on Google's set (82.31% vs 90.32%) for doubled coverage; (iii) under an execution-aware metric the extended specialist reaches 100.0% (Table 3); (iv) on-device deployment is demonstrated in the Handy Chat app at ~13 s/turn. The claims are mostly matched by the presented evidence, with reservations on metric provenance and statistical support.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy does the extended FunctionGemma mobile model achieve?
On the paper's 965-example held-out extended set, the fine-tuned 270M model scores 76.5% strict end-to-end exact-match accuracy (up from 29.3% for the base model), with 100% function-name accuracy across all fifteen categories. Under the authors' execution-aware metric, which fills two optional boolean arguments with safe defaults, it scores 100%.
What is the trade-off of the combined model?
The combined model, trained jointly on the new 15-category dataset and Google's 7-category Mobile Actions corpus, keeps 76.5% accuracy on the extended set and reaches 82.3% on Google's set — 8.0 percentage points below the 90.3% of Google's own specialist, in exchange for doubled category coverage.
Does it run on a real phone, and how fast is it?
Yes. The authors demo an Android app called 'Handy Chat' on a Pixel 8 Pro using a 285.6 MB INT8-quantized LiteRT-LM bundle on the CPU. A function-call turn takes roughly 13 seconds end to end (about 9 s first decode, 2 s tool execution, 2 s second pass). All accuracy figures in the paper were measured on unquantized checkpoints.
Is the Mobile-Actions-Extended dataset real user data?
No. It is fully synthetic: a teacher LLM generated about 9,500 conversations from tool schemas, which were then structurally validated (tool names, required arguments, types). Phone numbers and emails are placeholders (+14165550xxx, example.com), so no real user data is included.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The teacher LLM behind dataset generation is not named anywhere in the extracted text, so the data-generation step's reproducibility is partially unverifiable from the paper alone.
- Figure 1 (pipeline diagram) is only available as caption text; visual details of the workflow were not assessed.
- Repository links render as 'available here' placeholders in the extraction; actual URLs were not verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.