Beyond Fluent Generation: A CPU Reliability Benchmark for MCP-Style Tool Calling in Sub-2B Small Language Models for Edge Deployment
Abrar Shahriar Qurat-Ul-Ain Mastoi
cs.CL
Quick verdict
100-prompt CPU benchmark: sub-2B SLM tool-call reliability spans 0-79%; candid audit, but copying-only task and no edge hardware limit the deployment claims.
Summary
This benchmark study tests five open-weight sub-2B language models (Phi-1.5, Pythia-1.4B, TinyLlama-1.1B-Chat, Qwen2.5-0.5B, Qwen2.5-1.5B) on 100 templated prompts requiring MCP-style JSON tool calls across five mock tools, under greedy decoding and nucleus sampling, in FP32 on CPU. A recovery parser strips Markdown fences and extracts brace-delimited JSON; scoring covers parseability, tool-name correctness, argument completeness, and value agreement. Qwen2.5-1.5B reaches 75% (greedy) and 79% (sampling); Qwen2.5-0.5B reaches 72% greedy but collapses to 32% sampled; all other models score at most 7%. A strict audit finds only 5 of 1,000 raw responses directly parseable as JSON, showing near-total dependence on output recovery. A three-prompt CPU probe reports process memory and latency (e.g., Qwen2.5-1.5B: 7,960 MiB, 30.782 s). The paper candidly acknowledges major limitations: prompts reveal the tool and all arguments (a copying test rather than intent recognition), chat templates were not applied, base and chat checkpoints are mixed, the probe used unrecorded hardware with only three prompts in a shared process, and no edge board was actually tested. The notebook and 1,000 responses are released on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Sub-2B checkpoints vary dramatically in MCP-style JSON tool-call generation, from 0% (Phi-1.5 both conditions) to 79% (Qwen2.5-1.5B sampling) under a recovered-call criterion on 100 fully specified prompts.
- Only 5 of 1,000 raw generations are strict whole-response JSON, so any operational system must include an explicit recovery/repair layer that should be measured and hardened separately.
- Decoding effects are model-dependent: nucleus sampling collapses Qwen2.5-0.5B from 72% to 32% (exact McNemar p=1.62e-7) while modestly helping Qwen2.5-1.5B (75% to 79%, n.s.).
- CPU resource cost is not monotonic in parameter count (Pythia RSS anomaly), and SLMs should be treated as fallible proposal generators behind a deterministic supervisory layer enforcing schemas, permissions and confirmation.
Strengths
Strict-output audit is the paper's most transferable finding
Section 4.2 re-parses all 1,000 raw responses without recovery: only 5 are strict whole-response JSON, all from Qwen2.5-1.5B sampling (5%). This honestly reframes the headline 79% as a system-level (model + recovery parser) result rather than raw model compliance.
Stage-wise failure attribution localizes the bottleneck
Table 3 decomposes each condition cumulatively (recovered JSON → tool → args → values → extras); Section 4.4 shows 588 of 658 failures (89.4%) terminate at JSON recovery, versus 40 missing parameters, 27 wrong tool, 3 value mismatches — a clean diagnosis of where sub-2B models actually break.
Above-average statistical hygiene for a benchmark note
Candid, granular limitations section
Artifact release supporting reproduction
Weaknesses
Benchmark measures copying, not tool selection
Confounded checkpoint comparison: no chat templates, mixed base/chat models
Resource probe is anecdotal evidence, yet abstracted as headline numbers
Edge-deployment framing outruns the evidence
Single sampling run and missing strong baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This benchmark paper (arXiv 2609.07370, cs.CL) asks how reliably five open-weight checkpoints below 2B parameters — Phi-1.5 (1,418M), Pythia-1.4b-deduped (1,415M), TinyLlama-1.1B-Chat (1,100M), Qwen2.5-0.5B (494M), Qwen2.5-1.5B (1,544M; Table 1) — emit MCP-style JSON tool calls, and what CPU resources that costs. Central claims: (1) recovered-call success spans 0%–79% across ten model×decoding conditions; (2) only 5 of 1,000 raw responses are strict whole-response JSON, so a recovery layer is part of any working system; (3) decoding effects are model-dependent, most sharply Qwen2.5-0.5B's 72%→32% sampling collapse; (4) a reliability-resource trade-off exists. The authors are explicit that this is a controlled, MCP-inspired function-call baseline — no tool discovery, no execution, no trajectories (§1) — and that none of the five named edge boards was tested.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did this paper actually test Raspberry Pi, Jetson Nano, or other edge boards?
No. The paper states explicitly (Section 1 and Section 6) that Raspberry Pi, NVIDIA Jetson Nano, Arduino UNO Q, Orange Pi, and LattePanda were not benchmarked. All experiments ran in FP32 on an unspecified CPU; the boards are presented only as prospective motivation, and no quantised runtime was tested.
Which model performed best and what was its score?
Qwen2.5-1.5B with nucleus sampling achieved 79% recovered tool-call success (95% CI 70.0-85.8%) on 100 prompts; its greedy condition scored 75%. However, only 5% of its sampled raw responses were strict whole-response JSON — the rest required a recovery parser that strips Markdown and extracts brace-delimited text.
Why do most tool-calling attempts fail?
Of 658 failed evaluations, 89.4% (588) terminated at JSON recovery — the model output could not be extracted and parsed at all. Smaller shares failed at missing parameters (6.1%), wrong tool name (4.1%), or value mismatch (0.5%). Base models like Phi-1.5 and Pythia essentially never produced recoverable JSON.
Is the benchmark data and code publicly available?
Yes. The paper points to github.com/Abrar051/sml_mcp_benchmark, said to contain the notebook, the 100 benchmark prompts (benchmark_prompts.csv), all 1,000 detailed model responses (detailed_results.csv), summary metrics, error counts, and figures, under a fixed random seed of 42.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 appear only as captions in the extracted text; all quantitative assessments rely on Tables 2-6, whose values are fully present.
- The GitHub repository was not inspected (no network access in this review); artifact completeness is taken from the paper's own Reproducibility section.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.