Measuring the Serving Stack Instead of the Model: Hidden Confounds in Local Tool-Use Evaluation
cs.CL · cs.AI · cs.SE
Quick verdict
A measurement study showing Ollama's per-model tools gate and lossy harness logging can report capable local models at 0% fidelity; honest but thin-n evidence.
Summary
This measurement study argues that local serving stacks, not just models, determine reported tool-call success. Using a ReAct coding-agent harness on a fixed aggregation task (8 seeds per model; nine small local models served by Ollama 0.30.8 plus a cloud anchor), the authors show Ollama gates the tools= request per model via a static template flag: Qwen2.5-Coder is accepted (calls returned as text), Llama-3.2 returns native tool_calls, while Phi-3 and Gemma-3 are rejected with HTTP 400 before inference — and because the harness fails to persist rejection or retry exhaustion as structured metadata, naive per-turn analysis reports these never-run models at 0% fidelity. A control arm shows adding a text tool list while keeping the native channel recovers most fidelity for accepted models (e.g., Qwen-3B per-seed: 23%→82%), while a uniform text protocol drops Llama-3.2 from 82% to 44%. Cross-stack probes on llama.cpp, vLLM, and SGLang confirm the same weights and requests are handled differently per stack. Constrained decoding guarantees parseable calls but induces 600–900-turn non-termination on weak models, and turn-pooled versus per-seed rates differ by up to ~55 points. Limitations include thin denominators (8–16 produced turns), wide bootstrap CIs (e.g., 34% [9,59]), a single instrumented harness, version-pinned Ollama behavior, and vLLM/SGLang probes covering only two models.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Ollama 0.30.8 gates the OpenAI-compatible tools= request per model via a static template flag, rejecting Phi-3 and Gemma-3 with HTTP 400 before inference while accepting Qwen2.5-Coder (calls as text) and Llama-3.2 (native tool_calls).
- Because the instrumented harness does not persist rejection or retry exhaustion as structured failure metadata, a naive per-turn analysis misclassifies these never-run models as non-callers and reports 0% protocol fidelity.
- For models the server accepts, adding a fixed text tool list to the native channel recovers most fidelity (Qwen 0.5B–14B per-seed: 0/38/23/59/60% → 35/58/82/89/80%), whereas a uniform text protocol drops Llama-3.2 from 82% to 44%.
- The same weights and identical tools= request are handled differently across Ollama, llama.cpp, vLLM, and SGLang, so request acceptance and call recognition are stack-and-configuration properties, not model properties.
- Turn-pooled versus per-seed fidelity estimates diverge by up to ~55 points on thin denominators, and constrained decoding removes parse failures only by inducing 600–900-turn non-termination loops in weak models.
Strengths
Clean isolation of channel vs prompt
The three-condition design (native / native+hint / text-tools, §3) with a fixed model-agnostic hint separates the serving channel from prompt guidance, cleanly attributing the accepted-model gap: native→native+hint per-seed gains of 0→35, 23→82, 60→80 across Qwen sizes (§4, Fig. 3), while Llama-3.2's 82→44 drop under text-tools shows the channel can also be the bottleneck.
Cross-stack falsification of the '0%' reading
Table 2 shows identical GGUF weights Ollama rejects (HTTP 400 for Phi-3, Gemma-3-270m) return 200 on llama.cpp --jinja, vLLM refuses by default until --enable-auto-tool-choice/--tool-call-parser, and SGLang accepts but returns text — establishing request handling as stack-and-configuration policy, not a model property.
Unusual claim hygiene and declared negative result
Replication discipline matching the mechanism
Strong reproducibility detail
Weaknesses
Very thin sample sizes cap all quantitative claims
The headline 'hidden confound' is partly self-inflicted
Cross-stack probes are shallow and not quantization-controlled
Constrained-decoding evidence is single-step only
Version-fragile external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement-validity study, not a benchmark: it asks whether reported tool-call fidelity of small local coding agents reflects the model or the serving stack. Three qualitative claims are made and, in my reading, supported: (1) Ollama 0.30.8 gates the OpenAI-style `tools=` request per model via a static template flag — Qwen2.5-Coder accepted (calls returned as text), Llama-3.2 returning native `tool_calls`, Phi-3 and Gemma-3 rejected with HTTP 400 before inference (§3, Table 2); (2) the instrumented harness collapses rejection and retry exhaustion into ordinary assistant non-call turns, so naive per-turn analysis reports never-run models at 0% fidelity (§5); (3) the same request handled differently across four stacks shows request handling is stack policy, not model ability. A fourth, quantitative claim — which serving configuration maximizes fidelity — is explicitly left model-dependent, and the paper declares a negative result (no claims about scale, family, or reasoning). This framing discipline is the paper's most distinctive virtue.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why can a model be reported at 0% tool-call fidelity without ever running?
Ollama (version 0.30.8 in this paper) checks a static per-model template flag before dispatching a tools= request; unsupported models such as Phi-3 and Gemma-3 receive an HTTP 400 'does not support tools' rejection before inference. If the agent harness logs that rejection as an ordinary assistant 'no-call' turn instead of structured error metadata, downstream per-turn analysis counts it as a model failure and reports 0% fidelity — even though the model never generated anything.
Is Ollama's tool-call rejection a property of the model or of the server?
Of the serving stack, per the paper's cross-stack probe. The same GGUF weights that Ollama rejects run under llama.cpp with --jinja and answer the identical tools= request; vLLM refuses the request by default until launched with --enable-auto-tool-choice and a --tool-call-parser (independent of the model), and SGLang accepts it but returns the call as plain text unless a parser is configured. Request handling is therefore stack-and-configuration policy, not model ability.
Should I use native function calling or prompt-based tools with small local models?
The paper finds no single best configuration. For most Qwen2.5-Coder sizes, keeping the native channel and adding a text tool list was best or tied (e.g., Qwen-3B per-seed 23% native vs 82% native+hint); for Llama-3.2, which has genuine native tool-call support, forcing a text-only protocol dropped fidelity from 82% to 44%. Practical advice from the checklist: pin and report your serving stack and version, log serving-level failures as their own outcome category, and report per-seed rates with intervals rather than a single pooled number.
Does grammar-constrained decoding fix tool calling for local models?
It removes parse failures by construction — in the paper, all nine local models emitted a valid in-schema call on 8 of 8 single-step trials. But it does not remove failure overall: forced to emit a tool call every turn, weak models loop without terminating (600–900-turn episodes; one Qwen-0.5B run cut off at 869 turns). The paper positions constrained decoding as a baseline that moves the failure from parsing to termination, not a fix.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 appear in the extraction only as captions/ASCII renditions; plotted distributions and error bars could not be independently verified beyond CI values quoted in the text.
- The released repository (probe scripts, harness patch, classifier) was not inspected; reproducibility statements rest on the paper's own description.
- The paper is a preprint (arXiv 2609.26693v1); references include several 2026 preprints whose contents were not verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.