MCP-GRANITE Benchmark: GRANularity Interface TEsting for MCP-Based LLM Agents
Demetris Paschalides, Moysis Symeonides, Moysis Symeonides, George Pallis, Marios D. Dikaiakos
cs.DC · cs.AI · cs.LG
Quick verdict
MCP-GRANITE shows a 4-tool interface best aids edge LLM agents, but evidence rests on fixed-response mocks and confounded quantization.
Summary
MCP-GRANITE treats MCP tool-interface granularity as a controllable variable for edge-hosted LLM agents. The framework generates 81 multi-step scenarios across 9 edge/IoT domains, each instantiated at 4 granularity levels, and evaluates 9 locally deployed models (268M-20.9B parameters) over 8,748 trials on a single Tesla T4. Moderate consolidation to 4 task-class tools (L3) performs best: task completion of 0.49 improves 16.4% over fine-grained primitives (L4) and 33.6% over a single monolithic tool (L1), while argument accuracy nearly doubles (0.20 to 0.40). Model size correlates only weakly with task completion (ρ=0.285) and strongly with latency (ρ=0.946); its association with argument accuracy (ρ=0.686) loses significance after excluding two outliers. Notably, a 3.2B model at optimal granularity outperforms a 20.9B model at mismatched granularity. Main limitations: evaluation runs on mock servers returning fixed responses rather than live MCP deployments, quantization differs across models confounding scale comparisons, and correlations rest on only 9 models.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Tool-interface decomposition is an uncontrolled but consequential design variable that existing MCP/function-calling benchmarks ignore (Sec. I-II).
- Moderate consolidation to 4 task-class tools (L3) is optimal: TC 0.49, +16.4% over L4 and +33.6% over L1, with argument accuracy nearly doubling from 0.20 to 0.40 (Table II).
- Parameter count is weakly associated with task completion (rho=0.285, n.s.) and strongly with latency (rho=0.946), while the argument-accuracy association (rho=0.686) is not robust to outlier exclusion.
- Interface design can outweigh model scale: a 3.2B model at optimal granularity beats a 20.9B model at mismatched granularity.
- Over-consolidation causes qualitative breakdown, with 28.2% of L1 runs completing without invoking any tool (Sec. V-B).
Strengths
Genuine gap with a controlled design
Sec. I-II correctly identify that BFCL, ToolBench, MCP-Universe, and MCPGAUGE all hold the tool interface fixed; MCP-GRANITE varies only decomposition (4 levels over identical domain logic, Sec. IV) across 9x9x4x9x3 = 8,748 runs with 3 repetitions and temperature 0, isolating the factor.
Multi-metric evaluation with non-trivial statistics
Sec. IV reports paired bootstrap tests with Benjamini-Hochberg correction across 9 comparisons, 2,000 model-level bootstrap CIs (Table II), a logistic regression yielding L3 odds ratio 1.36x (CI [1.12, 1.66], p=0.002), and Spearman correlations with an explicit sensitivity check (rho 0.686 to 0.360 after dropping Qwen3 and GPT-OSS).
Resource-aware edge analysis rarely seen in tool-use benchmarks
Actionable failure phenomenology
Extensible, open infrastructure
Weaknesses
Quantization confound in all model comparisons
n=9 correlation analysis is underpowered and partially self-refuted
Mock servers with fixed responses limit external validity
Cross-granularity comparison changes task length
Imprecise headline claims and a naming collision
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MCP-GRANITE (MASCOTS 2026 submission, arXiv:2609.24161) asks a question most tool-use benchmarks skip: given the *same* capability set, how should an MCP server decompose it into tools? The framework instantiates 81 multi-step scenarios across 9 edge/IoT domains at 4 granularity levels — L4 (8-10 primitive tools), L3 (4 task-class tools), L2 (read/write split), L1 (one monolithic tool) — and runs a full factorial of 9 models x 9 domains x 4 levels x 9 scenarios x 3 repetitions = 8,748 trials on a Tesla T4. Central claims: (1) L3 is the best operating point (TC 0.49, +16.4% vs L4, +33.6% vs L1; ArgAcc 0.20 to 0.40); (2) model size weakly predicts TC (rho=0.285, p=0.458) but strongly predicts latency (rho=0.946); (3) a 3.2B model at optimal granularity beats a 20.9B model at mismatched granularity. The related-work positioning (Sec. II) is accurate: BFCL, ToolBench, ToolSandbox, MCP-Universe, and MCPGAUGE all hold the interface fixed.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is tool-interface granularity in MCP-GRANITE?
It is the number and abstraction level of tools an MCP server exposes for the same underlying capability set. The benchmark tests four levels: L4 (8-10 fine-grained single-purpose tools), L3 (4 task-class tools), L2 (read/write query-control split), and L1 (one monolithic tool), while keeping tasks, data, and domain logic identical.
Which granularity level is best for LLM agents using MCP tools?
According to the paper, a moderate 4-tool interface (L3) is best overall: task completion of 0.49, 16.4% higher than fine-grained primitives and 33.6% higher than a single monolithic tool, with argument accuracy nearly doubling (0.20 to 0.40). Over-consolidation to one tool makes 28.2% of runs finish without any tool call.
Do bigger LLMs do better at tool calling in this benchmark?
Not reliably. Across the 9 models (268M-20.9B), parameter count correlates only weakly with task completion (rho=0.285, not significant) and strongly with latency (rho=0.946). The association with argument accuracy (rho=0.686) disappears after excluding two outlier models (rho=0.360, p=0.427).
Is the MCP-GRANITE benchmark's data real or simulated?
Simulated: agents interact with containerized mock MCP servers exposing deterministic in-memory domain state and fixed responses, on a single NVIDIA Tesla T4. This enables controlled, repeatable comparison, but the authors note results characterize agent-side execution, not live IoT deployments.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 4-9 are unavailable as data in the extracted text; judgments about per-model and per-domain distributions rely on in-text numbers and table fragments.
- Table I values are embedded as raw text without column alignment, so a few per-model attributions (e.g., GPT-OSS error rate 55.6%) were cross-checked against the narrative but could shift slightly.
- The repository [15] was not inspected; 'open-source' and gold-standard verification are taken from the paper's claims.
- Statistical details of the logistic regression (specification, clustering) are compressed into one sentence in the paper and could not be fully audited.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.