What a Random Draw from the MCP Registry Contains, and What Tool-Use Benchmarks Contain Instead
cs.SE · cs.AI
Quick verdict
Claims only 48.8% of a random MCP draw works vs 66.7% curated, and BFCL v4 is 16.7% self-duplicated; transparent, careful measurement with minor power caveats.
Summary
This empirical measurement study asks what the Model Context Protocol (MCP) server ecosystem actually contains when sampled without the curation or repair steps other studies apply. From a 24,135-server registry census, the author draws 400 npm/stdio servers with a published seed and probes each exactly once — no credentials, no retry, no repair. Only 48.8% complete an initialize handshake, versus 66.7% on a hand-curated 24-server frame; the dominant failure is servers that never start (37.5%), not missing credentials (13.3%). Among the 195 running servers' 2,766 tools there are zero fatal JSON Schema violations, but 58.8% of tools lack optional safety annotations (41.5% on the curated frame). Holding one method constant (TF-IDF cosine after global deduplication), real MCP tools show 2.8% near-duplication at cosine 0.70, all within single servers (cross-author 0.0% at every threshold tested); BFCL v4 shows 16.7%, of which 16.4 points lie between independently presented tasks, and UltraTool shows 0.3%. Raw releases contain 68.8% (BFCL) and 85.6% (UltraTool) exact name-plus-description duplicates, against 0.4% for real MCP. Limitations include the npm/stdio-only scope (30.7% of a shifting population), single-probe inclusion being a lower bound, purely lexical similarity, a small 24-server comparison frame, an admitted power mismatch in the cross-unit decomposition, and no confidence intervals on the headline gaps.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- An unrepaired probability sample of the MCP registry yields only 48.8% inclusion (vs 66.7% curated), with servers that never start (37.5%) dominating credential-gated exclusions (13.3%).
- Among running servers, hard conformance is total — zero fatal JSON Schema violations across 2,766 tools — while 58.8% of tools omit optional safety annotations (vs 41.5% on a curated frame).
- Real deployed MCP tools show 2.8% lexical near-duplication at cosine 0.70, entirely within single servers; cross-author near-duplication is 0.0% at every threshold tested.
- BFCL v4 is internally redundant (16.7% near-duplicates at 0.70, 16.4 points across tasks; 68.8% exact raw duplicates) while UltraTool is cleaner than real tools (0.3%), so repetition is a BFCL property, not one of synthetic corpora as a class.
- Any statistic computed over the raw BFCL or UltraTool releases without global deduplication measures repetition rather than tools.
Strengths
Selection-free design with a full audit trail
Seeded mulberry32 partial Fisher–Yates draw (seed 20260819), SHA-256-pinned frame, and per-server outcomes for all 400 draws released — Table 1 shows no silent drops, so every aggregate can be independently recounted.
Curation bias is measured, not asserted
The same instrument on a 24-server hand-curated frame gives 66.7% inclusion vs 48.8% random and 41.5% vs 58.8% annotation omission (Figure 2), quantifying selection effects that run in opposite directions.
Within- vs cross-unit decomposition is the sharpest analytic move
Unusually candid threats section
Weaknesses
Single-probe design makes the headline a lower bound
Curated comparator is tiny and uncertainty-free
Cross-unit decomposition is admitted to be power-mismatched
Scope covers a shrinking minority of the ecosystem
Lexical similarity with no manual validation of pairs
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is an empirical measurement study of the Model Context Protocol (MCP) server ecosystem and of two tool-use benchmark corpora, framed as a methodological corrective: prior behavioral MCP studies (MCPZoo's 64,611→37,288 repair pipeline; curated reference sets) quietly select for servers that work. The paper measures what an unrepaired probability sample contains, then turns the collected tool descriptions into a reference distribution for judging benchmark redundancy. Central claims: (1) only 48.8% of 400 randomly drawn npm/stdio servers complete an initialize handshake vs 66.7% on a hand-curated frame, and non-starters (37.5%) outnumber credential-gated servers (13.3%) nearly 3:1; (2) among 195 running servers, zero of 2,766 tools show fatal JSON Schema violations, while 58.8% of tools omit optional safety annotations (vs 41.5% curated); (3) real MCP tools show 2.8% lexical near-duplication at cosine 0.70 — entirely within single servers — while BFCL v4 shows 16.7% (16.4 points between tasks) and UltraTool 0.3%; (4) raw benchmark releases are 68.8% (BFCL) and 85.6% (UltraTool) exact name+description duplicates vs 0.4% for real MCP, so un-deduplicated statistics measure repetition, not tools.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What percentage of MCP servers in the registry actually work?
In this study, only 48.8% of a seeded random draw of 400 npm/stdio servers from the official MCP registry completed an initialize handshake on a single probe with no credentials, retries, or repair; 37.5% never started at all and 13.3% required credentials. The author notes 48.8% is a lower bound because transient failures count as exclusions, and the npm/stdio slice covered is about 30.7% of the registry.
Are BFCL v4 benchmark tool definitions duplicated?
Yes. 68.8% of raw BFCL v4 tool records are exact name-plus-description repeats, and after global deduplication 16.7% of tools still have a near-duplicate at cosine similarity 0.70 — with 16.4 of those 16.7 points between independently presented tasks. By contrast, UltraTool's raw releases are 85.6% exact duplicates but only 0.3% near-duplicates after dedup, and real MCP tools show 2.8% (all within single servers), so heavy repetition is specific to BFCL in this comparison, not synthetic corpora as a class.
Are MCP tool JSON schemas usually malformed?
Not in this sample: across 2,766 tools advertised by 195 randomly drawn servers that started, the study found zero fatal JSON Schema violations and no server with any. This replicates at roughly fourteen times the scale an earlier result on a 24-server curated frame, contradicting the intuition that MCP tool schemas are frequently broken.
Is the paper's data and code available?
Yes. The mcp-probe instrument, harvester, seeded draw script, resumable probe runner, and redundancy measurement are open source on GitHub (itguruhaseeb/mcp-probe); the tool release and dataset are archived on Zenodo (DOI 10.5281/zenodo.21347997); and the per-server outcome of all 400 draws is released so the aggregate figures can be independently recounted using the published seed 20260819.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.