No-Box Vulnerability Analysis: Description-only Detection of Indirect Prompt Injection Vulnerabilities in MCP Servers
Zehua Zhang, Jie Hu, Pratham Hegde, Aditya Maheshbhai Gabani, Souradip Nath, Yibo Liu, Siyu Liu, Hongkai Chen, et al.
cs.CR · cs.AI
Quick verdict
Metadata-only no-box IPI analysis: MCPSEC recovers 94/95 confirmed vulnerable tools (98.9% recall); author-annotated labels and 65.7% precision warrant care.
Summary
The paper proposes 'no-box' vulnerability analysis, a paradigm in which an analyst reasons about potential vulnerabilities using only functionality metadata — no source code, binaries, or runtime interaction. The core idea is that metadata implies 'irreducible data flows' (minimal source-to-sink skeletons shared by all conforming implementations), which can be annotated with hypothesized security conditions to yield 'Theory-of-Concept' (ToC) attack scenarios for later validation. The authors implement MCPSEC, a two-stage LLM pipeline (speculative data-flow reconstruction plus a four-axis risk rubric: payload fidelity, attacker controllability, semantic executability, sanitization) that audits MCP servers for indirect prompt injection (IPI) from registration-time tool metadata alone, using GPT-5.4. On 177 tools from 20 popular open-source MCP servers, MCPSEC flagged 143 tools versus 97 for a single-prompt LLM baseline. Human evaluators (all authors) confirmed 95 tools vulnerable via ethically controlled PoCs; MCPSEC recovered 94 (98.9% recall) versus 80 (84.2%) for the baseline, with lower precision (65.7% vs 82.5%). RQ2 source-code checks found 24.5% of ToCs implausible, and 92.3% of examined tools applied no sanitization. Main limitations: author-annotated ground truth with no confirmed non-vulnerable negatives, a single frontier model, moderate pre-adjudication agreement (~80–83%), and a deliberately favorable, high-profile dataset.
Review complete
Assessment at a glance
4
risk flags identified
● high 1
● medium 2
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Vulnerability hypotheses can be formulated from functionality metadata alone by reasoning over irreducible data flows, without source, binaries, or runtime interaction (the no-box paradigm).
- MCPSEC recovers 94 of 95 human-confirmed IPI-vulnerable MCP tools (98.9% recall) from registration-time metadata alone, versus 80 (84.2%) for a single-prompt LLM baseline.
- Graph-conditioned reasoning over reconstructed data flows is what recovers Category C (authenticated SaaS) cases that metadata-only prompting misjudges as trusted internal state.
- IPI sanitization is rare in widely deployed MCP servers: 92.3% (132) of examined tools apply none on the path to the LLM context.
- No-box analysis is practical as a triage front-end: ~$1.65 per server, no credentials, no interaction risk to live services.
Strengths
Genuinely novel paradigm with a clear formal framing
§III defines no-box analysis via 'irreducible data flows' — the minimal source-to-sink skeleton shared by all metadata-consistent implementations — grounding later ToC generation; §IV motivates it with a measurement that 3,007/18,770 (16.0%) of MCP registry servers expose no public implementation.
Unusually honest staged evaluation architecture
Table IV freezes MCPSEC predictions before annotation and tests them against progressively stronger evidence (metadata RQ1 → source RQ2 → benign runtime RQ3 → PoC RQ4); §VII-E explicitly reports recall as the only exact metric and precision as a lower bound.
Real end-to-end validation, not just plausibility scores
Actionable ecosystem and spec-level findings
Ethically careful PoC protocol
Weaknesses
Ground truth contains no established negatives
Author-annotation circularity
RQ2 undercuts the core hypothesis-quality claim
Single-model pipeline with no ablations shown
Editorial/numeric inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper introduces 'no-box vulnerability analysis': reasoning about vulnerabilities from functionality metadata only — no source, binaries, or runtime interaction — positioned against white/gray/black-box paradigms (§II). The conceptual core is the 'irreducible data flow' (§III-C): the minimal source-to-sink skeleton shared by every implementation consistent with metadata M. Annotating such flows with hypothesized security conditions yields 'Theory-of-Concept' (ToC) attack scenarios for later validation. The instantiation, MCPSEC, audits MCP servers for indirect prompt injection (IPI) using only registration-time tool metadata (§IV–V), motivated by a measurement that 3,007 of 18,770 registry servers (16.0%) are closed-source. Central claims: (1) metadata alone suffices for useful vulnerability hypotheses; (2) MCPSEC recovers 94/95 human-confirmed IPI vulnerabilities (98.9% recall) versus 80/95 (84.2%) for a single-prompt LLM baseline; (3) the ecosystem is under-defended — 92.3% of examined tools apply no sanitization.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'no-box' vulnerability analysis?
A paradigm proposed by this paper in which an auditor reasons about potential vulnerabilities using only functionality metadata — no source code, no binaries, no interaction with the running system. Because every implementation consistent with the metadata must contain certain minimal 'irreducible data flows' (e.g., user input reaching a database), the analyst can hypothesize vulnerabilities as 'Theories-of-Concept' (ToCs) and validate them later if access becomes available.
What results did MCPSEC achieve?
On 177 tools from 20 popular MCP servers, MCPSEC flagged 143 tools as potentially vulnerable to indirect prompt injection using only registration-time tool metadata. Human-constructed proof-of-concepts confirmed 95 tools vulnerable; MCPSEC recovered 94 of them (98.9% recall) versus 80 (84.2%) for a simpler single-prompt LLM baseline, at the cost of lower precision (65.7% vs 82.5%), at about $1.65 per server.
What is an indirect prompt injection (IPI) vulnerability in an MCP server?
It occurs when an MCP tool retrieves attacker-influenced external content (a web page, repository comment, chat message) and returns it to the LLM's context, where that content can be interpreted as instructions that hijack the agent — for example, making it exfiltrate stored credentials through other tools. The paper found such injection channels are common: 92.3% of examined tools apply no sanitization on the path to the LLM context.
Is MCPSEC's code or dataset available?
Not yet. The paper states the authors will release analysis data and source code upon acceptance, and will publish generated ToCs and PoCs only after affected vendors fix the reported issues. At review time, no artifact is publicly available.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The sampler skipped middle text: part of §III-A formalization, Stage 2 design/§V-B rubric full text, §VII-C (RQ2 conclusion) and the RQ3 setup, and portions of Limitations/Related Work are absent from the excerpt; judgments touching those areas rely on the visible remainder.
- Figure 1 and Tables I–IX appear only as extracted caption/cell text; visual layout details of the pipeline figure were judged from its caption and surrounding prose.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.