SoK: When Safe Agents Fail Together: The Security of Multi Agent LLM Systems
Rui Yang, Junjie Xu, Zhengyu Liu, Neil Fendley, Yang Hong, Ziyang Li, Yinzhi Cao
cs.CR · cs.AI
Quick verdict
SoK mapping 197 multi-agent LLM security works through an A→I→R framework and a 44-work benchmark audit; rigorous lens, but coding validity is unverified.
Summary
This systematization-of-knowledge paper organizes the security of multi-agent LLM systems (MAS) around end-to-end execution rather than component-level taxonomies. From a corpus of 197 works — 115 peer-reviewed or with at least ten citations, 82 emerging — the authors derive a common representation: adversary position (A1–A4) through interaction interface (I1–I6) to system-level risk (R1–R7), conditioned by eight configuration dimensions (C1–C8), and group recurring executions into eight attack paths (P1–P8). Defenses are compared through a five-part contract (path target, observation, intervention, trust boundary, recovery) and four control primitives (D1–D4). An audit of 44 evaluation works finds that 42 use fixed membership, only 6 of 21 interaction-effect evaluations include matched single-principal comparisons, the 13 works computing ASR define it incompatibly, and only Agent-BOM reports a dependency graph. The main limitations are methodological: coding was performed by the authors without reported inter-rater reliability; the corpus mixes peer-reviewed venues with Zenodo, SSRN, and low-visibility journals; and Table 1's comparison of prior surveys is self-coded. No new experiments are presented; findings are qualitative syntheses. The paper is strongest where it is most conservative — flagging P5 and P8 as evidence-sparse and demanding counterfactual-verified interaction claims — and offers a reusable vocabulary for MAS security evaluation.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Observing a failure in a multi-agent setting is not evidence of a MAS-specific effect; such claims require relation-removal counterfactuals or relation-preserving comparisons (Section 2.1).
- The A→I→R execution representation — adversary position, interaction interfaces, system-level risk, conditioned by C1–C8 — unifies fragmented attack mechanisms into eight recurring paths (P1–P8).
- Defense claims should be stated as five-part contracts, and current defenses systematically exhibit local-protection-without-path-closure and containment-without-recovery gaps (Section 5.3).
- Current evaluation practice under-supports interaction isolation (only 6 of 21 relevant audits use matched single-principal comparisons), metric comparability (13 incompatible ASR definitions), component reuse, and open-system operation (42 of 44 fixed membership).
Strengths
Execution-centered unit of analysis with a falsifiable claim standard
Section 2.1's four interaction-effect roles (inherited / interaction-amplified / composition-induced / structurally multi-agent) plus the relation-removal counterfactual requirement give the field a concrete test for when a failure is genuinely MAS-specific rather than an inherited single-agent bug.
Honest evidence bookkeeping throughout
Table 2 marks P5 as 'Sparse; adjacent evidence' and P8 as 'Sparse; direct evidence' (CORBA [195] the sole direct citation); Section 4.2 concedes R4's direct evidence 'remains thinner'; the R1→R4→R3→R6→R5 chain is explicitly labeled a cross-paper synthesis, not a demonstrated chain.
Quantified benchmark audit with actionable counts
Defense-side symmetry via a reusable contract
Immediately actionable metric proposal
Weaknesses
Coding validity asserted, not demonstrated
Corpus quality heterogeneity with flattened evidentiary weight
Self-coded comparison to prior surveys
Audit codings sometimes too broad to inform
Proposals undemonstrated on worked examples
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This SoK reframes multi-agent LLM system (MAS) security around end-to-end execution. Its central normative move is Section 2.1's four-way decomposition of how interaction can figure in a security result — inherited, interaction-amplified, composition-induced, structurally multi-agent — with relation-removal counterfactuals required before any MAS-specific causal claim. The stated contributions: an execution-centered unit of analysis; the A→I→R representation (four adversary positions A1–A4, six interaction interfaces I1–I6, seven risk families R1–R7) conditioned by eight configuration dimensions (C1–C8), yielding eight attack paths (P1–P8); a five-part defense contract with four control primitives (D1–D4); and an audit of 44 evaluation works. Claims are consistently hedged — the R1→R4→R3→R6→R5 risk chain is explicitly labeled 'a cross-paper synthesis rather than a claim that one study demonstrates the full chain' (Section 4.2).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the A-I-R framework for multi-agent LLM security?
A representation introduced by this SoK that traces an attack from its adversary position (A1 external, A2 user-level, A3 member, A4 infrastructure) through interaction interfaces (I1 admission, I2 message transfer, I3 state propagation, I4 authority transfer, I5 collective commitment, I6 detection/intervention) to a system-level risk (R1 containment failure, R2 loss of independence, R3 collective-decision integrity, R4 trajectory/goal integrity, R5 cross-principal confidentiality, R6 transitive authorization, R7 availability/resource isolation).
How many papers does this multi-agent security SoK analyze?
197 works on multi-agent LLM security, split into Set 1 (115 peer-reviewed or works with at least ten citations) and Set 2 (82 emerging works). Of these, 44 benchmark/evaluation works are audited in detail, with per-work mappings in the paper's Table 4.
What gaps does the paper identify in MAS security evaluation?
Four: (1) evaluations rarely isolate whether a failure actually depends on multi-agent interaction — only 6 of 21 relevant audits use matched single-principal comparisons; (2) metrics like attack success rate are defined incompatibly across the 13 benchmarks computing them; (3) benchmark components are rarely configurable or extensible across systems; (4) 42 of 44 benchmarks use fixed membership, leaving open-system operation with changing trust boundaries unevaluated.
Is the paper's literature corpus or code publicly available?
Not yet. The authors state the literature corpus and associated artifacts will be released publicly with the final version of the paper; no public artifact accompanies the arXiv v1 at review time.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: the C6–C8 configuration-dimension definitions (end of §3.1), the remainder of attack path P8 (§4.3), parts of §6.2.2–6.2.3, and references [77]–[83] and [176]–[178] were skipped by sampling; no review judgment above relies on that skipped content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.