Black-Box Red Teaming of Agentic AI: A Taxonomy-Driven Framework for Automated Risk Discovery
Divyanshu Kumar, Nitin Aravind Birur, Tanay Baswa, Sahil Agarwal, Prashanth Harshangi
cs.AI
Quick verdict
Claims a 7-domain taxonomy + automated black-box agent red teaming; results suggestive but from 2 toy systems, 4 models, judge-only scores, no baselines.
Summary
The paper presents a black-box framework for security evaluation of LLM-based agentic systems, built on three components: a seven-domain risk taxonomy (governance, output quality, tool misuse, privacy, reliability/observability, agent behavior, access control), an automated SAGE-RT pipeline generating 120 multi-turn adversarial scenarios per domain, and LLM-judge scoring (GPT-4o, temperature 0, 5-7 binary rubric criteria) with manual review of high-risk findings. The authors evaluate two systems — a CrewAI single-agent Restaurant Receptionist built in-house on synthetic SQLite data, and Unit 42's AutoGen multi-agent Stock Advisory Assistant — each across four models (GPT-4.1-nano, Mistral Small, Gemini-2.5-Flash, Kimi K2). Reported results include 56.25 average governance risk on both systems, 65 privacy risk for the multi-agent configuration, agent-behavior risk peaking at 85 (GPT-4.1-nano, Stock Advisory), and similar risk rankings across models, which the authors read as architecture — not model — driving vulnerability. Main limitations: only two small demonstration systems (N=8 agent-model combinations) underpin sweeping claims; scenario validation regenerates scenarios until they elicit concerning responses, creating selection circularity; all quantitative scores come from a single LLM judge with thin agreement statistics; there is no comparison against existing agent-security benchmarks and no defense evaluation; and the reported confidence intervals and a '500+ test cases' figure are inconsistent or unexplained.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A seven-domain taxonomy plus fully automated SAGE-RT generation enables black-box agent risk evaluation requiring only basic system descriptions, with 120 adversarial scenarios per domain.
- Governance risk averages 56.25 across both tested systems and privacy risk reaches 65 in the multi-agent Stock Advisory configuration.
- Agent-behavior vulnerabilities peak at 85 (GPT-4.1-nano on the Stock Advisory assistant), with high cross-model variance.
- Vulnerability patterns cluster by agent architecture rather than base model, implying architecture-driven — and therefore predictable — risk profiles.
- Current agent systems are claimed to be 'fundamentally unprepared for adversarial environments,' a conclusion the evidence base does not fully support.
Strengths
Operationalized, well-mapped taxonomy
Seven domains expanded into 35 sub-risks and concrete observable failure modes (appendix 'Risk Categories Expanded'), grounded in OWASP agentic guidance and NIST RMF with per-sub-risk citations — more actionable than the capability-level taxonomies the paper critiques.
Fully reported quantitative results
Tables 1-2 give complete per-domain, per-model scores for both systems; every derived mean/std in the prose (Restaurant 29.49±3.65, Stock 40.53±5.70, Stock agent-behavior 46.25±22.47) reproduces exactly from the tables, enabling verification.
Transparent failure documentation
Honest limitations section
Weaknesses
Evidence base far too small for the claims
Circular scenario validation
Judge-dependent measurement with unquantified agreement
Decorative formalism and unspecified weights
No external baselines and inconsistent counts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This workshop paper (arXiv:2609.09647; AAAI 2026 Workshop on LLM-based Multi-Agent Systems) proposes a black-box security evaluation framework for LLM agents comprising: (1) a seven-domain risk taxonomy — governance, agent output quality, tool misuse, privacy, reliability & observability, agent behavior, access control — expanded into 35 sub-risks; (2) automated SAGE-RT generation of 120 multi-turn adversarial scenarios per domain; (3) GPT-4o judge scoring on 5-7 binary rubric criteria (Eq. 4) with human review of high-risk findings. Validation covers two systems: a CrewAI single-agent Restaurant Receptionist built by the authors (synthetic SQLite backend: 500 reservations, 50 tables, 100 menu items) and Unit 42's AutoGen multi-agent Stock Advisory Assistant (three agents, round-robin group chat, simulated trades), each across four models (GPT-4.1-nano, Mistral Small, Gemini-2.5-Flash, Kimi K2). Headline numbers from Tables 1-2: governance risk averages 56.25 on both systems, privacy 65 on Stock Advisory, agent behavior 85 for GPT-4.1-nano on Stock Advisory. The authors conclude vulnerabilities cluster by architecture rather than model.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SAGE-RT?
SAGE-RT (Synthetic Alignment data Generation for Safety Evaluation and Red Teaming) is an automated red-teaming pipeline from the same research group (Kumar et al., 2024). In this paper it automatically generates taxonomy-aligned, multi-turn adversarial test scenarios — 120 per risk domain — and executes them against target agent systems without human intervention, before LLM-judge scoring.
What are the seven risk domains in this agent taxonomy?
Governance (reward/objective manipulation), agent output quality (hallucination, bias, toxicity), tool misuse (e.g., SQL injection via natural language), privacy (PII and data leakage), reliability and observability, agent behavior (manipulation, unsafe actuation), and access control (privilege escalation, confused deputy). They expand into 35 concrete sub-risks mapped to observable failure modes.
What were the main findings of the paper?
Testing a CrewAI single-agent restaurant receptionist and an AutoGen multi-agent stock advisory assistant with four models, governance risk averaged 56.25 on both systems, privacy risk reached 65 in the multi-agent setup, and agent-behavior risk peaked at 85 with GPT-4.1-nano. Risk rankings were similar across models, which the authors interpret as architecture-driven vulnerability — though only two systems were tested.
Is code or data available for this paper?
The conclusion states the framework and taxonomy are open-source, but the paper text provides no repository URL, evaluation rubrics, seed prompts, or generated scenario sets. The underlying SAGE-RT system is described in a separate 2024 paper by the same group.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (risk heatmap, box plots, architecture diagrams) are images in the PDF; their numeric content was recovered from captions and Tables 1-2, which together cover all reported scores.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.