The Agent Incident Registry: Toward Preventing Repeated AI Agent Failures
Divyanshu Kumar, Rohith HN, Nitin Aravind Birur, Sahil Agarwal, Prashanth Harshangi
cs.AI
Quick verdict
Curates 487 source-linked AI-agent incident records with unusually honest limits; valuable for evaluation audits, but labels rest on one non-blinded curator.
Summary
The paper introduces the Agent Incident Registry (AIR), a source-linked catalog of 487 agent-related events disclosed between 2022 and 2026, each carrying a verbatim evidence quote, a stable identifier, and missingness-aware labels for causal role, disclosure class, mechanism, and outcome. From 602 candidates, documented merge rules produced 487 records, and 52 unsupported leads were quarantined. In the 336-record primary population (generative systems in which the agent acted), 81 records (24%) involve realized harm, concentrated in in-the-wild (92 of 110) and safety-failure (87 of 92) classes, while responsible disclosures (2 of 199) and research demonstrations (3 of 86) are predominantly demonstrated capability. The authors explicitly decline incidence, prevalence, or causal claims, instead reporting Wilson intervals and a first-source-host cluster bootstrap (the 24% share spans 15–38%; publisher exclusions move it to 40%). A second reviewer checked all records but saw existing labels, so no inter-rater agreement is reported. A pinned projection maps InjecAgent's 1,054 attacker-triggered cases onto three of AIR's twelve surfaces, exposing a no-adversary gap: 92 safety failures that attack-only evaluations do not cover. Main limitations: single-curator labeling with non-independent review, English-only collection without geographic coding, weak coverage of tool-privilege and control-outcome fields, and the employer's conflict of interest in selling agent-security products.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 1
● low 3
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AIR's 487 source-linked, missingness-aware records make agent-mechanism fields (causal role, autonomy, surface, vector, guardrail) mandatory where prior repositories' documented core schemas leave them optional (Table 1).
- In the 336-record primary population, 24% (81/336) of records involve realized harm, and this share characterizes collection composition rather than deployment risk.
- Realized outcomes concentrate in in-the-wild (92/110) and safety-failure (87/92) classes, while responsible disclosures (2/199) and research demonstrations (3/86) are almost entirely demonstrated capability.
- Attack-only evaluations such as InjecAgent (1,054 attacker-triggered cases across 3 of 12 AIR surfaces) structurally cannot cover AIR's 92 no-adversary safety failures, exposing an evaluation-coverage gap.
Strengths
Epistemic discipline in schema and admission rules
Realized harm is separated from demonstrated capability, causal role from disclosure route (§3.3); admission requires a fetched URL plus verbatim quote; 52 unsupported leads are quarantined and enter no statistic; 457/487 records are high-confidence (§3.2).
Uncertainty is quantified, not gestured at
Every proportion carries a 95% Wilson interval plus a deterministic first-source-host cluster bootstrap (10,000 replicates, fixed seed); Fig. 3 shows the primary-population 24% share spanning 15–38%, publisher exclusions moving it to 40%, and agency_debatable bounds of 27–42%.
The no-adversary evaluation gap is a genuinely useful finding
Reproducibility infrastructure is unusually complete
Weaknesses
Label validity rests on a single, non-blinded workflow
Headline differentiating fields are largely unpopulated
Limited signal beyond composition diagnostics
Selection and coverage blind spots
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
AIR is a curated registry of 487 source-linked records of agent-related events disclosed between 2022 and 2026, built by an Anaconda/Enkrypt AI team. The paper advances three claims: (1) a schema contribution — mandatory agent-mechanism fields (causal role, disclosure class, autonomy, attack surface, vector, guardrail) with explicit missingness, which the authors argue (Table 1) general repositories such as AIID, OECD AIM, and AIAAIC do not require in their core documented schemas; (2) descriptive composition analysis — most prominently 81/336 (24%; Wilson 20–29%) realized-harm records in the primary population of generative systems in which the agent acted; and (3) an evaluation-coverage audit, worked through a pinned InjecAgent projection. Unusually for incident papers, every section hedges deliberately: the registry is declared a selected corpus, not a census, supporting case retrieval and evaluation-scope auditing only — never failure-rate, prevalence, vendor-risk, or control-efficacy estimation (§1, §5, §6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Agent Incident Registry (AIR)?
AIR is a public, source-linked catalog of 487 AI-agent-related events disclosed between 2022 and 2026. Each record has a verbatim evidence quote, a stable identifier (AIR-YYYY-NNNN), and labels for causal role, disclosure class, mechanism, and outcome. It is built to support case retrieval and auditing of what agent-security evaluations cover, not to estimate failure rates.
Does the paper show that 24% of AI agent deployments cause harm?
No. The 24% (81 of 336 records) is the realized-harm share within the registry's own selected sample of public disclosures. The authors explicitly state the registry has no deployment denominators and cannot estimate incidence, prevalence, deployment risk, or control efficacy.
What is the no-adversary gap the paper identifies?
AIR contains 92 safety-failure records with no attacker-supplied trigger (87 of them with realized harm), such as destructive shell actions or uncommanded publication. Benchmarks like InjecAgent consist entirely of attacker-triggered indirect prompt injection cases, so evaluations of that style never exercise spontaneous, non-adversarial agent failures.
Who built AIR and is the data available?
Researchers affiliated with Anaconda/Enkrypt AI. An anonymized artifact with the frozen corpus (2026-09-05), coding protocol, ASI crosswalk, InjecAgent mapping, and analysis code accompanies the paper, with persistent archiving promised upon acceptance; the project page is enkryptai.com/air.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure graphics (Figs. 1–5) are images and cannot be inspected from text extraction; assessments rely on captions, tables, and in-text statistics, all of which were present in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.