Defining AI Agents: A Compendium of Criteria, Metrics, and Benchmarks
cs.AI · cs.MA
Quick verdict
Survey organizing agent evaluation into five agenticness dimensions plus a public compendium; useful taxonomy, but no new empirics and editorial errors.
Summary
This survey tackles the absence of a standard definition of 'AI agent' by organizing evaluation methods around five dimensions of agenticness—environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence—following Bent (2025). For each dimension the authors review how the capability has been conceptualized and catalog metrics, benchmarks, and frameworks: success weighted by path length (SPL) for navigation; MultiCAT-Bench and the Tool Dexterity Index for tool-use; consistency, robustness, and reliability metric families with measurement protocols; the cognitive efficiency score; predictability via calibration, AUC-ROC, and Brier score; four autonomy taxonomies plus safety metrics; the FIT-Eval intentionality protocol; goal execution and resilience metrics; and drift/hysteresis metrics for temporal coherence. The paper argues that existing benchmarks (e.g., HAL's nine benchmarks, MMAU) measure task-specific capability rather than components of agenticness dimensions, and identifies evaluation gaps: no hybrid-architecture evaluation, no model-agnostic navigation metrics, undefined task-mastery criteria, and no temporal-coherence benchmarks. It introduces the Agent Compendium, a public website to be maintained through AI-assisted literature monitoring plus expert review. Main limitations: the survey describes no systematic search methodology; the dimension-to-component mapping is asserted rather than validated; reproduced metric definitions contain inconsistencies (e.g., Table 3's outcome-consistency normalization); the temporal-coherence dimension rests on a single recent preprint (McCants, 2026); and the text contains cross-reference errors (Sections 1.3 and 2.6 cited for content in Sections 3.1.3 and 3.2.6) plus a citation-year mismatch (Anthropic 2026 in text, 2024 in references). No empirical results are presented.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The term 'AI agent' lacks a shared definition, which undermines evaluation, comparison, reproducibility, and user trust in agent systems (§1).
- Existing benchmarks (e.g., HAL's nine benchmarks, MMAU) measure task-specific capabilities rather than components of the five agenticness dimensions (§2.2).
- Agent evaluation methods can be systematically organized under environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence (§3).
- No standardized repository of agent-characteristic evaluation methods existed prior to this work; the public Agent Compendium fills that niche (§2.2, §5).
- Four evaluation areas are empty and merit active research: hybrid-architecture evaluation, model-agnostic navigation metrics, task-mastery criteria, and temporal-coherence benchmarks (§4.1).
Strengths
Fills a genuine organizational gap
§2.2 argues agent evaluation is siloed; §3 then connects classical agent literature (Wooldridge & Jennings 1995; Brooks 1986) with LLM-era infrastructure (HAL, MMAU, AgentDojo) under one five-dimension structure, which pre-LLM-era surveys rarely do.
Metrics reproduced with operational protocols, not just names
Tables 3, 6, 13 and 20 give equations plus measurement protocols (K runs per task, confidence binning for ECE, clamped ratios for fault/environment/prompt robustness), making them directly implementable — more actionable than typical survey listings.
Specific, honest gap identification
Multi-taxonomy juxtaposition enables comparison
Weaknesses
No systematic review methodology
Citation and entry errors undermine reference-work reliability
Reproduced metric definitions contain internal inconsistencies
Dimension boundaries overlap without an adjudication rule
Single-source dimension with unoperationalized metrics
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a narrative survey (arXiv 2609.11018v1, 10 Sep 2026, ~29 pages) that organizes AI-agent evaluation methods around the five agenticness dimensions of Bent (2025): environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence (the last attributed to McCants, 2026). The stated claims (§1, §2.2, §5): (1) definitional ambiguity of "agent" harms evaluation, reproducibility, communication, and user trust (citing Shome et al.'s 2026 finding of "prompt gambling"); (2) existing benchmarks measure task capability, not components of agenticness; (3) no standardized repository of agent-characteristic evaluations exists — a gap the public Agent Compendium (agent.duketrustlab.com) fills. §4 commits to four specific evaluation gaps as future work.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Agent Compendium?
A free public website (agent.duketrustlab.com) introduced by this paper that organizes AI-agent evaluation methods — criteria, metrics, benchmarks, and frameworks — under five capability dimensions: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. The authors plan to keep it current via automated literature monitoring followed by human expert review.
What are the five dimensions of agenticness?
Per Bent (2025), the framework this survey builds on: environmental interaction, learning and adaptation, autonomy, goal-directed behavior, and temporal coherence. 'Agenticness' means the degree to which a system exhibits agent-like characteristics; the paper maps existing metrics and benchmarks onto each dimension.
Does this paper propose new benchmarks or run experiments?
No. It is a survey: it catalogs and organizes existing metrics and benchmarks (e.g., SPL, MultiCAT-Bench, AgentDojo, HAL), reproduces metric definitions with equations, and recommends new evaluation areas — hybrid-architecture evaluation, model-agnostic navigation metrics, task-mastery criteria, and temporal-coherence benchmarks — as future work.
How does this survey differ from other LLM-agent evaluation surveys?
Zhu et al. (2025) categorize evaluations by environment (code, web, OS, mobile, games) and capability; HAL (Kapoor et al., 2025) is evaluation infrastructure and a leaderboard. This survey instead organizes evaluation methods by the definitional components of agenthood itself, so systems can be compared on what makes them agentic rather than on task domains.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Equations were extracted from PDF and may flatten subscripts or inline conditions; the Table 3 normalization mismatch and the Eq. (3) garbling were verified against the extracted text but should be checked against the source PDF before being raised with the authors.
- The live Agent Compendium website could not be inspected; assessment of the resource (§5) relies solely on the paper's own description.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.