CivBench: A Long-Horizon Benchmark for Tool-Mediated Agents in Civilization VI
Austin Tudor David Andrews, Liam Wilkinson, Jamie Heagerty, Harry Coppock, Jakob Nicolaus Foerster, Rui Ponte Costa
cs.AI
Quick verdict
CivBench: an MCP-based Civ VI benchmark with novel trace metrics PMR and RAG@10; a candid but small 23-run pilot that measures agent behaviour, not rankings.
Summary
CivBench connects language-model agents to Civilization VI through the Model Context Protocol, exposing 76 tools and a narration layer that converts visual game state into text the agent must explicitly query. Because non-local state enters context only on request, the environment separates information availability from information use, enabling two trace-level metrics: Proactive Monitoring Rate (PMR), the share of non-infrastructure tool calls devoted to strategic monitoring, and RAG@10, the fraction of commitments stated in structured planning diaries executed within ten turns. Across 23 admissible full-game runs from four model families (Claude Opus 4.6, GPT-5.4, Gemini 3.1 Pro with 6-8 runs each; Kimi K2.5 with one), the authors explicitly disclaim model ranking: aggregate outcomes do not discriminate (3 victories in 23 runs; Fisher's exact p=0.488; Kruskal-Wallis p=0.594). Two patterns recur. PMR stays between 0.96% and 2.13% despite playbook guidance to query victory progress every 20 turns, with actual intervals of 30-75 turns, and in 7 of 20 detectable defeats no victory query occurred within the 20-turn warning window. RAG@10 ranges from 48.2% to 65.8%, with LLM-extracted commitments validated against a single human rater on 50 items (kappa=0.879). Limitations include the small uneven sample, absence of random or scripted baselines, the shared human playbook confound, and reliance on an LLM labeller. Environment, logs, and analysis pipeline are open-sourced.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CivBench's MCP interface with a query-only narration layer separates information availability from information use, enabling direct measurement of attention allocation and plan execution from tool-call traces.
- Across 23 admissible runs, aggregate outcomes (win/loss, normalised score) do not discriminate between the four model families (Fisher p=0.488; Kruskal–Wallis p=0.594), so the dataset is a pilot reference set, not a ranking.
- Agents under-monitor queryable strategic state: PMR is 0.96–2.13% of non-infrastructure calls, victory progress is queried only every 30–75 turns despite per-20-turn playbook guidance, and 7 of 20 detectable defeats lacked any query in the 20-turn warning window.
- Agents frequently fail to execute their own near-term commitments: RAG@10 is 48.2–65.8% across the three families with sufficient commitments, using an LLM extraction pipeline validated at κ=0.879 against a single human rater.
- Both patterns are deviations under instruction rather than evidence of absent capability, suggesting monitoring may require explicit mechanisms (schedules, commitment trackers) instead of emergent behaviour.
Strengths
Metrics computed from traces, not outcomes
The narration layer (Sec. 3.3) makes 'available but not queried' state observable, so PMR (0.96–2.13%) and RAG@10 (48.2–65.8%) are derived directly from tool-call logs — exposing variation that Sec. 5.1 shows aggregate scores compress (12 of 13 candidate metrics in Table 5 have ICC near zero or negative).
Unusually candid statistical self-assessment
The paper reports its own null results (Fisher p=0.488, Kruskal–Wallis H=1.90 p=0.594 in Sec. 5.1), labels 12/13 ICC-tested metrics 'noise' (Table 5), and consistently frames findings as pilot observations under a fixed protocol rather than rankings.
Failure-linked analysis, not just rates
Validated LLM-labelling pipeline for RAG
Release quality and cost transparency
Weaknesses
Sample too small and uneven for the headline weight
No baseline of any kind to calibrate PMR
Playbook confound narrows the construct
Single human rater and small cells in RAG validation
Commercial, version-pinned environment limits independent re-running
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
CivBench — Structured Preliminary Review
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CivBench?
CivBench is an open-source benchmark that connects language-model agents to the strategy game Civilization VI via the Model Context Protocol (MCP). Agents play full 300+ turn games through 76 tools and a text 'narration layer' that only reveals game state when the agent explicitly queries it. The paper introduces two trace-based metrics — Proactive Monitoring Rate (PMR) and RAG@10 — measuring whether agents retrieve available strategic information and whether they execute plans they state. Code and logs are released on GitHub/HuggingFace; users need their own copy of Civilization VI.
Does CivBench show which AI model is best at Civilization VI?
No. The authors explicitly state the 23-run dataset is a pilot, not a model ranking: with only 3 victories across four model families, Fisher's exact test (p=0.488) and Kruskal–Wallis (p=0.594) find no significant between-model differences. The contribution is the environment and the two behavioural metrics, not a leaderboard.
What is the 'sensorium effect' in the CivBench paper?
It is the gap between what a human player passively sees (minimap, score ticker, health bars) and what an agent sees, since CivBench agents only observe state they explicitly query. Despite playbook instructions to check victory progress every 20 turns, models queried it only every 30–75 turns, and in 7 of 20 detectable defeats they never queried it within the final 20-turn warning window before losing.
What is RAG@10?
RAG@10 (Reflection–Action Gap at 10 turns) measures the share of concrete commitments an agent writes in its structured planning diary that are actually executed within the next 10 turns. In the paper it ranges from 48.2% to 65.8% across the three model families with enough data, meaning roughly a third to a half of stated near-term plans were not carried out. Commitments were extracted by an LLM (Claude Haiku 4.5) and validated against human labels (κ=0.879 on 50 items).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract supplied with the task is cut off mid-sentence, but the embedded full text contains the complete abstract, body, appendices, and NeurIPS checklist, so no substantive paper content was missing.
- Figures are present only as embedded caption/axis text in the extraction (e.g., Fig. 3–4 values, Fig. 5 counts), so visual details such as CI bar widths could not be inspected directly.
- The affiliation line for Harry Coppock renders as three concatenated strings ('UK AI Security Institute Imperial College London 10 Downing Street'), an extraction artifact of the multi-affiliation layout; the true affiliation structure cannot be verified from this text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.