CIVI: A Framework for Diagnosing Search Agent Failures in Civic Information
Dingying Liu, Yunshun Zhong, Wentao Zhang, Yiyuan Li
cs.AI
Quick verdict
A well-built 4,097-QA civic benchmark; its ARISE diagnostic attributes 72.1% of search-agent failures to retrieval, though the human baseline uses 5 people.
Summary
The paper introduces CIVI, a benchmark of 4,097 expert-verified multiple-choice QA pairs grounded in 576 authoritative government web pages, stratified across four COFOG functions, three countries (Canada, US, Australia) and three jurisdictional levels. Questions were generated by Claude Opus 4.6 with verbatim-quote grounding and dual human validation (96.8% raw agreement; 94.1% retained). Ten frontier LLMs were evaluated through a shared Exa search substrate under agentic, search-disabled and single-retrieval RAG conditions. The best model, Gemini 3.1 Pro, scores 78.28% versus a 92.67% human baseline established by five participants on 150 questions; accuracy declines monotonically with jurisdictional level (65.6% federal to 56.7% municipal). The ARISE diagnostic assigns each failure to one of four mutually exclusive modes via source-injection ablation plus trace analysis: 72.1% of failures are retrieval-bound (34.5% search bypass, 37.6% retrieval failure) and only 2.3-4.8% persist with the gold page in context. Agentic search outperforms closed-book by 10.0 points and single-retrieval RAG by 9.1. Main limitations: the human baseline uses only five unpaid participants, all questions are LLM-generated under strict exact-set-match scoring, and the headline attribution is conditioned on a single search substrate and a tight retrieval budget.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No evaluated frontier search agent matches an attentive human baseline: the best model (Gemini 3.1 Pro, 78.28%) trails 92.67% by 14.4 points (Table 2).
- ARISE attributes 72.1% of all observed failures to retrieval-bound causes, split as 34.5% search bypass and 37.6% retrieval failure, with only 2.3-4.8% comprehension failures (Figure 4).
- Accuracy declines monotonically with jurisdictional level, from 65.6% (federal) through 62.6% (state/provincial) to 56.7% (municipal), while inter-model spread widens to 38.1 pp at the local level (Figure 1, Section 4.1).
- Agentic search improves average accuracy by 10.0 pp over search-disabled answering and 9.1 pp over single-retrieval RAG, with RAG adding less than 1 pp over closed-book (Table 3).
- Authoritative-source citation (meanAHR) is only weakly aligned with general capability rank, correlating at Spearman rho = 0.27 with LMArena Text-Arena Elo (Section 4.3).
Strengths
Rigorous, internally consistent dataset construction
576 pages across 36 stratification cells with candidate pools of at least 4x the target, an AI-use-restriction exclusion filter (Section 2.1, Appendix E); dual independent validation with 96.8% raw agreement and 94.1% acceptance (Section 2.3). Table 4 cell counts sum exactly to 4,097 and match Figure 1's per-level n (1,310/1,334/1,453).
Mechanical, checkable failure taxonomy
ARISE's four modes are mutually exclusive and exhaustive by trace definition (Section 5): no invocation means bypass, invocation without the gold URL means retrieval failure, gold URL retrieved means grounding failure, and error under injected gold page means comprehension failure. Only 2.3-4.8% of errors persist with the gold page in context (Figure 4).
Behavioral metrics beyond accuracy
Deployment-realistic and transparent protocol
Weaknesses
Human baseline rests on five participants
Headline attribution conditioned on one substrate and a tight budget
All-LLM-generated questions under strict exact-set-match
Grounding-failure mode under-analyzed
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CIVI (Civic Intergovernmental Verification of Information) is presented as the first framework jointly spanning cross-national and inter-jurisdictional civic QA under an international classification standard (Sections 1 and 7). The claims are threefold: (1) a 4,097-question expert-verified benchmark stratified by 4 COFOG functions x 3 countries x 3 jurisdictional levels (Table 1); (2) ARISE, a deterministic four-mode failure attribution combining source-injection ablation with search-trace analysis (Section 5); and (3) the empirical finding that no frontier search agent matches an attentive human baseline, with 72.1% of failures retrieval-bound. Novelty claims are appropriately hedged ('to our knowledge'), and the internal bookkeeping checks out: Table 4's cell counts sum exactly to 4,097 and match Figure 1's per-level n (1,310 / 1,334 / 1,453).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CIVI benchmark?
CIVI is a benchmark of 4,097 expert-verified multiple-choice questions about government services, built from 576 authoritative public-sector web pages and stratified across four government functions from the UN COFOG standard, three countries (Canada, US, Australia), and three jurisdictional levels (federal, state/provincial, municipal).
What does ARISE measure and how does it work?
ARISE is CIVI's failure-attribution procedure. Each wrong answer is rerun with the gold source page injected and search disabled: if it is still wrong, the failure is comprehension; otherwise the original search trace decides between search bypass (no search invoked), retrieval failure (gold page not retrieved), and grounding failure (gold page retrieved but the answer was still wrong).
What is the paper's key finding?
Across ten frontier search agents, none matched a small human baseline: the best model, Gemini 3.1 Pro, scored 78.28% versus 92.67% for five human participants, and 72.1% of all model failures were retrieval-bound — 34.5% from not searching at all and 37.6% from failing to retrieve the authoritative page.
Is the CIVI dataset and code publicly available?
Yes. The paper states the code is on GitHub under the MIT license (github.com/dingyingliu/CIVI) and the dataset on Hugging Face under CC BY 4.0 (huggingface.co/datasets/dingyingliu/CIVI).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-6 are extracted as interleaved text blobs (axis tick values, dot lists, JSON snippets), so per-model values that appear only inside figures (e.g., per-model grounding-failure bars in Figure 4, meanAHR points in Figure 2) could not be read beyond what the prose restates; the paper text itself is otherwise complete through Appendix E, Figure 6 and the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.