Evaluating Deep-Search Agents under Hierarchical Web Evidence Poisoning
Zhongan Bi, Qiwen Wang, Jianrong Jiang, Jigang Ding, Wenwen Xiong, Changhua Meng, Xuanang Gao, Kepeng Lin, et al.
cs.CR · cs.AI · cs.IR
Quick verdict
HAE-GEO benchmark tracks agents from poison exposure to recovery; controlled design strong, but LLM-judge scores and tiny recovery subsets cap certainty.
Summary
The paper introduces HAE-GEO, a benchmark for evaluating deep-search LLM agents against Generative Engine Optimization (GEO) poisoning, where manipulated web pages promote fabricated brands. Unlike prior work that scores only retrieval or endorsement outcomes, it decomposes each trajectory into five observable states — exposure, adoption, verification, recovery, endorsement — via a deterministic rule layer over a Search-Scrape interface, with three escalating attack levels (L1 direct assertion, L2 contextual camouflage, L3 apparent corroboration) built on a controlled corpus of 72,039 clean pages plus 770 poisoned pages per level across 8 categories and 154 brands. Ten proprietary and open-source agents were run under BASE and DEF (defense) prompts on a fixed 120-query Chinese set, scored by behavioral metrics (exposure-conditioned fooled rate, strict verification yield, verification-to-recovery) and a six-dimension rubric judged by two isolated GPT-5.5 judges. Main findings: macro-averaged risk handling and poison recognition drop sharply from L1 to L3 (e.g., BASE M1 0.83→0.30); agentic search cuts fooled rate (up to 42.7 pp vs static retrieval) without improving recognition or utility; defense prompting lifts verification yield from ~2% to ~17%, yet verification-to-recovery is estimated on tiny subsets (defended n=11/9/3 at L1-L3). Limitations include single-judge scoring without reported human agreement, an acknowledged evaluator version discrepancy (v5 vs v6), incomplete corpus release, and generator-evaluator overlap with the Claude family.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- HAE-GEO is the first benchmark to jointly instrument exposure, adoption, verification, endorsement, and recovery under a controlled, escalating GEO-poisoning protocol (Table 1).
- Macro-averaged risk handling and poison recognition decline from L1 to L3 under both prompts (BASE M1 0.83→0.30, M2 1.02→0.33), the 'corroboration trap'.
- Agentic Search-Scrape reduces final fooled rates by 13.6-42.7 pp versus static full-context retrieval without significantly improving evidence recognition (M2) or utility (M6).
- Defense prompting raises strict verification yield from ~2% to ~17% at every level, but post-adoption verification stays rare and converts to recovery less often at L3 (81.8% at L1, n=11, vs 66.7% at L2/L3, n=9 and 3).
- Explicit reasoning improves every reported robustness measure in the two tested models (e.g., Qwen3.8-27B DEF fooled rate -19.8 pp).
Strengths
Trajectory-state decomposition is a real diagnostic advance
Deterministic E/A/V/R/Y rules (Sec 3.4, A.8) separate exposure, adoption, verification, and recovery from model-visible traces only; Table 1 shows recovery-after-prior-adoption is measured by no prior benchmark in its comparison set.
Controlled, budget-matched paired environments
All four conditions per task share the query and clean background with an identical poisoning budget and isolated retrieval namespaces/caches (Sec 3.2, A.7), so L1-L3 contrasts isolate presentation sophistication rather than attack volume.
Split-judge rubric prevents label contamination
Unusually candid self-auditing
Deployment-relevant cost and trade-off analysis
Weaknesses
Headline recovery claim rests on tiny conditional samples
Single-judge semantic scoring without human validation
Generator-evaluator family overlap untested
Chinese-only tasks with unexamined language confound
Partial release limits exact reproduction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
HAE-GEO (arXiv 2609.06027v2; Zhejiang University and Ant Group) is a benchmark for evaluating deep-search LLM agents against Generative Engine Optimization (GEO) poisoning, where attacker-authored pages promote fabricated brands inside an otherwise real web corpus. Its stated novelty (Sec 2, Table 1) is jointly instrumenting the exposure-to-recovery trajectory while holding the fabricated entity, false claim, and poisoning budget fixed across three escalating presentation levels: L1 direct assertion, L2 contextual camouflage, L3 apparent corroboration. The evaluation covers 10 agent families under BASE and DEF (defense) prompts on a fixed 120-query Chinese set (60 recommendation plus 60 verification; Sec 4.1, A.14). Three headline claims are made: a 'corroboration trap' (recognition declines as presentation sophistication grows), agentic search yielding resistance without recognition or utility gains, and verification rarely converting into recovery.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is HAE-GEO?
HAE-GEO is a benchmark that measures how well search-augmented LLM agents cope with GEO poisoning — attacker-written web pages that promote fabricated brands. It tracks each agent's trajectory through five states (exposure, adoption, verification, recovery, endorsement) across three escalating attack levels, using a controlled corpus of 72,039 clean pages plus 770 poisoned pages per level over 8 product categories and 154 brands.
Which models resisted GEO poisoning best?
In the paper's 120-query evaluation, Kimi-K3 with the defense prompt was the most robust (fooled rate 10.9-24.2% across levels), followed by Claude-Opus-4.8 with defense (17.2-33.3%). GLM-4.7-Flash was least robust, staying above 46% fooled rate even with the defense prompt. Defense prompting helped some models substantially and others barely at all.
Does multi-turn search make agents harder to fool?
Yes for final recommendations, no for diagnosis. In a paired ablation on two models (Kimi-K3 and Qwen3.8-27B), agentic Search-Scrape cut fooled rates by 13.6-42.7 percentage points versus static full-context retrieval, but poison-evidence recognition (M2) and answer utility (M6) did not significantly improve.
Is the HAE-GEO code and data public?
A code/benchmark repository is linked (github.com/ant-research/HAE-GEO), and prompts, judge templates, and the query-selection protocol are published verbatim in the appendices. However, only a subset of the clean web corpus is released due to crawling and redistribution restrictions, and the release status of the poisoned pages and retrieval index is not fully specified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Verbatim Chinese prompt excerpts in A.12-A.13 are garbled by PDF extraction (duplicated/interleaved characters); judgments about prompt content rely on the paper's own English translations.
- The metadata author list (8 names) differs from the byline in the extracted text, which lists 14 names.
- Per-level confidence intervals promised in Sec 5.1 do not appear anywhere in the provided text (Tables 5 and 9 show point estimates only); whether they are genuinely absent or reside in non-extracted material cannot be determined.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.