Characterizing Web Search by Conversational LLM Agents: From Search Decisions and Strategies to Results and Responses
Mahsa Amani, Seungeon Lee, Abhisek Dash, Asmaa El Fraihi, Yunah Jang, Elisabeth Kirsten, Qinyuan Wu, Krishna P. Gummadi, et al.
cs.AI · cs.IR
Quick verdict
First cross-platform measurement of agentic Web search on 4 platforms; rich lifecycle findings, but judge-model heterogeneity and confounds temper conclusions.
Summary
This paper presents the first end-to-end characterization of Web search by conversational LLM agents across four major platforms (ChatGPT, Claude, Grok, DeepSeek). It combines an invivo analysis of 171,264 conversations donated by 613 users under GDPR data-access rights with an invitro setup that replays 1,000 sanitized prompts through the platforms' APIs. On search decisions, invocation varies sharply (in replay, GPT-5.3-chat searched 140 of 1,000 prompts versus 825 for Claude Sonnet 4.6), shaped by both harness instructions and model self-decisions; Web search generally improves factuality, completeness, and relevance for invoked prompts - with mixed effects for Claude - while prompts the models chose not to search score highest. On querying, agents compress verbose prompts into short queries, iteratively refine them with rising temporal, geographic, and entity specificity, and platform search engines return only about 5-15 URLs per query, concentrated on a few preferred domains (ChatGPT's top-10 domains cover 21.3% of invivo results; reddit and youtube are absent from Claude). On responses, citation rates range from 1.8% (Grok invivo) to 34.1% (DeepSeek invivo), hallucinated citations are only 1-2%, yet 14-53% of claims rest solely on uncited search results, and ungrounded claims score lowest on factuality (2.84/5 for ChatGPT). Key limitations: per-platform LLM judges, small human-validation samples (n=30), unreleasable donated data, and missing DeepSeek queries.
Review complete
Assessment at a glance
5
risk flags identified
● high 2
● medium 2
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Web-search invocation decisions differ sharply across platforms - on the same 1,000 replayed prompts GPT-5.3-chat searched 14% versus Claude Sonnet 4.6's 83% - and are shaped by both harness instructions and model self-decisions.
- More frequent Web-search invocation does not guarantee better responses: search improves factuality, completeness, and relevance for invoked prompts on most models but shows mixed effects for Claude, and prompts models decline to search score highest overall.
- Platform-specific search engines return few URLs per query (about 5-15) with strong, divergent domain preferences - reddit.com and youtube.com dominate ChatGPT and Grok results but are completely absent from Claude's.
- Although responses are largely grounded in cited sources, 14-53% of claims on ChatGPT, Claude, and Grok are supported only by uncited search results, and ungrounded claims have the lowest factuality (e.g., 2.84/5 on ChatGPT).
Strengths
Unique cross-platform invivo corpus
171,264 conversations from 613 users across ChatGPT, Claude, Grok, and DeepSeek with full search traces (queries, result URLs, citations), collected via GDPR Article 15(3) donations (Table 1); no prior work observes agentic search on deployed commercial systems at this scale and breadth.
Controlled replay plus harness-swap design
The invitro replay of the same 1,000 prompts (Table 3) enables like-for-like model comparison, and the GPT-only harness-instruction swap (Table 4) attributes invocation differences to both instructions (o4-mini: 634→478 searches when given GPT-5.3-chat's rules) and model-intrinsic propensity (o4-mini still searches 3-4x more than GPT-5.3-chat under any harness).
Lifecycle framing yields actionable per-stage findings
Unusually transparent judge methodology
Weaknesses
Per-platform, self-family judges undermine cross-platform comparability
Near-zero judge reliability where conclusions are strongest
Confounded invivo comparisons and silent platform missingness
Selection-effect phrasing invites causal overreading
Irreproducible evidence base
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper claims the first cross-platform, end-to-end measurement of agentic Web search across ChatGPT, Claude, Grok, and DeepSeek, structured around a three-stage lifecycle: the decision to invoke search (§3), query formulation and returned results (§4), and citation/grounding of responses (§5). Evidence combines an invivo corpus of 171,264 conversations from 613 users collected via GDPR Article 15(3) data donations (Table 1: ChatGPT 143,730 conversations/310 users; the other platforms ~9,000 each/~100 users) with an invitro replay of 1,000 PII-sanitized first-turn prompts through the platforms' APIs (Table 2). Headline claims: invocation decisions vary widely (14% vs 83% on identical prompts, Table 3); both harness instructions and model self-decisions drive them (Table 4); more search does not guarantee better answers; platform search engines concentrate results in a few preferred domains; and 14-53% of claims rest on uncited search results.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which AI chatbot searches the web the most?
In this paper's controlled replay of the same 1,000 prompts through each platform's API, Claude (Sonnet 4.6) invoked Web search for 825 prompts (83%), Grok-4.3 for 766, DeepSeek-v4-flash for 584, and GPT-5.3-chat for only 140 (14%). In real-user traces, the share of responses involving Web search has been rising steadily on all four platforms, with strong topic differences (travel prompts trigger search far more often than health prompts).
Do ChatGPT and other AI assistants cite their sources accurately?
Largely yes at the URL level: only about 0.3-2% of cited URLs were classified as hallucinated (no trace in web archives of ever existing). But grounding is weaker at the claim level: across ChatGPT, Claude, and Grok, 14-53% of response claims were supported only by search results that were retrieved but never cited, and claims left ungrounded by any retrieved source received the lowest factuality scores (e.g., 2.84/5 on ChatGPT).
Does web search make LLM answers more accurate?
In the replayed-prompt experiments, responses where the model chose to search scored higher on factuality, completeness, and relevance than the same prompts answered without search for GPT-5.3-chat, Grok-4.3, and DeepSeek-v4-flash (paired bootstrap p<10⁻³ for most metrics), but Claude's results were mixed - searching sometimes hurt. Prompts the models judged not to need search scored best of all, so more search does not automatically mean better answers.
Is the paper's dataset publicly available?
No. The 171,264 donated conversations cannot be released for privacy reasons. The authors publish analysis code, all judge prompts, and mock data on GitHub (AgenticSearchLens), so the analysis pipeline is inspectable, but the headline numbers cannot be independently re-derived from the private donation corpus, and the replay results depend on now-frozen commercial API versions.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Section 4.1 (querying strategies) is truncated mid-passage in the extracted text; the fan-out vs. iterative strategy analysis lives mostly in appendix figures whose plotted content could not be read
- Figures appear only as axis-label/tick text dumps; figure-derived numbers were taken from captions and in-text statements, not the plots themselves
- Tables 10-11 (per-source factuality) are truncated (DeepSeek row cut off) and the reference list is incomplete
- Review is based on the provided arXiv v1 extraction; the released GitHub repository (AgenticSearchLens) and full appendices were not independently inspected
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.