WildSEEK: Evaluating Language Models for Information-Seeking
Tanise Ceron, Joachim Baumann, Elisa Bassignana, Berat Cabuk, Dirk Hovy, Debora Nozza
cs.CL · cs.CY
Quick verdict
Claims analytical queries elicit more LLM failures; WildSEEK dataset/framework is solid, but weak judge agreement on US bias (κ=.40) tempers findings.
Summary
The paper introduces WildSEEK, a manually annotated dataset of 3,077 information-seeking queries drawn from four corpora of real human-LLM conversations (WildChat, ShareGPT, LMSYS-Chat-1M, and SES). Queries are labeled for user intent, six risk-sensitive domains, and open-endedness (factoid vs. analytical). ModernBERT classifiers trained on these labels (macro-F1 0.81–0.83) annotate over 1.8M queries, showing information-seeking accounts for ~40% of user turns on average, 37% of those queries touch risk-sensitive domains, and 60% are analytical. The authors then evaluate GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Flash-Lite with and without web search on reliability (Loki-based factuality, MBFC source credibility), fairness (source diversity, US bias), and safety (sycophancy, overreliance, vulnerable populations, anthropomorphism, dual use), using an LLM-as-a-judge validated on 222 human-annotated pairs. Analytical queries elicit higher failure rates in 4 of 6 criteria (+4.03pp overall; overreliance peaks at 14.31%), and search does not consistently improve results—only Gemini improves significantly across safety criteria. Limitations include English-only data, WEIRD-skewed sources, proprietary models only, low inter-annotator agreement on the US-bias criterion (κ=0.40), and moderate judge precision for sycophancy (0.62).
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Information-seeking queries average ~40% of user-LLM turns across four corpora; 37% of them involve risk-sensitive domains and 60% are analytical rather than factoid.
- LLM responses fail most often on overreliance (14.31% average failure rate), US-centric framing, sycophancy, and vulnerable-population handling (~9% each).
- Analytical queries elicit significantly higher failure rates than factoid queries in 4 of 6 criteria (+4.03pp overall; overreliance +13.96pp).
- Enabling web search does not consistently improve reliability, fairness, or safety; only Gemini-3.1 improves significantly across safety criteria, and search raises aggregate failure from 8.84% to 10.75%.
Strengths
Real in-the-wild data with documented annotation quality
Queries come from four real interaction corpora (WildChat, ShareGPT, LMSYS-Chat-1M, SES); inter-annotator agreement is reported per stage: intent κ=0.80 (Table 7), risk domains Fleiss κ 0.72→0.82 with per-class breakdowns (Table 9), and full guidelines in Appendices A–C.
Classifier scaling with honest baselines
ModernBERT-large/256 reaches macro-F1 0.82 (intent; 0.90 for info-seeking), 0.81 CV / 0.796 held-out (domains), 0.83 (open-endedness), all far above majority baselines of 0.22/0.22/0.38 (Tables 12–14); throughput (1k predictions in 11s) is also reported.
Multi-criteria framework with auditable rubrics
Careful response-level statistics
Unusual transparency
Weaknesses
US-bias headline rests on the weakest-validated criterion
Judge-family overlap and sycophancy precision
Thin annotation depth behind open-endedness labels
No uncertainty on the 1.8M-scale prevalence estimates
Confounded model comparison and one-sided testing
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WildSEEK (arXiv 2608.30683) addresses three research questions: how often in-the-wild LLM queries are information-seeking, risk-sensitive, and analytical (RQ1); what risks model responses carry (RQ2); and whether integrated web search helps (RQ3). Central claims: information-seeking averages ~40% of user turns; 37% of those queries touch six risk-sensitive domains and 60% are analytical; failures concentrate in overreliance, US-centric bias, sycophancy, and vulnerable-population handling; analytical queries fail more; and search 'does not consistently mitigate safety failures' (§6.1). This is primarily a resource/benchmark paper with an attached empirical audit of three proprietary models.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the WildSEEK dataset?
A publicly released (ODC-By licensed) set of 3,077 real user queries to chatbots, drawn from WildChat, ShareGPT, LMSYS-Chat-1M, and SES. Each query is manually labeled with user intent, one of six risk-sensitive domains (politics, finance, security, health, legal, moral values/religion), and whether it is factoid or analytical. Fine-tuned ModernBERT classifiers for all three dimensions are also released.
Does turning on web search make LLM answers more reliable or safe?
In this study, not consistently. With search forced on, the aggregate failure rate across safety and fairness criteria rose from 8.84% to 10.75%, mainly because US-centric bias increased (9.13% to 13.13%), although sycophancy roughly halved (19.11% to 9.01%). Only Gemini-3.1 improved significantly across all safety criteria with search, and only Gemini showed a significant factuality gain (0.867 to 0.916).
What did the study find about analytical versus factual questions?
Analytical questions (advice, comparisons, predictions) produced higher failure rates than factoid questions in 4 of 6 evaluation criteria, by 4.03 percentage points on average. The largest gap was overreliance: 21.29% failure for analytical queries versus 7.33% for factoid ones, a pattern that held for all three models with and without web search.
Which models were evaluated and how?
GPT-5.4, Claude Sonnet 4.6, and Gemini 3.1 Flash-Lite (preview), each with and without integrated web search, on ~3k real information-seeking queries. Responses were scored with an automated factuality tool (Loki), Media Bias Fact Check source ratings, domain-diversity statistics, and a GPT-5.4-mini LLM judge for six binary safety/fairness criteria; the open-weight Llama-3.3-70B appears in an appendix comparison without search.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Values plotted inside Figures 2–4 and 9–11 are not recoverable from the text extraction; figure-based statements in this review rely on the authors' in-text descriptions of those figures.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.