You Can Tell Who's Asking: What the Web's Questions Are Made Of, and Where They Come From
Calvin Zhou, Vincent McCloskey, Krishna Srinivasan
cs.CL
Quick verdict
Careful 12-year study finds web questions shifting human→manufactured (genuine −42–79%); labels rest on heuristic host rules and small author-run validation.
Summary
The paper tests the assumption that questions scraped from the web reflect human intent, using 13.4B question occurrences extracted from 110 English FineWeb snapshots spanning 2013–2025. Each question-bearing sentence is labeled by a deterministic host/URL cascade into provenance classes (genuine-UGC, manufactured-template via cross-host duplication, commerce-FAQ, editorial, unknown). Three findings: (1) question form carries provenance signal — a logistic model separates genuine from manufactured questions at AUC 0.725, driven by length (odds ratio 2.09 per SD), but only AUC 0.554 against commerce-FAQ writing; (2) frequency is not demand — 86.8% of the top-1,000 most-frequent questions in 2020 were manufactured templates, versus 3–4% of distinct questions overall; (3) the genuine-UGC share of occurrences fell from 9.3% to 1.9% (−79%), shrinking to −42%/−56% on a fixed host panel and −45% after size adjustment, while mean length fell 13.41→11.85 tokens. Methods are unusually careful: pre-specified hypotheses, Mann-Kendall trends with Hamed-Rao correction, bootstrap CIs, and honest withdrawal of claims that fail controls. Main limitations: provenance labels are heuristic host rules validated on only 150 items annotated by the authors themselves (authenticity κ=0.23; unknown-class precision 0.43), and per-occurrence text is not released.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Question form carries a provenance signal: a logistic model separates genuine from templated/manufactured questions at AUC 0.725, driven by length (OR 2.09 per SD), but only AUC 0.554 against commerce-FAQ writing.
- Question frequency in web corpora measures publishing intensity, not demand: 71.7%/86.8%/93.2% of the top-1,000 most-frequent questions were manufactured-template in 2015/2020/2024, versus 3–4% of distinct questions corpus-wide.
- The genuine-UGC share of web question occurrences fell from 9.3% to 1.9% over 2013–2025 (−79% crawl-wide; −42%/−56% on a fixed host panel; −45% size-adjusted, the paper's best estimate).
- Question form degraded corpus-wide (mean length 13.41→11.85 tokens, scaffolded share 9.95%→6.12%), driven by within-class decline plus composition shift, while markup-mined WebFAQ matches the manufactured-question form profile.
Strengths
First diachronic, occurrence-level measurement of question provenance at extreme scale
13.4B occurrences across 110 FineWeb snapshots (CC-MAIN-2013-20 to 2025-26), versus Web Data Commons' markup-only, one-snapshot-per-year counts (§2); the schema-less linguistic majority of questions is reached for the first time.
Exemplary statistical hygiene and honest claim withdrawal
Pre-specified hypotheses (App. E, one failure reported: question type Cramér's V=0.03), Mann-Kendall with Hamed-Rao correction (n*≈9–13), frozen eras with ±1-year boundary checks, and explicit withdrawal of the duplication increase (p=0.81) and template-share rise (p=0.49) once size-adjusted (Limitations).
Two independent confound controls converge
External cross-check against an independent dataset
Substantial artifact release with dual-use candor
Weaknesses
Validation is small and author-annotated
Unknown class is large and poorly measured
Form-based detection is nearly blind to the fastest-growing manufactured class
Recall of the provenance classifier per era rests on 5–11 items
Frequency analysis uses only three snapshots
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a measurement study of the web's 'question layer': every question-bearing sentence (5–100 words, question-word onset or interrogative keyword) in 110 English FineWeb snapshots, 2013–2025 (CC-MAIN-2013-20 through CC-MAIN-2025-26), totaling 13.4B occurrences. Three headline claims: (1) provenance is partly readable from form — a logistic model separates genuine from manufactured questions at AUC 0.725 (§4, Table 1), but only 0.554 against commerce-FAQ; (2) frequency measures publishing, not demand — 86.8% of the top-1,000 questions in 2020 were manufactured-template versus 3–4% of distinct questions (§5, Figure 2); (3) the genuine-UGC share fell from 9.3% to 1.9% (−79%) over twelve years, with controlled estimates at −42%/−56% on a fixed host panel and −45% size-adjusted (§6). The paper positions itself against markup-mined corpora (CCQA, WebFAQ), autocomplete harvesting (GooAQ) and synthetic sets (PAQ), claiming to be the first diachronic, occurrence-level provenance analysis.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper claim about questions scraped from the web?
It argues that frequently occurring web questions are mostly manufactured, not asked by humans: in 2020, 86.8% of the top-1,000 most-frequent questions were templated boilerplate. Counts of question occurrences therefore measure how often a string was published, not how often anyone actually asked it.
Is the decline in genuine web questions caused by ChatGPT or AI?
No, mostly not. The decline largely predates ChatGPT: the cross-host duplication peak came about twenty months before its release, and the genuine share was already falling steeply during the 2016–2019 SEO era. The authors explicitly avoid claiming any specific question was AI-written.
How much did the share of genuine human questions on the web fall?
The crawl-wide share fell 79% between 2013 and 2025 (9.3% to 1.9% of occurrences). After controlling for which hosts the crawl covers, the estimate is −42% to −56%, and the authors' best estimate after adjusting for crawl size is about −45%.
Is the code or data from this paper available?
Partially. The GitHub repo bodhiumlabs/tell-whos-asking releases the classifiers, the LLM-judge script, the validation sample and guidelines, per-figure data, and a deterministic rule to rebuild the 2.2M-item sample from public FineWeb. Individual question texts and host/URL data are withheld for privacy reasons.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The extracted text is truncated: parts of §4, §7 and appendices E–G are missing, so pre-registration details, the commerce/editorial host lists, and parts of the context analysis are judged from quoted fragments only.
- Figures 1–4 could not be inspected; all curve values cited come from captions and inline text.
- Some references (2026 venues, gemini-3.5-flash as rater model) are as-extracted and were not independently verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.