Sources of Truth: A Multi-Platform, Multilingual Audit of Citations in AI Mental Health Information Queries
Phuong Anh Nguyen, Jill Noorily, Matthew Flathers, Haruka Notsu, Laura Ospina-Pinillos, Tommy Nguyen, Samantha Clark, Aoife Keane, et al.
cs.CY · cs.CL · cs.IR
Quick verdict
Transparent, well-validated audit: AI mental-health answers concentrate 43.6% of citations in 10 domains; multilingual routing lags; largely descriptive.
Summary
The authors audit which sources three consumer AI search products—ChatGPT (free tier), Perplexity, and Google AI Overview—cite when answering mental health questions. Seven trained annotators submitted 20 English questions under two prompt conditions (with and without an appended 'List your Sources.'), five times each, and translated three questions into Spanish, Japanese, Ukrainian, Hindi, Nepali, and Twi, collecting 1,140 responses, 15,942 citations, and 1,713 unique domains between June 25 and July 20, 2026. Every citation was classified into a nine-category organizational typology by a deterministic rule-based classifier validated against human coding (κ = 0.86–0.89). English citations concentrated heavily: the top ten domains accounted for 43.6%, with government, commercial health, and academic sources roughly tied at ~22% each. Platforms showed similar median citation counts (10–12) but sharp differences in consistency (ChatGPT mean 32.3, SD 60.5) and source mix (ChatGPT: Wikipedia 6.1%; Google AI Overview: social/video 8.2%). Requesting sources barely changed volume or composition. Non-English queries returned fewer citations and lower language-appropriate routing (Twi: 2.5% ccTLD, 37.6% counting native-language pages). An appendix comparing vendor APIs finds the ChatGPT product layer supplies most of its source diversity (gpt-5.4-mini: 31 domains, 50.3% search invocation). Limitations include a single-point snapshot of undisclosed free-tier models, US-based annotator IPs, manual citation transcription without recall validation, and purely descriptive statistics.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Citations in AI mental health answers are heavily concentrated: the ten most-cited domains account for 43.6% of English citations, with government, commercial health, and academic sources each at roughly 22%.
- The three platforms differ little in median citation volume (10–12 per response) but sharply in consistency and in which source types they favor (ChatGPT: government 26.4% and Wikipedia 6.1%; Google AI Overview: commercial 25.7% and social/video 8.2%; Perplexity: academic 24.3%).
- Explicitly appending 'List your Sources.' changes little at the product level: mean citations rise from 16.6 to 19.9 and source-type composition shifts by at most 2.1 percentage points.
- Non-English queries surface fewer citations and much lower language-appropriate routing, with ccTLD shares of 3.3% for low-resource languages (Twi rising to 37.6% once native-language pages on international domains count).
- The product wrapper, not the underlying model, supplies most of ChatGPT's source diversity: the paired API model cited only 31 domains with a 70.6% government share versus 641 English domains at 26.4% for the product.
Strengths
Large-scale, fully released audit corpus
15,942 citations across 1,140 responses and 1,713 domains; the nine-category typology, deterministic classifier, annotated corpus (Hugging Face DOI 10.57967/hf/10067) and AGPL collection/analysis code (github.com/mindbench-ai/search-source-audit) are all public, making every §3 estimate re-derivable from released data.
Credibly validated classification instrument
§2.7: the 100 most-cited domains were fully adjudicated by two authors, plus a 200-domain random sample spread across categories, yielding Cohen's κ of 0.88 between annotators and 0.89/0.86 between each annotator and the classifier.
Systematic robustness analysis over every published estimate
Two-surface design separates product wrapper from model
Novel security linkage with appropriately hedged claims
Weaknesses
Statistical reporting is descriptive-only despite 'significant' language
Annotator retry rule conflicts with reported no-citation rates
Citation extraction completeness never validated
Unexplained count discrepancies
Thin and confounded multilingual instrument
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint audits what sources three consumer AI search products — ChatGPT (free tier), Perplexity, and Google AI Overview — actually cite when answering mental health questions. Seven annotators ran a factorial design (3 platforms × 20 English questions × 2 prompt conditions × 5 repetitions), plus a 3-question subset in six further languages (Spanish, Japanese, Ukrainian, Hindi, Nepali, Twi), collecting 1,140 responses, 15,942 citations, and 1,713 unique domains between June 25 and July 20, 2026 (§2.1). The central claims: (1) citations concentrate on a small institutional core — the top ten domains carry 43.6% of English citations, with government, commercial health, and academic sources nearly tied at ~22% each (§3.3); (2) platforms match on median citation volume (10–12) but diverge sharply in consistency (ChatGPT mean 32.29, SD 60.50 vs Perplexity 9.7, SD 1.69) and source mix (§3.2–3.3); (3) appending "List your Sources." moves mean volume from 16.6 to 19.9 and composition by at most 2.1 pp (§3.4); (4) non-English queries receive fewer citations and far lower language-appropriate routing — a 3.3% ccTLD share for low-resource languages, rising to 37.6% for Twi once native-language pages count (§3.5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What sources do ChatGPT, Perplexity, and Google AI Overview cite for mental health questions?
This audit of 15,942 citations found heavy concentration: the ten most-cited domains accounted for 43.6% of English citations. Government sites, commercial health publishers (e.g., WebMD, Healthline), and academic sources were nearly tied at roughly 22% each, with the exact mix depending on which product you use.
Does asking an AI chatbot to 'list your sources' improve the citations you get?
Barely. Appending 'List your Sources.' raised mean citations per response only from 16.6 to 19.9 (driven almost entirely by ChatGPT), and shifted the source-type mix by at most 2.1 percentage points toward more institutional sources.
Are AI search answers worse in languages other than English?
Yes, substantially. Non-English queries returned fewer citations on average (9.2 vs 12.3) and much lower language-appropriate routing: low-resource languages got only 3.3% of citations on country-code domains, though counting native-language pages on international domains raised Twi's share from 2.5% to 37.6%.
Is the data and code from this study publicly available?
Yes. The annotated corpus of classified citations is on Hugging Face (DOI 10.57967/hf/10067) and the nine-category typology, deterministic classifier, and analysis code are on GitHub under an AGPL license.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–5 are referenced via captions and in-text restatements only; their rendered images were not part of the extracted text, but every quantitative claim cited in this review was restated in body text, captions, or Appendix S4 tables.
- The Devanagari question texts in Appendix S2 (Hindi, Nepali) show PDF-extraction glyph corruption, so the translation wording itself could not be verified; the surrounding design details were readable.
- The extraction appears complete — the text runs through the final page of Appendix S4 (Table D2) with references and all four appendices present; no truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.