Challenges of Auditing: Variability in Outputs of Large Language Models for Health
Yuan Pu, Yewon Chang, Furong Jia, Xunjian Yin, Jessica Ma, Ayman Ali, Monica Agrawal
cs.CL
Quick verdict
Finds systematic ChatGPT-vs-API output differences for health queries; sound paired design, but uncorrected p-values and LLM-judged features temper claims.
Summary
The study tests whether the same large language model gives materially different answers to health questions depending on how it is accessed. The authors submitted 50 patient-authored questions from the HealthCareMagic-100k corpus three times each to GPT-5.3 and GPT-5.4 through the ChatGPT web interface and the OpenAI API, adding a ChatGPT Health condition for 42 questions (600 plus 126 responses total), under controlled conditions: fresh sessions, no memory, no custom instructions. Responses were scored on form (length, Flesch-Kincaid readability, formatting, emoji), engagement behavior (disclaimers, follow-up prompts), cited web sources, and clinical content, using paired within-question Wilcoxon tests. Differences were systematic: in GPT-5.3, interface answers were longer, easier to read, more heavily formatted (36.7% vs 0.7% emoji prevalence) and far more likely to invite further interaction (98.0% vs 70.7% continuation moves); in GPT-5.4 several of these gaps reversed direction. Cross-mode overlap of cited webpages fell to 10.9% versus roughly 17-18% within a single access mode, while clinical concepts overlapped less across modes (e.g., 0.615 vs 0.653) but remained high. Limitations include a single vendor and two models, only 50 questions, unadjusted p-values across many tests, LLM-based feature annotation without reported human validation, and a Health condition collected four months after the primary data.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
6 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Even with fresh sessions, no memory, and matched settings, the same GPT model produces systematically different health answers through the chatbot interface than through the API.
- Access-mode effects on form and engagement are not stable across model versions: differences seen in GPT-5.3 (longer, more formatted, more interactive interface answers) largely reverse in GPT-5.4.
- Cross-mode divergence in cited web sources (10.9% webpage overlap) exceeds run-to-run variability within a mode (~17-18%), while clinical content diverges significantly but only modestly.
- Because most evaluations run through APIs while patients use chatbot interfaces, audit conclusions risk characterizing a different system than the one consumers experience, and regulators should require providers to expose reproducible consumer-pathway access.
Strengths
Paired design with a run-to-run variability baseline
Section 5.5 makes the question the unit of analysis and Table A3 compares cross-mode Jaccard against within-mode repeated-run Jaccard (e.g., webpages 0.109 cross vs ~0.175 within, p < 0.001), cleanly separating systematic access-mode effects from ordinary sampling stochasticity.
Unusually careful access documentation
Section 5.2 reports exact model snapshots (gpt-5.3-chat-latest, gpt-5.4-2026-03-05), tool configurations, Temporary Chat mode, empty system prompt, and situates the April 1-22, 2026 collection window between the March 16 GPT-5.3 update and the April 23 GPT-5.5 release.
Multi-dimensional, clinically informed feature framework
Transparent handling of the Health condition
Honest scoping of the clinical-content result
Concrete, evidence-grounded policy recommendation
Weaknesses
No multiple-comparison correction
LLM-as-judge annotation without reported validation
Temporal confound in the ChatGPT Health condition
Narrow disclaimer operationalization feeding a safety narrative
Bounded external validity and unverifiable artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks a deceptively simple question: does the same GPT model answer the same health question differently depending on whether it is reached through the ChatGPT web interface, ChatGPT Health, or the OpenAI API? The authors claim (1) that differences are systematic across 50 consumer health questions even under tightly controlled conditions (fresh sessions, Temporary Chat, no memory or custom instructions, empty API system prompt); (2) that these differences are not stable — form and engagement effects reverse direction between GPT-5.3 and GPT-5.4; and (3) that because evaluations overwhelmingly run through APIs while patients meet chatbots, audit validity is compromised and providers should be required to expose reproducible consumer-pathway access. The claims are appropriately scoped to two OpenAI models and framed as an access-mode problem, not a model-quality judgment.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 6 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do ChatGPT and the OpenAI API give different answers to the same health question?
According to this study, yes, systematically. Across 50 consumer health questions asked three times in each mode, ChatGPT interface answers were longer, easier to read, more heavily formatted (emoji in 36.7% of runs vs 0.7% via API) and more likely to invite follow-up (98.0% vs 70.7%) under GPT-5.3; several of these gaps reversed direction under GPT-5.4. Even cited web sources overlapped only about 10.9% across access modes, versus 17-18% for repeated runs within one mode.
What is ChatGPT Health and how did the study test it?
ChatGPT Health is a health-specific experience inside the ChatGPT interface that activates automatically when a personal health question is detected (broad US rollout from July 2026), with optional connection to medical records. The authors manually selected Health for 50 questions; it activated for 42 (questions about someone else's health did not trigger it), and those 42 were compared against matched API and standard ChatGPT responses for the same questions.
What does this mean for AI medical evaluations and benchmarks?
Most evaluations query models through APIs, but consumers meet vendor-managed interfaces whose hidden instructions, tools, and safeguards researchers cannot replicate. The paper argues that API-based evaluations may therefore measure a different system than patients actually experience, and calls on regulators to require providers to offer auditable, reproducible access to the consumer pathway.
Is the paper's data or code available?
The authors state that the sampled questions, collected responses, code, and prompts will be released at github.com/yuan-pu/health-llm-variability, subject to source-data terms. As of the reviewed version these artifacts were promised rather than published, so the reported tables could not yet be independently verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are available only as scattered extracted text; visual annotations could not be fully verified.
- GitHub artifacts are promised but not published, so data/code claims rest on the Declarations section alone.
- Only p-values are reported, no test statistics or confidence intervals, limiting assessment of effect-size precision.
- Reference section extracted with some formatting noise, but citation content remained legible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.