Clinician-Grounded Quality Assurance for AI-Assisted Psychiatric Intake
King Shi, Amanda Li, Jonathan Ivey, Synthia Qia Wang, Guan Gui, Hyunseo Kim, Peter Zandi, Jason Straub, et al.
cs.AI · cs.CL
Quick verdict
Pilot shows a GPT-4o intake interviewer beats 6 clinicians on field recovery (88% vs 39%) yet follows up safety disclosures poorly (33% vs 67%); promising QA platform, but tiny n and unvalidated simulator limit conclusions.
Summary
This pilot presents InterviewPlayground, a clinician-grounded quality-assurance platform for AI-assisted psychiatric intake. Memory-augmented simulated patients are grounded in expert-authored vignettes with fixed ground-truth fields and an embedded safety concern. Six clinicians and ten GPT-4o interviewer runs completed simulated intakes, the LLM matched to clinician message counts. Simulated patients stayed largely consistent (18/20 binary fields) and were rated realistic, though only two vignettes were analyzed. Results show a trade-off: the LLM recovered far more positive clinical fields (88% vs 39%), but made unsupported inferences more often (57% vs 28%) and characterized identified safety concerns less often (33% vs 67%), frequently changing subject after disclosures of suicidal ideation. Key limitations: a very small sample without statistical testing, an unpublished core simulator component (InterviewPlayground), vignettes that underspecify negative field values, and no validation against real patient encounters.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Memory-augmented simulated patients grounded in expert vignettes remain consistent enough across repeated interviews to support cross-interviewer comparison (18/20 binary evaluation fields adhered).
- A 25-minute workflow (median 25.3 min; recall form 2.8 min) is a practical, low-burden quality-assurance format that practicing clinicians completed.
- A GPT-4o intake interviewer recovered more positive ground-truth fields than clinicians (88.0% vs 38.9%) but made more unsupported inferences (56.8% vs 27.8%).
- The LLM identified safety concerns more often (9/10 runs vs 3/6 clinician sessions) yet characterized them with qualifying detail far less often (3/9 vs 2/3), typically changing subject after disclosures of suicidal ideation.
- A multi-dimensional, clinician-grounded evaluation platform can serve health systems as a repeatable pre-deployment quality-assurance process for AI intake tools.
Strengths
Evaluation goes beyond a single information-recovery score
R3 separates recall, precision/accuracy, unsupported inference (Table 2: 27.8% clinician vs 56.8% LLM) and safety identification vs characterization (2/3 vs 3/9), explicitly motivating why an aggregate score would hide the safety follow-up failure mode.
Pre-registration and participatory design
An OSF pre-registration (10.17605/OSF.IO/54XEJ) was filed before data collection by a ten-person team split evenly between CS/AI and behavioral health; design choices (e.g., dropping 3×8-min interviews after pilots with n=35 showed failure) were piloted or voted.
Simulator consistency is quantified, and failures are disclosed honestly
Budget matching reduces an obvious confound
Internal consistency of the recall instrument is checked
Weaknesses
Very small sample with no inferential statistics
Potential scoring asymmetry between arms
GPT-4o occupies all three seats: patient, interviewer, judge
Documented simulator deviations contaminate the ground truth
Core component unpublished and internal reporting inconsistent
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Clinician-Grounded Quality Assurance for AI-Assisted Psychiatric Intake (arXiv:2609.21149)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
How did an LLM interviewer compare with psychiatrists in this study?
In this Johns Hopkins pilot, a GPT-4o intake interviewer recovered more of the clinically relevant facts embedded in simulated patient cases than six clinicians did (88.0% vs 38.9% of positive ground-truth fields) and spotted embedded safety concerns more often (9 of 10 runs vs 3 of 6 clinician sessions). However, it made unsupported clinical inferences more often (56.8% vs 27.8%) and, when it did flag a safety concern, added qualifying clinical detail far less often (3 of 9 cases vs 2 of 3), often changing the subject right after a patient disclosed suicidal ideation.
What is InterviewPlayground?
InterviewPlayground is a simulated-patient platform for evaluating interviewers in open-ended conversation. Each synthetic patient is grounded in a bank of roughly 200 retrievable autobiographical memories generated offline from an expert-authored case vignette; for each interviewer question, the five most similar memories are retrieved and paraphrased into the patient's reply. It was built by some of the same Johns Hopkins authors and, as of this paper, is described as a manuscript in preparation.
Can this study prove AI psychiatric intake is safe to deploy?
No. The authors explicitly frame it as a pilot: six clinicians and ten LLM runs over only two case vignettes, with no statistical significance testing, an unvalidated-against-real-patients simulator, and LLM-based scoring judges. It demonstrates an evaluation process a health system could adopt for ongoing quality assurance, not deployment readiness of any specific tool.
Is the code or data available for this paper?
The paper does not mention public code, data, prompts, or vignette releases. It cites a pre-registration (OSF 10.17605/OSF.IO/54XEJ) for the study design. Reproducing the human arm would require recruiting clinicians; the LLM arm would additionally require rebuilding the vignettes and memory banks.
What does 'clinician-grounded' mean in this context?
The evaluation criteria and reference behavior come from practicing clinicians rather than automated metrics alone: clinicians authored the case vignettes with fixed ground-truth fields, completed reference intake interviews and recall forms, and rated simulator realism. Their participation was deliberately kept short (median 25.3 minutes total) so the process could be rerun routinely as a quality-assurance workflow.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are not analyzable from extracted text (only captions); interface and architecture details are taken from prose.
- The embedded author list (10 authors, including Jacob Taylor and Ananya Joshi) differs from the supplied metadata (8 authors); both are reported here as found.
- The text is otherwise complete through the references, so section-level judgments rest on the full narrative, but underlying tables/figures could not be inspected directly.
- Some counts (e.g., whether Table 2 clinician numbers derive from the recall form or transcript extraction) are inferred from prose wording and marked as such in the review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.