Evaluating Scaffolding-Oriented Multi-Agent Large Language Model System for Clinical Interview Training
Luming Yang, Haoxian Liu, Siqing Li, Rong Jia, Yue Xiao, Guanhua Chen, Li Lu
cs.MA · cs.AI · cs.HC
Quick verdict
RCT (N=95): multi-agent AI-SP scaffolding raises OSCE exam scores vs non-LLM control, but LLM-scored endpoint and missing single-agent baseline cap certainty.
Summary
The paper presents MeduAI-SP, a multi-agent LLM platform for clinical interview training comprising a patient agent, a Socratic tutor, and a turn-level evaluator, and reports a two-arm randomized controlled trial with 100 third-year medical students at Guangzhou Medical University (95 analyzed: 47 multi-agent, 48 control). The control was a structured non-LLM progressive-disclosure activity built from the same three acute-abdominal cases (appendicitis and pancreatitis for learning; perforated peptic ulcer for examination). After two learning sessions, all students took an examination with a patient-only AI-SP, scored with an OSCE-aligned rubric. The multi-agent group scored higher on the final examination (71.8% vs 55.6% mean; Mann–Whitney P=5.51×10⁻⁵, Hedges' g=0.81; regression β=16.4 points, 95% CI 7.7–25.1), driven by communication (3.53 vs 2.64) and an empathy checklist item (+31 percentage points, Holm-corrected P=8.30×10⁻⁴); diagnostic accuracy did not differ (86% vs 84%). The authors also release a corpus of 207 expert-annotated sessions (4,815 messages) with good OSCE reliability (ICC 0.80–0.85). Main limitations: the comparator is non-dialogue reading material, so dialogue practice and scaffolding are confounded; no single-agent LLM arm tests the multi-agent architecture itself; and the primary endpoint appears to be scored by the platform's LLM final evaluator rather than blinded human raters.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Multi-agent AI-SP training with Socratic scaffolding improves OSCE-aligned final examination performance versus a structured non-LLM progressive-disclosure control (71.8% vs 55.6% mean; β=16.4 points, 95% CI 7.7–25.1).
- The largest and most consistent gains are in observable consultation behaviors — communication (3.53 vs 2.64) and empathic expression (+31 pp checklist item, Holm-significant) — not in binary diagnostic accuracy (86% vs 84%, P=1.000).
- The absence of a diagnostic-accuracy difference reflects appropriate experimental control, since both arms accessed identical case content and the tutor never disclosed diagnoses.
- The released 207-session, 4,815-message corpus has sufficient expert-annotation reliability (dialogue-act κ=0.84; OSCE ICC(2,k)=0.80–0.85) to support future AI-SP and automated-assessment research.
- About 24.1% of student utterances needed scaffolding, rising from ~16.1% early to ~34.8% late in consultations, which can inform tutor-trigger design.
Strengths
Genuine RCT with a clean transfer test
100 students randomized 1:1 (computer-generated sequence), 95 analyzed; both arms receive identical YAML-encoded case content and identical time allocation, and the endpoint is measured in a common patient-only examination environment (§8.1, §8.6), removing knowledge-access confounds for the diagnostic endpoint.
Honest null result and disciplined causal framing
Diagnostic accuracy is foregrounded as null (86% CT vs 84% MA, P=1.000, §2.4), and the Discussion explicitly disclaims empathy-as-trait, superiority to human SP feedback, and replacement of faculty judgment — unusually careful interpretive boundaries.
Above-norm statistical transparency
Annotated corpus with per-dimension reliability, including ugly numbers
Reproducibility infrastructure on the system side
Weaknesses
Primary outcome scoring provenance is ambiguous and possibly LLM-only
Active ingredients confounded; the architecture claim is untested
Scaffolding targets and scored outcome items coincide
Single exam case near ceiling; negative within-arm trajectory under-analyzed
No preregistration, power calculation, or ITT analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper introduces MeduAI-SP, a web-based multi-agent LLM platform for clinical interview training: a patient agent generates inquiry-dependent case responses, a tutor agent issues Socratic prompts (focused history taking, diagnostic reasoning, summarizing, empathy) without disclosing diagnoses, and a turn-level evaluator monitors progress; a final evaluator scores encounters in the background (§1, §8.4). The empirical core is a two-arm randomized controlled trial at Guangzhou Medical University — 100 third-year students randomized 1:1, 95 analyzed (MA n=47, CT n=48; Table S1) — comparing multi-agent scaffolding against a structured non-LLM progressive-disclosure control built from the same YAML-encoded cases (appendicitis and pancreatitis for learning, perforated peptic ulcer for the exam; §8.2). Two learning sessions are followed by a common patient-only examination scored on an OSCE-aligned rubric. A secondary deliverable is an expert-annotated corpus (207 sessions, 4,815 messages; §2.8). Headline claims: MA lifts final exam scores by 16.4 points (95% CI 7.7–25.1; Mann–Whitney P=5.51×10⁻⁵, g≈0.81; §2.2), with the largest gains in communication (3.53 vs 2.64, §2.3) and a +31 pp empathy checklist item (Holm P=8.30×10⁻⁴, §2.4), while diagnostic accuracy is unchanged (86% vs 84%, P=1.000).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MeduAI-SP?
A web-based training platform in which medical students practice clinical interviews with a multi-agent LLM system: one agent role-plays the patient, a tutor agent gives Socratic hints (without revealing the diagnosis), and an evaluator agent monitors each conversation turn; a separate final evaluator scores the encounter afterwards.
Did AI-based training improve students' diagnostic accuracy?
No. In the randomized trial, correct primary diagnosis on the exam case was 86% in the control group and 84% in the multi-agent group (P=1.000). The measured benefits were in process quality: communication scores (3.53 vs 2.64 on a 1–5 scale) and an empathy checklist item (+31 percentage points).
How large was the effect of the multi-agent training?
About 16.4 percentage points on the final OSCE-aligned exam score (95% CI 7.7–25.1; Hedges' g≈0.8) versus a non-LLM reading-based control. Note the comparator had no dialogue practice, and the exam appears to have been scored by the platform's own LLM evaluator, so the attribution and magnitude carry caveats.
Is the MeduAI-SP dataset available?
The paper states the annotated dataset (207 sessions, 4,815 messages, three-expert annotations of dialogue acts, scaffolding needs, patient fidelity, and OSCE scores) will be released publicly with a DOI upon manuscript acceptance; during review it is available from the corresponding author. Code is at github.com/skylynf/agent-medu.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures were available only through their captions in the extraction; all quantitative claims cited in this review were verified against the body text and Tables 1 and S1–S7, none rely solely on figure content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.