MedRoundsQA: A Persona and Difficulty Aware Evaluation for Multi-Turn Medical Consultations
Youssef Mohamed, Ahmed Heakl, Qinrong Cui, Junhong Liang, Rafiq Ali, Bdour Babillie, Nazira Dunbayeva, Lang Gao, et al.
cs.AI
Quick verdict
Turns 1,387 board cases into controlled multi-turn consultations; counterfactual personas isolate communication effects, though LLM-judge dependence remains.
Summary
MedRoundsQA converts 1,387 board-exam and NEJM diagnostic cases (292 MedQA, 646 NEJM, 449 MedMCQA; 17 specialties) into controlled multi-turn doctor-patient consultations. Each vignette is normalized into a 24-slot clinical record (GPT-4o extraction revised by three resident physicians), dialogues are generated by dual LLM agents with a fixed DeepSeek-Chat patient simulator, and patient personas vary education, affect, language proficiency, and occupation while clinical content is held fixed. Cases are stratified easy/medium/hard via model-uncertainty signals validated against blinded physicians (88.9% exact agreement, kappa=0.88). Across fifteen doctor LLMs under a 20-turn budget, multi-turn accuracy falls 13-39 points below single-turn accuracy on standardized records (e.g., GPT-4o 67.5 to 31.6); question relevance stays high (90-99%) but slot coverage is low (10-37%), and accuracy plateaus after 6-12 turns. Education personas shift accuracy by roughly 7.9 points from lowest to highest level; on 60 difficult cases, three residents (48.3%) nearly doubled the AI average (26.7%) using fewer turns. Error analysis of 943 incorrect consultations implicates hedging (40%) and insufficient elicitation (30%). Limitations: GPT-4o serves as judge while also being evaluated, a single fixed doctor prompt is used, persona and human-gap claims lack significance testing, and turn budgets differ across analyses (20 vs 12).
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Moving from single-turn diagnosis on standardized records to multi-turn consultation degrades LLM diagnostic accuracy by 13-39 points on identical cases (Table 2, 20-turn budget).
- More dialogue turns reliably improve question relevance (90-99%) but diagnostic accuracy exhibits diminishing returns and plateaus after roughly 6-12 turns (Sec. 4.4, Fig. 4).
- Holding clinical content fixed, patient education level shifts diagnostic accuracy by about 7.9 points between lowest and highest personas (Fig. 5), an equity risk invisible to single-turn benchmarks.
- On 60 difficult cases, resident physicians (48.33%) nearly double the AI average (26.67%) while using fewer turns (Table 3).
- Incorrect consultations cluster into four recurring failure modes, led by uncertainty-as-output (40%) and insufficient elicitation (30%), with near-identical distributions across the frontier models audited (Table 5).
Strengths
Counterfactual persona control is a genuine design advance
Each case is instantiated under multiple personas via a shared 24-slot record with clinical content fixed (Sec. 3.4); Table 1 shows prior interactive benchmarks (AgentClinic, MediQ, AI Hospital) vary patients across cases, confounding communication effects with case difficulty, and the same-case ST-MT comparison isolates the interactive component.
Above-average validation discipline for an LLM-judge pipeline
Locked judge protocol (Sec. 3.6): rubric frozen on a 100-dialogue dev split, GPT-4o judge reaches 0.88-0.90 accuracy and kappa of roughly 0.70-0.80 on 200 held-out dialogues vs 3-physician majority; extractor selected against a 100-case GP reference (Fig. 7, GPT-4o BERTScore 90.3 vs 84.8/85.4); difficulty labels validated on 90 blinded physician cases (88.9% exact, kappa=0.88, adjacent-band-only errors, Table 8).
Honest decomposition of the single-turn baseline
Simulator choice is shown not to drive results
Findings are behaviorally specific and actionable
Weaknesses
GPT-4o circularity across extractor, judge, and evaluated model
No inferential statistics for the headline gaps
Turn-budget inconsistency across analyses
Single fixed doctor prompt and chart-blurb format
Thin cells and missing reliability in subgroup analyses
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MedRoundsQA (arXiv 2609.12851) is a benchmark paper for interactive medical diagnosis. It converts 1,387 filtered diagnostic vignettes (292 MedQA, 646 NEJM, 449 MedMCQA; 17 specialties) into simulated multi-turn doctor-patient consultations grounded in a canonical 24-slot clinical record. Three claims anchor the paper: (1) multi-turn diagnosis drops 13-39 points against single-turn diagnosis on the same standardized records (Table 2, 20-turn budget); (2) question relevance rises with more turns while accuracy plateaus after roughly 6-12 turns (Sec. 4.4, Fig. 4); (3) patient persona, chiefly education, shifts accuracy by about 7.9 points between lowest and highest levels (Fig. 5), supported by a human baseline where three residents (48.33%) nearly double the AI average (26.67%) on 60 hard cases (Table 3). The claims are directionally well supported by the presented data; the weaknesses lie in inferential rigor and evaluation circularity rather than design.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MedRoundsQA?
A benchmark that converts 1,387 medical board-exam and NEJM diagnostic cases into simulated multi-turn doctor-patient consultations. Each case becomes a structured 24-slot clinical record, and LLM doctor agents interview an LLM patient simulator whose persona (education, emotion, language proficiency, occupation) varies while the clinical facts stay identical, letting researchers measure how patient communication style affects diagnostic accuracy.
What are the main findings of MedRoundsQA?
Across 15 evaluated LLMs, moving from single-turn diagnosis to multi-turn consultation cuts accuracy by 13-39 points. More turns improve question relevance but diagnostic accuracy plateaus after roughly 6-12 turns. Patient education level shifts accuracy by about 7.9 points between lowest and highest personas, and on 60 difficult cases three resident physicians (48.3%) roughly doubled the AI average (26.7%) using fewer turns.
How is MedRoundsQA different from AgentClinic or MediQ?
Its counterfactual design: the same clinical case is instantiated under multiple patient personas with content held fixed in a shared 24-slot record, and single-turn versus multi-turn settings are evaluated on identical normalized cases. Prior interactive benchmarks vary patients across different cases, so communication effects are confounded with case difficulty and cannot be causally attributed.
Is the code and data for MedRoundsQA available?
Yes. The paper links a public GitHub repository (github.com/youssefkhalil320/MedRoundsQA), and its appendices pin exact model snapshots for all 15 evaluated models, fixed decoding parameters, and the full prompts used for dialogue generation and judging.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-embedded values (Figs. 2, 4, 5, 6) are only partially recoverable from PDF text extraction; the plateau claim for Fig. 4 is assessed from the authors' prose and axis labels rather than the plotted curves.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.