A Structured Debate-Mixture-of-Agents Framework for Complex Clinical Diagnostic Decision Support
Chang Xia, Leilei Ouyang, Huimin Wang, Yong Zhao, Kang Li
cs.CL · cs.AI
Quick verdict
DMoA multi-agent debate lifts GPT-4o pooled top-1 accuracy 10.2 pp on hard cases; gains credible but LLM-judged, one-sided tests, partly non-significant.
Summary
The paper introduces Debate-Mixture-of-Agents (DMoA), a multi-agent framework for complex clinical diagnosis built on the Mixture-of-Agents (MoA) architecture: each of several Debaters pairs a Presenter, which generates an initial diagnosis, with a Rebutter that critiques unsupported reasoning, missing evidence, and safety concerns before revision; a single Aggregator then merges all revised hypotheses without forcing consensus. Using GPT-4o, GPT-3.5, and GPT-4o mini as base models, the authors evaluate on 297 rare-disease cases and 1,719 diagnostically challenging cases from a published ClinDiag dataset, plus 1,000 sampled MedQA questions. Pooled across both case sets, GPT-4o-based DMoA (4×2 structure) raised exact-match top-1 accuracy from 16.58% to 26.79% (+10.21 pp) and top-1 safety rate by 11.36 pp (to 74.15%), with larger gains on challenging cases than on rare diseases; several rare-disease comparisons were not significant (e.g., top-3 accuracy p=0.1185). Ablations show prompt-only emulation, MoA without the Rebutter, and MAC-style debate each underperform full DMoA. DMoA brought no benefit on basic MedQA questions, and a mixed configuration (lightweight debaters with a GPT-3.5 Aggregator) underperformed single GPT-3.5. Main limitations: accuracy and safety are scored solely by an LLM judge (GPT-4o mini) without clinician validation; one-sided McNemar tests without multiple-comparison correction; published cases may sit in model training data; and the case source is a non-peer-reviewed preprint from a collaborating group.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- DMoA's role-constrained generate-critique-revise-aggregate topology raises exact-match top-1 diagnostic accuracy by 10.21 pp (16.58%→26.79%) and top-1 safety rate by 11.36 pp (to 74.15%) over single GPT-4o, pooled across 297 rare-disease and 1,719 challenging cases.
- The gains reflect the structured workflow rather than more models, longer outputs, or prompts alone, since prompt-only emulation, MoA without the Rebutter, and MAC-style debate all underperform full DMoA (Sec. III.E).
- DMoA's advantage scales with task complexity: it yields little or no benefit on 1,000 basic MedQA questions (Table V).
- DMoA improves even lightweight base models (GLM-4-9B rare-disease top-1 accuracy 8.08%→15.15%, safety 31.99%→55.56%, p<0.001), but a GPT-3.5 Aggregator over lightweight debaters does not beat single GPT-3.5.
- Performance depends on configuration: the 4×2 structure, stronger base models, and larger token budgets help, while information partitioning across debaters does not.
Strengths
Comprehensive ablation design isolating the mechanism
Full DMoA is compared against prompt-only emulation, MoA without the Rebutter (4×1), and MAC-style single-Debater debate (1×2, five rounds) across three base models on both case sets (Tables III–IV), cleanly separating topology from prompt length, model count, and multi-turn dialogue; every reduced variant underperforms full DMoA.
Honest negative and boundary results
No gain on 1,000 MedQA questions (GPT-4o 89.03% vs 88.00%, p=0.1092; GPT-4o mini DMoA worse, 74.10% vs 75.90%, Table V); mixed lightweight+GPT-3.5 aggregation below single GPT-3.5 (19.53% vs 22.22% top-1, p=0.9381, Sec. III.G); information partitioning hurts (29.63% vs 35.20%); reward-model variant helps safety (69.36% vs 62.63%) but lowers accuracy (31.65% vs 35.20%).
Multi-dimensional safety rubric with quantified judge agreement
Practical deployment and cost analyses
Reliability testing and artifact release
Weaknesses
Safety construct has no clinical anchor
Statistical reporting favors the hypothesis
Contamination and data-provenance risk
Error mass unexamined in the main text
Framework description exceeds what was evaluated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This IEEE-JBHI-format paper (arXiv 2609.05069) proposes Debate-Mixture-of-Agents (DMoA), a role-constrained multi-agent topology for complex clinical diagnosis built on Mixture-of-Agents [13]. N Debaters each pair a Presenter (initial diagnosis) with a Rebutter that flags unsupported reasoning, missing evidence, alternative diagnoses and safety concerns; the Presenter revises, and one Aggregator merges all revised hypotheses into top-1/top-3 outputs without forcing consensus (Algorithm 1, Fig. 2). Central claims: (1) pooled across 297 rare-disease and 1,719 challenging published cases, GPT-4o-based DMoA (4×2) lifts exact-match top-1 accuracy from 16.58% to 26.79% (+10.21 pp) and top-1 safety rate by 11.36 pp to 74.15% (Abstract; Sec. III.E); (2) the gains reflect the structured workflow rather than more models, longer outputs or prompt design alone; (3) benefits scale with task complexity, vanishing on 1,000 basic MedQA questions (Sec. III.I, Table V).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DMoA and how does it differ from Mixture-of-Agents (MoA)?
DMoA is a multi-agent framework for clinical diagnosis in which several 'Debaters' each generate a diagnosis (Presenter), critique it (Rebutter), and revise it, after which one 'Aggregator' merges all revised hypotheses into the final top-1 and top-3 diagnoses. Standard MoA only aggregates multiple models' answers; DMoA adds the structured per-agent rebuttal-and-revision loop, and the paper's ablations show that removing the Rebutter lowers both accuracy and safety on its case sets.
How much does DMoA improve diagnosis over GPT-4o alone?
Pooled over 297 rare-disease and 1,719 challenging published cases, GPT-4o-based DMoA (4×2 structure) raised exact-match top-1 accuracy from 16.58% to 26.79% (+10.21 percentage points) and top-1 safety rate by 11.36 points (to 74.15%). Gains were larger on challenging cases (14.50% to 25.34%) than on rare diseases (28.62% to 35.20%), where some comparisons were not statistically significant. All scores were assigned by an LLM judge (GPT-4o mini), not by clinicians.
Does DMoA help on easy medical questions?
No. On 1,000 sampled MedQA questions, DMoA showed no significant benefit: GPT-4o scored 89.03% with DMoA versus 88.00% as a single model (p=0.1092), and with GPT-4o mini as base, DMoA actually scored lower (74.10% vs 75.90%). The authors argue the framework's benefit only appears as diagnostic task complexity increases.
Is the code and data available for this paper?
Yes. The authors release code under the Apache-2.0 license and their filtered MedQA question set via GitHub (Ivesxia/Debate-Mixture-of-Agent). The 297 rare-disease and 1,719 challenging cases come from a third-party ClinDiag dataset described in a Research Square preprint, to be used under the original authors' data-use terms.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Supplementary Tables 1–15 and Notes 1–3 are cited throughout but their contents are not part of the provided text; several claims (per-dimension safety detail, error-analysis percentages, exact prompts) rest on them.
- The provided main text ends at reference [46] with no truncation; all quoted numbers were verified against extracted Tables I–V and section text. The pooled single-GPT-4o safety baseline (~62.8%) is arithmetic from Tables III–IV, not a quoted figure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.