EMR: Self-Evolving Medical Multi-Agent System via Experience Mining and Reuse
Dongsheng Shi, Yue Li, Xin Yi, Linlin Wang
cs.CL · cs.AI
Quick verdict
Claims medical multi-agent self-evolution via experience mining (81.1 vs 77.8 avg acc, Qwen3-8B); strong ablations but no significance tests or code.
Summary
The paper introduces EMR, a training-free medical multi-agent framework in which a planner assigns question- and option-level clinical domains, department agents reason per domain, and a summary agent integrates the analyses. After each case, a collector distills the multi-agent trajectory into reusable natural-language experiences — 'golden' insights from correct answers and 'warning' lessons from failures — organized into a three-level library (principles, patterns, cases). A bge-m3 similarity gate (θ=0.85) decides whether new entries are added or merged. On nine medical MCQ benchmarks with Qwen3-8B and GPT-4o backbones, EMR reports average accuracy of 81.1% and 93.9%, exceeding MDAgents by 3.3 and 1.5 points and experience-based baselines such as EvolveR and MDTeamGPT. Ablations attribute the largest share to principle-level experiences (−4.6 on MedQA when removed) and to warning experiences (−2.6 when golden-only). Mined experiences transfer across backbones (GPT-4o-mined experiences lift Qwen3-8B by +9.6 on MedQA over no-experience) and, less strongly, across datasets. Main limitations: no significance testing or variance reporting despite three runs and margins as small as +0.6 (MedMCQA, GPT-4o); inconsistent numbers across tables (74.3 vs 75.3 for the same no-experience condition; 15 s vs 45 s inference time); evaluation restricted to exam-style MCQ with half the evaluation distribution used for mining; and no code or library release.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- EMR's hierarchical experience library (principles/patterns/cases) with golden+warning entries consistently beats static multi-agent baselines (avg +3.3 over MDAgents on Qwen3-8B; +1.5 on GPT-4o, Table 1) and experience-based methods (EvolveR, MDTeamGPT, G-Memory).
- Experiences are transferable: GPT-4o-mined experiences improve Qwen3-8B by +9.6 on MedQA over no-experience, and cross-dataset transfer yields +3.2/+3.5 (Tables 4-5).
- Warning experiences from failed trajectories are essential: golden-only settings drop 2.6 points on MedQA (Table 6), and principle-level entries are the largest hierarchy contributor (−4.6 when removed, Table 3).
- The mechanism is architecture-agnostic: plugging EMR into MDAgents and ReConcile improves MedQA accuracy to 96.5 and 94.8 under GPT-4o (Table 8).
Strengths
Unusually complete ablation program
Component removal (Table 2: experience −6.7; keeping experience while dropping planner+departments costs only −2.9), hierarchy levels (Table 3: principles −4.6, patterns −2.5, cases −1.2 on MedQA), θ sweep and golden-vs-warning (Table 6: golden-only −2.6), library size/noise robustness (Table 10: 76.8 with 20% library, 79.3 with 20% noise), agent count, retrieval size and epoch curves (Figs 5-7).
Strong, broad main results
Table 1: average accuracy 81.1 (Qwen3-8B) and 93.9 (GPT-4o) across 9 benchmark subsets, above all 8 baselines including EvolveR (79.6/90.3), MDTeamGPT (78.4/88.9) and MDAgents (77.8/92.4); largest single gain MedQA/GPT-4o 96.8 (+4.1 over EvolveR).
Effective abstraction design, empirically validated
Transferability evidence
Reproducibility-minded disclosure
Weaknesses
No statistical rigor
Inconsistent reported numbers
Same-distribution mining confound
Baseline provenance and compute parity unclear
No artifact release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
EMR (arXiv:2609.15161, ECNU / City University of Hong Kong) is a training-free medical multi-agent framework. Its central claim: an LLM multi-agent system can self-evolve by mining its own reasoning trajectories into a persistent hierarchical experience library — principles, patterns, cases — split into golden (success-derived) and warning (failure-derived) entries, then retrieving them to guide future medical QA. Four supporting claims: (1) consistent wins over static multi-agent baselines (avg 81.1 vs MDAgents 77.8 on Qwen3-8B; 93.9 vs 92.4 on GPT-4o, Table 1) and over experience-based methods (EvolveR 79.6/90.3, MDTeamGPT 78.4/88.9); (2) cross-backbone and cross-dataset transfer of experiences (Tables 4-5); (3) hierarchy and warning experiences are each necessary (Tables 3, 6); (4) the mechanism is architecture-agnostic (Table 8).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is EMR (the medical multi-agent system)?
EMR is a training-free framework where multiple LLM agents (a planner assigning clinical departments, department experts, and a summary agent) answer medical questions, while the system automatically mines reusable 'experiences' from its own reasoning traces — success strategies ('golden') and failure lessons ('warning') — into a three-level text library (principles, patterns, cases) that is retrieved to guide future questions. No model parameters are updated.
How much does EMR improve accuracy over prior methods?
Across nine medical multiple-choice benchmarks (MedQA, MedMCQA, PubMedQA, MMLU subsets, MMLU-Pro Health/Biology), EMR reports average accuracy of 81.1% with Qwen3-8B (vs 77.8% for MDAgents, 79.6% for EvolveR) and 93.9% with GPT-4o (vs 92.4% for MDAgents). Its largest single gain is 96.8% on MedQA with GPT-4o, +4.1 points over EvolveR. Note the paper reports no statistical significance tests.
Does EMR fine-tune or retrain the underlying LLM?
No. The experience library is external text retrieved at inference time via embeddings (bge-m3, similarity threshold 0.85, top-3 entries); model weights stay frozen. Mining cost is reported at about $5 per 100 samples.
Is EMR validated for real clinical use?
No. Evaluation is limited to offline exam-style multiple-choice benchmarks with half of each dataset used for experience mining. The authors explicitly state the system has not been validated in real clinical environments and its outputs should not be interpreted as medical advice.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 and 5-7 are not analyzable from the extracted text (only captions available), so agent-count and epoch-scaling claims rest on the authors' descriptions.
- The metadata abstract is truncated mid-sentence, but the full text including appendices A-M was extracted, so coverage is good.
- Table values come from text extraction; the 74.3/75.3 discrepancies could partly stem from extraction artifacts, though both values appear in clearly labeled rows of different tables.
- No code repository is referenced anywhere in the text, so code availability is judged as not provided.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.