MedRSI: Recursive Self-Improvement for Medical Agents via Clinically Aligned Self-Evolution
Junde Wu, Jiayuan Zhu, Minghao Hu, Fenglin Liu, Jiazhen Pan
cs.AI
Quick verdict
Claims first medical RSI framework for agents; strong benchmark gains, but LLM-judged clinical cost and private data limit verification.
Summary
MedRSI is presented as the first recursive self-improvement (RSI) framework for medical agents. Starting from a general OpenHands agent with GPT-4o, it runs a five-stage cycle of diagnosis, failure prioritization, reflection, tool invention (code generation, tool composition, model training) and trial-based registration. Two mechanisms are central: failures are prioritized by clinical consequence (qi = ei(1+λci), λ=2, scored by a frozen GPT-4o judge against task rubrics), and candidate tools enter the persistent registry only after gains hold across three subsequent patient cohorts without raising clinical cost. On REFUGE2 glaucoma and MITEA heart disease benchmarks, balanced accuracy rises from 56.2%/56.5% to 94.1%/82.3% over 20 rounds (five seeds), surpassing MedAgent-Pro by 3.7 and 4.5 points (rerun comparisons p=0.004/p=0.011). On two private tasks the agent invents a three-branch multimodal fusion network (92.1% vs 87.8% for a human-built fusion model) and a CycleGAN synthetic-data pipeline for contrast echocardiography (normalized MAE 0.55 vs 0.66). Ablations show uniform prioritization loses 5.4 points and immediate registration causes collapse over 30 rounds. Limitations: clinical cost is LLM-judged (κ unreported in the text), two of four datasets are private, per-run compute (212 GPU-hours) is unmatched to agent baselines, and only 16 of 118 candidates register.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MedRSI is the first recursive self-improvement framework for medicine, converting diagnostic failures into clinically aligned tools via consequence-weighted prioritization and cohort-validated registration.
- Starting from a general OpenHands+GPT-4o agent, MedRSI reaches 94.1% (glaucoma) and 82.3% (heart disease) balanced accuracy in 20 rounds, surpassing the manually engineered MedAgent-Pro by 3.7 and 4.5 points.
- On two private unseen tasks the agent autonomously invents a multimodal fusion network (92.1% vs 87.8% human-built) and a synthetic-data generative pipeline (nMAE 0.55 vs 0.66) that beat budget-matched human models.
- Both clinical mechanisms are necessary: uniform error priority costs 5.4 balanced-accuracy points (88.7% vs 94.1%), and replacing slow registration with immediate adoption causes progressive collapse over 30 rounds via error propagation.
Strengths
Staged registration with versioned, frozen candidates
Every candidate is hash-frozen at pool entry with its reference agent snapshot, evaluated paired on K=3 disjoint later cohorts under matched patients, seeds and inference settings, and registered only if mean ΔP>δ and ΔC≤0 (Sec 2.3, Eq. 13-16). The rim-texture case (+6.1 discovery vs -2.3/-5.4/-3.1 trial gains, Fig. 5f) shows the gate catching real instability before persistence.
Above-average statistical hygiene for an agentic paper
Five seeds per trajectory (final bAcc ranges 93.4-94.7% glaucoma), patient-level bootstrap CIs from 2,000 replicates (Table 2), 10,000-permutation paired tests vs round 0 (P<0.001) and vs MedAgent-Pro reruns (P=0.004/P=0.011, reruns within 0.8 points of published values), with the final round prespecified as primary endpoint (Sec 3.3).
Unusually transparent cost and funnel reporting
Backbone and hyperparameter generality
Budget-matched human baselines on the invention tasks
Weaknesses
LLM-judged clinical cost is both compass and scoreboard
Novelty claims rest on private data
Compute asymmetry on public benchmarks
'First' framing under-argued
Invention space is human-curated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MedRSI (arXiv 2609.24838, cs.AI) claims to be the first recursive self-improvement (RSI) framework for medicine. The unit of improvement is deliberately narrow: the agent Hr = (M, P, TS_r) keeps the reasoning model M and workflow P fixed and evolves only the stable tool registry. A five-stage loop (diagnose, prioritize failures, reflect, build candidates, trial and register) converts diagnostic failures into code-generated tools, tool compositions, or trained models. Two mechanisms anchor the clinical alignment story: (1) clinical-cost-aware prioritization, qi = ei(1+λci) with consequence si in {0..4} from a frozen GPT-4o judge under task-specific rubrics (λ=2); (2) fast discovery with slow registration - candidates enter an experimental pool on any positive discovery gain (Eq. 13) but are registered into the persistent agent only if mean paired gain exceeds δ over K=3 later patient cohorts with no clinical-cost increase (Eq. 16). Headline claims: a general agent becomes a specialist beating hand-built MedAgent-Pro; it invents new models for unseen private tasks; both mechanisms are necessary.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MedRSI in one sentence?
A framework that lets a medical AI agent improve itself: it reviews its own diagnostic failures, invents new tools or models to fix them, and permanently adopts only those whose benefit persists across later patient cohorts without raising clinical cost.
How well does MedRSI perform?
On the public REFUGE2 glaucoma benchmark it reaches 94.1% balanced accuracy (from 56.2% at round 0) and on MITEA heart disease 82.3% (from 56.5%), beating the hand-designed MedAgent-Pro agent (90.4%/77.8%) after 20 improvement rounds, averaged over five runs; on two private hospital tasks it also beats human-built models trained with the same budget.
Is MedRSI code and data available?
The paper points to code at github.com/ImprintLab/MedRSI and uses two public benchmarks (REFUGE2, MITEA), but two of the four tasks rely on private hospital cohorts, and full runs require GPT-4o API access plus roughly 212 A100 GPU-hours and 4.5 days per improvement trajectory.
Why does the paper argue against immediately adopting newly invented tools?
One discovered tool gained +6.1 points on its discovery cohort but lost 2.3-5.4 points on later cohorts due to camera and lighting sensitivity; when registered immediately it was invoked in 81% of cases and contaminated later self-improvement, contributing to collapse over 30 rounds - MedRSI's three-cohort trial gate discards such tools before they become persistent (43 candidates were rejected this way, 29 with negative trial gains).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Sections 3.6, parts of 3.8-3.9, Table 6, and all appendices were omitted from the extracted text; some ablation and trajectory numbers come from figure captions and axis labels rather than tables.
- The weighted κ for the clinical judge may appear in the omitted text; 'not reported' refers strictly to the provided extraction.
- Figures 2-6 are only partially parseable from text, so figure-derived claims (e.g., exact 30-round curves) could not be independently verified.
- The reference list shows one extraction artifact ('Appendix E.1.' as an entry); citations otherwise appear consistent.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.