Rethinking How We Evaluate Methodological Progress in Health AI
Florent Pollet, Matthew McDermott
cs.LG · cs.AI
Quick verdict
EHR algorithm rankings largely transfer from random tasks and across 2 ICU datasets; wide-sparse XGBoost wins. Credible but small-scale; code not yet released.
Summary
The authors re-implement 12 EHR/ICU prediction algorithms (2016-2025, from RETAIN and ICU-XGBoost to MOTOR, DuETT, and MEDS-EIC-AR) under one MEDS-based interface, largely via LLM coding agents, and evaluate them on 40 binary tasks: 10 expert-authored clinical and 10 randomly generated tasks on each of MIMIC-IV and NWICU. Aggregate pairwise AUROC comparisons agree 76-92% across the four dataset-by-task-family settings, significantly above chance after Holm correction; MEDS-Tab-large (XGBoost over a wide, sparse tabularization) ranks first in every setting, with TECO, MEDS-Tab-LR, and BEHRT last. Clinically meaningful tasks show stronger task-algorithm interaction (15-17% task-level reversals vs 8-11%; about 24% interaction variance vs under 10%), and conceptually matched tasks across datasets induce more similar rankings (Kendall tau-b 0.601 vs 0.437, p=0.019). Exploratory analyses partially support pretraining-alignment hypotheses (MOTOR-FT's edge grows with horizon on NWICU, rho=0.795). Overall message: methodological conclusions may need less clinical task engineering than assumed, and decade-scale progress lies in data representation rather than architecture recency. Main limitations: two ICU datasets, ten tasks per setting, single training runs without seed repetition, best-effort re-implementations of unverified fidelity, and code not yet released.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Aggregate pairwise algorithm comparisons transfer strongly across evaluation settings (76-92% agreement, significantly above chance after Holm correction), including from randomly generated to clinically meaningful tasks and across datasets.
- Clinically meaningful tasks exhibit greater task-algorithm interaction than randomly generated tasks (reversal rates 15.2-16.7% vs 8.3-10.9%; interaction variance about 24% vs under 10%), so task properties help explain when specific modeling choices win.
- Newer algorithms do not consistently outperform older ones: the best pipeline is gradient-boosted trees (MEDS-Tab-large) paired with a wide, sparse tabularization, implying decade-scale progress comes from data representation rather than architecture recency.
- Randomly generated tasks requiring no clinical curation can serve as a reliable first-pass substrate for methodological research in EHR AI.
Strengths
Controlled apples-to-apples comparison
All 12 algorithms train on identical task bundles through one MEDS interface, with staged verification: synthetic single-signal recovery, shuffled-label negative controls, MEDS-DEV integration checks, and MIMIC demo runs (Sec 2.1), and roughly equal implementation effort per model counters the effort-confound of historical comparisons.
First empirical test of the task-meaningfulness barrier
Quantifies whether comparisons on clinically 'meaningless' tasks match expert-authored ones: 91% MG-to-MC agreement (Kendall tau 0.818, pHolm 4.6e-5, Table 17) and all six setting pairs above chance after Holm correction, turning a previously speculative barrier (barrier (i) in Sec 1) into measured evidence.
Layered statistical reporting
Compute-performance tradeoff made explicit
Conclusions calibrated to evidence
Weaknesses
Pipelines conflated with algorithms
Single training runs, tiny rank gaps
Unvalidated LLM-agent re-implementations
Narrow task and dataset sample
Code not yet released
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper on evaluating health AI progress?
The paper finds that relative performance rankings of 12 EHR prediction algorithms agree 76-92% across datasets and between randomly generated and expert-designed clinical tasks (all significantly above chance). This suggests cheap, randomly generated benchmarks can guide methodological comparisons, though clinically meaningful tasks still retain more task-specific algorithm behavior worth studying.
Which model performed best in the comparison?
MEDS-Tab-large - gradient-boosted trees (XGBoost) applied to a very wide, sparse tabularization covering the full EHR vocabulary - ranked first in all four evaluation settings. Architecture recency did not predict performance: models from 2016-2017 remained competitive, while several recent neural models underperformed.
Is the code and data available for this study?
The data (MIMIC-IV and NWICU) are publicly available on PhysioNet. At the time of writing, the authors state the re-implemented code and the shared template repository will be released upon publication; they are not yet inspectable.
What are the main limitations of the study?
Only two ICU-centric datasets, ten tasks per setting, single training runs without seed repetition, best-effort re-implementations whose fidelity to original papers is not externally validated (implementations were AI-agent-generated and original authors were not consulted), and evaluation limited to AUROC.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 appear in the extraction as raw text dumps; curve shapes and axes could not be inspected, though the underlying numbers are available in Tables 9-23.
- Review covers arXiv v1 (2609.18134), a non-peer-reviewed ML4H 2026 proceedings-track submission; appendices appear fully included in the extraction.
- Exact clinical logic of the expert-authored tasks (ACES predicates behind Tables 6-7) could not be audited from the extracted text, so task-definition quality is judged only from reported statistics.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.