Asclepius: An Adaptive Harness for Long-Horizon Clinical Agents
Grace Chang Yuan, Xiaoman Zhang, Sung Eun Kim, Luyang Luo, Pranav Rajpurkar
cs.AI
Quick verdict
Claims adaptive harness lifts clinical agents' execution (+22% critical actions, held-out); credible but LLM-judged, single rollout per batch, n=48 held-out.
Summary
The paper studies the gap between diagnosing correctly and executing complete, timely treatment when LLM agents run long multi-patient clinical shifts in the Clinical Environment Simulator (CES), a six-hour emergency-department simulation graded by an LLM judge on four 1-5 dimensions. Across four current configurations, diagnosis averages 4.39/5 while critical actions average 2.94/5 - the 'execution gap'. The authors operationalize three failure modes as per-trace counters (instruction-adherence drift, treatment incompleteness, severity-equity neglect) and propose Asclepius: a self-evolving harness whose proposer agent rewrites the operating manual between shifts from trace feedback, a curated clinical skills library, and three context-isolated subagents for triage, diagnosis, and treatment completion. Across ten patient batches (six search, four held out), pooled gains reach +25% critical actions and +13% timeliness; on the 48 held-out patients the critical-action gain is +0.63 (p=0.024), with overall (+0.19) and timeliness (+0.31) positive but not significant. Five LLM judges from three model families reproduce the pattern, and ablations show no single component suffices. Limitations: one rollout per configuration per batch (no run-level variance), fully LLM-judged evaluation, and judge-human agreement inherited from prior work rather than rerun on these traces.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current LLM agent configurations on CES exhibit an execution gap: high diagnostic accuracy (4.39/5) but low critical-action scores (2.94/5) and middling timeliness (3.34/5).
- The gap decomposes into three per-trace failure counters - instruction-adherence drift, treatment incompleteness, and a severity-equity neglect gap - forming a coupled bottleneck that no single component resolves.
- Asclepius improves critical-action correctness by 22% on held-out batches (p=0.024) and by 25% (critical actions) / 13% (timeliness) pooled over ten batches, while preserving diagnostic accuracy.
- The gains replicate across five LLM judges from three model families, so they are not an artifact of the primary GPT-4.1 grader.
- The gains are not explained by inference compute: full Asclepius uses only 1.21x the LLM calls and 1.52x the wall-clock of the baseline.
Strengths
Clean held-out discipline with transparent compression
Four validation batches (48 patients) are never shown to the proposer (Sections 5.2, D.2); the gain compression from +0.36 (search) to +0.19 overall (held-out), ~47%, is reported openly in Table 5/E.1 and framed as a lower bound on the generalizable component rather than hidden.
Multi-judge robustness check addresses grader circularity
Because the primary judge (GPT-4.1) shares a family with the simulator's Patient Engine, the authors re-grade all 120 patients under baseline and full configurations with four additional judges across three families (Claude Sonnet 4.5/5, Gemini 3.1 Pro, GPT-5), all reproducing the ordering (Section 5.1, Table 3, D.1).
Above-average statistical machinery
Failure modes are mechanistically grounded, not anecdotal
Cost accounting and search trajectory fully reported
Weaknesses
Single rollout per configuration per batch
Headline pooled figures are partially in-sample
Ablations graded by a single judge
Simulation-only evaluation with inherited judge validation
Diagnosis preservation is slightly overstated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets long-horizon agent execution: LLM agents managing a full simulated emergency-department shift in the Clinical Environment Simulator (CES), graded by an LLM judge on four 1-5 dimensions (diagnosis, critical actions, timeliness, disposition). Central claims: (1) an **execution gap** exists - diagnosis 4.39/5 vs critical actions 2.94/5 across four current configurations (Section 1); (2) the gap decomposes into three per-trace counters - instruction-adherence drift (baseline naked-disposition rate rising 16%->57%), treatment incompleteness (19% zero-treatment), and a severity-equity neglect gap (1.13 timeliness difference, Section 6.3); (3) Asclepius - a self-evolving manual plus skills library plus three isolated subagents - closes much of it (+22% critical actions on held-out batches, p=0.024; +25%/+13% pooled); (4) the three failure modes form a **coupled bottleneck** released only when all components act together.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the 'execution gap' in this paper?
It is the mismatch between an LLM agent correctly diagnosing patients and actually delivering complete, timely treatment under multi-patient load. In the CES emergency-department simulation, agents score about 4.39/5 on diagnosis but below 3.0/5 on critical actions (2.94/5) and 3.34/5 on timeliness across four existing configurations.
What is Asclepius and how does it improve agent performance?
Asclepius is an agent scaffolding with three parts: a self-evolving operating manual that a proposer agent rewrites between simulated shifts using trace-level feedback, a curated clinical skills library of regimen and dosage knowledge, and three isolated subagents (triage, diagnosis, treatment completion). Together they raise critical-action scores by about 25% pooled over ten batches, and +0.63 points (p=0.024) on 48 held-out patients.
Was Asclepius validated on real patients or by doctors?
No. All evaluation is in the CES simulation with LLM-judged scoring. Human-agreement statistics for the judge (Gwet's AC around 0.86-0.93) come from the simulator's prior validation on other trajectories, and the authors state no expert-agreement study was run on Asclepius's own traces.
Are the reported improvements statistically significant?
Partially. On the 48 held-out patients, only the critical-action gain (+0.63) is significant (p=0.024); the overall (+0.19) and timeliness (+0.31) gains are positive but not significant. The larger pooled figures (+25% critical actions, +13% timeliness) include six batches that were used during harness evolution, so they are partially in-sample.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with '[...sampling gap...]' markers; the numeric values of Table 1 (main results), Table 3 (five-judge validation), and Table 8 (ablation significance) fall inside sampling gaps and were not visible, so per-dimension held-out significance for diagnosis could not be checked.
- Parts of Section 3 (harness details), Sections 6.1-6.3, and Appendices B, C, and F are inside sampling gaps; judgments about those components rely only on the visible portions.
- No code or data availability statement appeared in the sampled text; one may exist in unsampled sections (e.g., later appendices).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.