Same Patient, Different Order: Action-Level Reliability of Clinical LLM Agents Under Repeated Runs
Rohith Reddy Bellibatlu, Manpreet Singh, Zhoutian Han, Wenbin Zhang
cs.CL · cs.AI · cs.LG · cs.SE
Quick verdict
Shows clinical LLM agent orders diverge across identical runs while scores stay flat; rigorous, honest, but small 2-model panel limits generalisation.
Summary
The paper asks whether a clinical LLM agent given identical inputs produces identical actions across repeated runs, and shows that MedAgentBench's single-attempt scoring cannot see the answer. The authors introduce 'same-input rerun': each of 50 write-capable tasks is replayed five times with every input fixed, for two open-weight models (Qwen3-4B and Llama-3.1-8B, both four-bit quantised) at temperatures 0.05 and 0.7, yielding 1000 runs scored with six action-level reliability metrics. In the headline arm (Llama-8B, 0.7), all 43 ordering groups emit different order sets across runs, 26 omit the order on some runs, and 28 post a different code, dose or analyte; in 22 of the 43 the benchmark's pass/fail verdict stays the same while the behaviour differs. A live FHIR server check shows misrouted orders would be rejected while the agent is told it succeeded. The authors are explicit that the study establishes divergence exists and passes unrecorded, not that any rate generalises: the panel is two small quantised models on one benchmark, tasks are a deterministic prefix, and family-clustered intervals collapse in two arms. Code, analysis scripts and all 1250 run records are released, with an automated gate re-deriving every table.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Replaying identical inputs five times shows clinical agent orders diverge materially in every measured arm, including different codes, doses or analytes in 28 of 43 ordering groups for Llama-3.1-8B at temperature 0.7.
- Single-run benchmark scoring can report the same pass/fail verdict over materially different behaviour: 22 of 43 divergent groups in the Llama@0.7 arm and 10 of 10 in Qwen@0.7 hold one verdict while orders change.
- MedAgentBench's runtime replies success to any parseable write without executing it, so an order the real FHIR server would reject (400 HAPI-0450 at the collection root vs 201 at the typed endpoint) reaches the agent as a confirmed success.
- Order-set divergence and omission rise with decoding temperature (0.05 to 0.7) in both models, with Holm-corrected paired McNemar significance across all six temperature contrasts.
- The study establishes only that divergence exists and can pass unrecorded by the score; no rate is claimed to generalise beyond this panel, scaffold and benchmark.
Strengths
Execution check against a real record server
Section 6 settles whether endpoint divergence is cosmetic by posting an identical ServiceRequest payload to the typed endpoint (201 Created) and the collection root (400 Bad Request, HAPI-0450) on the shipped HAPI FHIR server — misrouted orders would not exist, while the benchmark tells the agent every parseable write succeeded.
Infrastructure attribution before blaming the agent
A noise floor (20 identical calls x 18 cells, Appendix B) proves serial greedy decoding is exactly reproducible on this stack, and a batched-serving experiment shows 4 of 12 greedy cells become nondeterministic under concurrent load — a result useful to any benchmark deploying at temperature 0.
Denominator and budget discipline
Exceptional artefact release
Statistical honesty
Weaknesses
Narrow, weak model panel
Deterministic task prefix
Clinical-content divergence mechanism unresolved
Original scaffold only
Fragile generalisation inference
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The thesis is deliberately narrow: a clinical agent benchmark scoring one run per task cannot see whether identical inputs produce identical actions. The authors introduce 'same-input rerun' — replay each task five times with every input fixed and compare the emitted orders, not the score — with six reliability metrics (order-set divergence, omission split, endpoint divergence, clinical-content divergence, trajectory first-divergence, verdict instability). Applied to 50 write-capable MedAgentBench tasks, two open-weight models (Qwen3-4B, Llama-3.1-8B; both q4_K_M via Ollama) at T=0.05/0.7, it yields 1000 runs (1250 records with the 8-bit Qwen arm). Claims are explicitly existential: divergence exists and can pass unrecorded by the score; no rate is claimed to generalise. Headline numbers — Llama-8B@0.7's 43/43 divergent ordering groups — are consistently paired with arm-specific denominators and Wilson intervals, a scoping discipline rare in this literature.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'same-input rerun' in this paper?
It is an evaluation protocol that replays a benchmark task multiple times with every input held identical and compares the actions (orders) the agent issues across runs — which resource it posted, where, with what payload, or whether it posted at all — rather than only comparing the pass/fail score.
Does this paper show clinical LLM agents are unsafe?
No. The benchmark never forwards agent writes to the record server, so the paper measures the orders issued, not orders received; the authors explicitly do not claim clinical agents are unsafe to deploy, and their panel is two small quantised open-weight models.
What is wrong with MedAgentBench according to the study?
Nothing undisclosed: it scores one attempt per task and states so, and it does not execute writes. The paper's criticism is that the runtime tells the agent every parseable write succeeded, so a request a real FHIR server would reject (wrong endpoint, 400 vs 201) looks successful to the agent, hiding misrouting.
Do the reported divergence rates apply to larger models like GPT-4?
Unknown. The study tested only Qwen3-4B and Llama-3.1-8B at 4-bit quantisation on one local serving stack; the authors state the protocol transfers but the rates do not, and showing stable success would need a stronger model panel.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1 and 2 are known only through captions and surrounding prose; the graphic content could not be inspected in the extracted text.
- The metadata abstract ends mid-word ('execut'), but the full text contains the complete closing sentence, so no abstract content is missing.
- Hyphenated line breaks inside words (e.g. 'MedA-gentBench') are PDF extraction artifacts, not content gaps.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.