LLMs Anchor on Chief Complaint and Fail to Integrate Evidence in Sequential Clinical Triage
Dipankar Srirag, Haokai Zhao, Ashutosh Kumar, Eleanor Hopper, Michael Dalton, Quoc Dung Nguyen, Aditya Joshi, Salil S. Kanhere, et al.
cs.CL
Quick verdict
Six LLMs anchor on chief complaints in sequential triage, plateauing despite more evidence; rigorous design but rests mainly on LLM-generated dialogue.
Summary
The paper introduces a sequential evaluation of LLM-based emergency triage: instead of classifying completed retrospective EHR records, models must assign an Emergency Severity Index (ESI) label from growing prefixes of a nurse-patient conversation, evaluated at five checkpoints beginning with the chief-complaint exchange (k=4). Six LLMs (Gemma-4, Qwen-3.5, Claude Opus 4.6, GPT-5.5) are tested on 425 LLM-generated conversations (SIMULATED, from TriageSim and EHR2Dial seeded by MIMIC-IV-ED) and 50 clinician-authored ones, using quadratic weighted kappa (QWK) with 10,000-resample bootstrap CIs. Offline scores (best: Claude, QWK 0.701) collapse in the sequential setting to roughly 0.31-0.50 on the synthetic corpus, plateauing at the first checkpoint; reasoning, Plan-and-Solve, and self-consistency prompting do not help. Perturbation experiments localize the decision signal to the chief-complaint turns, while deleting later turns has near-zero effect. Models recover chief complaints and red flags from later turns (cosine 0.57-0.77), yet surprisal of the true label rises (+0.74 to +1.82 nats). Three clinician annotators reach QWK-at-commit 0.887-0.943 versus 0.295 for the best model. Main limitations: the primary corpus is synthetic, the clinician corpus is small (n=50) with CIs spanning zero for all plateau deltas, and the annotators' stated ED experience is modest despite the 'expert' framing.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- All six LLMs degrade from offline QWK (best 0.701, Claude Opus 4.6) to a fair-to-moderate plateau fixed at the first checkpoint (k=4), and prompting interventions (native reasoning, Plan-and-Solve, self-consistency) fail to lift it.
- Controlled perturbations localize the triage decision to the chief-complaint turns: altering them shifts predictions at every checkpoint, while deleting later turns has near-zero effect.
- Models extract clinically relevant content from post-CC turns (CC recovery cosine 0.57-0.77; red-flag density 3.8/5.0 at ESI-1 vs 1.8/0.8 at ESI-5) yet surprisal of the true label rises +0.74 to +1.82 nats, so evidence integration rather than extraction fails.
- On the same 50 conversations, three clinician annotators reach QWK-at-commit 0.887-0.943 versus 0.295 for the best model; models commit earlier, concentrate on ESI-2/3, under-triage ESI-1, and agree with each other more than with the ground truth, so ensembling would worsen the failure.
Strengths
Novel sequential evaluation protocol
Predicts ESI from growing conversation prefixes at five checkpoints (k=4 covering the chief-complaint exchange, then +25/50/75/100%), directly addressing the gap that retrospective-record evaluations (Masanneck 2024, Williams 2024) use the nurse's conclusion rather than mid-encounter evidence (Sections 1-2).
Clean causal localization of anchoring
The CC-DELAYED perturbation matches CC-REMOVED through k=+75% and recovers to baseline exactly at +100% on every model, separating anchor content from anchor position; later-interval removals (REMOVED-I/II/III) yield near-zero deltas across all models (Section 4.2, Table 8, Figure 6).
Triangulated extract-but-not-integrate mechanism
Human comparison under matched evidence and interface
Statistical and artifact transparency
Weaknesses
Primary corpus is LLM-generated
Clinician corpus does not support the blanket plateau claim
Inconsistent and under-documented human baseline
Modality confound in the OFFLINE-SEQUENTIAL contrast
Ecological validity of the label distribution
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper defines **sequential clinical triage**: predicting an Emergency Severity Index (ESI) label from a growing prefix of a nurse-patient conversation, evaluated at five checkpoints (k=4, then +25/50/75/100% of remaining turns), against the conventional OFFLINE setting that classifies the full structured MIMIC-IV-ED record at once. Four research questions structure the study: how well models perform sequential triage; where in the conversation they commit; how their commitment compares with clinicians under identical evidence; and why more evidence does not help. Central claims: (1) all six models degrade from offline QWK (best: CLAUDE 0.701, Table 2) to a fair-to-moderate plateau fixed at the first checkpoint; (2) the decision signal localizes to the chief-complaint (CC) turns; (3) models extract but fail to integrate post-CC evidence, with surprisal of the true label rising +0.74 to +1.82 nats (Table 11); (4) clinicians far outperform models at commit time (0.887-0.943 vs 0.295, Table 3), and ensembling would worsen failure because models agree with each other (Krippendorff alpha 0.67-0.69 on SIMULATED) more than with the truth.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is 'sequential clinical triage' in this paper?
Instead of classifying a completed retrospective record, a model must predict an ESI urgency label at successive checkpoints from a growing prefix of the nurse-patient conversation. The first checkpoint (4 turns) covers the chief-complaint exchange; the remaining four reveal 25/50/75/100% of the later turns.
How much worse are LLMs at sequential triage than on full records?
Substantially. Offline, the best model (Claude Opus 4.6) reaches quadratic weighted kappa 0.701 on the structured records. Sequentially, every model plateaus at fair-to-moderate agreement from the first checkpoint onward (about 0.31-0.50 on the simulated corpus for the best configurations), and adding later conversation turns does not improve - and sometimes slightly hurts - performance.
What does 'chief-complaint anchoring' mean here?
The models' triage label is essentially fixed by the opening chief-complaint turns. Perturbing those turns (deleting, delaying, reordering) changes predictions at every checkpoint, while perturbing or deleting later turns has near-zero effect - the models behave as if later evidence did not exist, mirroring the anchoring bias documented in human clinical decision-making.
Why does the paper say ensembling LLMs will not fix this?
Because the six models largely make the same mistakes: their predictions agree with each other substantially (mean pairwise QWK about 0.71-0.73 on the simulated corpus) - more than any model agrees with the ground truth (about 0.31-0.50). Majority voting therefore pools the same anchored errors instead of cancelling them, and the paper's VOTE ensemble indeed performs worse than the best single model.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2-11 and 13-19 are referenced but not visually analyzable from the extracted text; plot-level claims rely on the appendix tables, which are present.
- Table 5/6 CI bounds show minor internal inconsistencies between sections (e.g., GEMMA-L CLINICIAN k=4 listed as [0.427, 0.797] and [0.427, 0.802]); exact bounds were not exhaustively cross-checked.
- Minor source inconsistencies (abstract's 0.887-0.929 vs Table 3's 0.943; n=40 vs 50 in confusion-matrix captions) may be extraction artifacts or source typos; flagged for authors either way.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.