Safe to Stop? Risk-Constrained Stopping for Sequential Clinical Diagnosis Agents
cs.AI
Quick verdict
CROS adds LTT-style finite-sample stop/defer risk control to clinical diagnosis agents; results are honest but exploratory (labels previously viewed).
Summary
The paper presents CROS (Clinical Risk-constrained Optimal Stopping), a stopping layer for sequential clinical diagnosis agents that must decide when to stop requesting tests, commit to a diagnosis, or defer. CROS combines a learned state-wise error ranker over backbone trajectories, policy design on disjoint development splits, and LTT-style exact binomial tests that jointly control selective diagnostic error (≤ 0.25) and minimum autonomous coverage (≥ 0.70) at family-wise level δ = 0.05. Episode-wise randomized LP mixtures with at most three support policies extend the guarantee. On a 1,834-episode MIMIC-derived abdominal-pain benchmark using a Qwen2.5-7B LoRA backbone under a common-path protocol, the full ranker reaches state-error AUROC 0.853 (versus 0.715 for maximum class probability and 0.552 for the native stop score), and CROS-Mix yields 16.9% selective error at 78.8% coverage with cost 5.57, versus 30.8% error at 100% coverage and cost 8.14 under native stopping. The paper is unusually candid about negative findings: calibration and evaluation labels were previously inspected, so no realized statistical guarantee holds; the no-history ranker nearly matches the full ranker; the uniform-weight mixture is cheaper on evaluation; CROS-Mix passes nominally in only 6 of 20 resplits; and class-level selective risks reach 0.50-1.00. The authors require a prospectively frozen, never-viewed cohort before any safety claim.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LTT-style exact joint tests provide finite-sample guarantees on selective diagnostic error and minimum autonomous coverage for complete frozen sequential stopping policies, including episode-wise randomized mixtures (Theorem 1, Proposition 1).
- A learned state-error ranker substantially outperforms confidence-based and native scores at identifying wrong backbone diagnoses (state-error AUROC 0.853 vs 0.715 and 0.552, Table 2).
- On the evaluation split, CROS-Mix attains 16.9% selective error at 78.8% coverage with cost 5.57, versus 30.8% error at 100% coverage and cost 8.14 under the backbone's native stopping.
- Forced continuation is non-monotone on this benchmark (28.3% error with HPI alone rising to 34.3% after full workup), motivating an explicit, testable stopping layer.
- Because calibration and evaluation labels were previously accessed, all reported results are exploratory feasibility and audit evidence, not a confirmatory safety certificate.
Strengths
Rigorous statistical architecture
Exact binomial intersection-union tests for both selective error and coverage (Eqs. 5-8), Theorem 1 with a correct proof handling discreteness and the M=0 case, and Proposition 1's ≤3-support LP sparsity; Appendix A explicitly identifies the two failure modes that would break validity (shared component draw, weights re-estimated on calibration).
Exceptional transparency about negative results
Label reuse, the null No-history ablation, uniform-mixture cost advantage, 6/20 resplit pass rate, and class risks up to 1.000 are all reported in the abstract and main text rather than buried - rare candor that makes the paper's claims easy to bound precisely.
Common-path protocol cleanly isolates stopping
Exemplary reproducibility documentation
Honest benchmark framing
Weaknesses
The central guarantee is not realized
Design pipeline is fragile and selection-split overfit
Feature ablation undermines the risk-ranker motivation
Severe class-level failures under a marginal-only guarantee
Single backbone, single seed, single-center retrospective data with heavy missingness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets an under-specified decision in LLM-based clinical diagnosis agents: when to stop requesting tests and commit to a diagnosis, or defer. It contributes (1) CROS, a stopping layer combining a state-wise error ranker, policy design on disjoint development splits, and LTT-style exact joint tests of selective diagnostic error (α = 0.25) and minimum autonomous coverage (γ = 0.70) at FWER δ = 0.05; (2) episode-wise randomized LP mixtures with a ≤3-support sparsity result (Proposition 1); (3) a 1,834-episode MIMIC-derived abdominal-pain benchmark with 12 costed actions and a common-path protocol; and (4) an unusually candid account of negative results. The headline claim is deliberately hedged: because calibration and evaluation labels were previously accessed, everything is framed as exploratory feasibility and audit evidence, not a confirmatory safety certificate (Abstract, §4.1, §6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CROS?
CROS (Clinical Risk-constrained Optimal Stopping) is a stopping layer for sequential clinical diagnosis agents: a learned risk score decides at each step whether to accept the model's current diagnosis, continue with the next proposed test, or defer to a human. When the candidate policies, testing rule, and any randomization are frozen before calibration labels are accessed, exact binomial (LTT-style) tests provide a finite-sample joint guarantee on selective diagnostic error and minimum autonomous coverage.
Is CROS safe to deploy in clinics?
No. The paper explicitly states its results are exploratory because evaluation labels had been inspected during earlier development, so no statistical guarantee was realized. Class-level selective error reached 50-100% for some diagnoses, and the authors call for a new prospectively frozen, never-viewed cohort before any safety claim.
What data was used in the study?
1,834 emergency-department episodes for abdominal pain built from MIMIC-IV-ED v2.2 linked with MIMIC-IV-Ext-CDS v1.0.2 (PhysioNet, credentialed access), covering nine diagnosis classes and 12 test-action groups, split into 1,100 development, 367 calibration, and 367 evaluation episodes with no patient overlap.
How much does CROS reduce testing cost?
On the 367-episode evaluation split, the CROS-Mix controller had expected cost 5.57 with 0.68 requested tests versus 8.14 and 1.53 tests for the backbone's native stopping, while cutting selective error from 30.8% to 16.9%. Caveats: the uniform-weight ablation was cheaper still, and all numbers are exploratory due to prior label access.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 2–4 are extracted only as axis-label/legend fragments; curve shapes could not be visually inspected, though the underlying per-stage and per-scenario numbers appear in Tables 11 and 15.
- The metadata abstract is cut mid-sentence, but the complete abstract is present in the embedded full text; no end-of-text truncation was detected.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.