Misaligned Clinical Risk Classification and Cost Asymmetry in Open-Weight Large Language Models
Star S. D. Liu, Xiyu Ding, Robert B. Barrett, Alberto Santamaria-Pang, Nic Dobbins, Harold P. Lehmann
cs.LG · cs.AI
Quick verdict
Claims LLMs encode but fail to integrate clinical cost tradeoffs (2/12 cells cost-correct); rigorous design but one small dataset and no code yet.
Summary
This paper asks whether four open-weight LLMs (Qwen-2.5-7B/32B, Llama-3.1-8B/70B) internally represent and behaviorally use false-negative/false-positive cost asymmetries in clinical classification. Using the public Pima Indians Diabetes dataset (768 patients), the authors vary 11 FN:FP cost ratios across three phrasings and three prompt conditions (180 configurations), extracting HIGH/LOW next-token probabilities and per-layer residual-stream activations. Linear probes recover patient risk (AUC 0.81-0.83, matching supervised classifiers) and cost direction in all models. Behaviorally, however, only 2 of 12 model-phrasing cells (Llama-3.1-70B and Qwen-2.5-32B, baseline phrasing) pass both the mirror test and the coherence test; magnitude responses are predominantly direction-agnostic (common-mode dominant in 8 of 12), and a cost direction fit on one side never inverts when transferred to the other. Representation-behavior correlations appear only in the two larger models (Pearson r up to 0.73 for Qwen-2.5-32B). Limitations include a single small binary dataset, forced-choice output format, observational non-causal probing, no significance tests on mirror contrasts, and code not yet released. The authors conclude LLMs encode cost information but do not reliably integrate it, recommending tradeoff-conditioning tests and default operating-point reporting in clinical evaluations.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Patient risk and cost direction are linearly recoverable from residual-stream activations in all four open-weight LLMs (risk probe AUC 0.81-0.83, comparable to supervised classifiers).
- Behavioral decisions track the specified cost tradeoff in only 2 of 12 model-phrasing cells (Llama-3.1-70B and Qwen-2.5-32B, baseline phrasing only).
- Responses to cost magnitude are predominantly direction-agnostic (common-mode dominant in 8 of 12 cells), and a cost direction fit on one side never inverts when transferred to the other, ruling out mirror-symmetric encoding.
- Representation-behavior linkage exists only in the two larger models (Pearson r up to 0.73 for Qwen-2.5-32B, 0.56 for Llama-3.1-70B).
- Clinical LLM evaluations should include tradeoff-conditioning tests, phrasing-sensitivity checks, and reporting of default operating points.
Strengths
Rigorous falsifiable framework localizing failure modes
The three necessary conditions plus paraphrase-invariance constraint (Section 2 Desiderata), operationalized via mirror, coherence, differential/common-mode decomposition, split-half reliability ceilings (Appendix C.4.1), and cross-side transfer (C.4.3), cleanly separate mirrored cost encoding from shared numeral-magnitude response.
Same-forward-pass behavioral + activation extraction
Behavioral readout and residual-stream activations come from the same forward pass (Appendix B.3); probes are parameter-free difference-of-means with patient-stratified 5-fold CV, avoiding probe-overfitting objections.
Honest, fully tabulated negative result
Risk recovery calibrated against conventional ML
Actionable clinical evaluation recommendations
Weaknesses
Single small binary testbed limits external validity
No uncertainty quantification on mirror contrasts
Preprocessing and reference baselines undisclosed
Probing is observational and single-method
Hint condition entangles instruction and cost
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether open-weight LLMs internalize the decision-theoretic cost structure of clinical classification. The authors formalize three necessary conditions - representation (cost direction/magnitude recoverable from activations), decision-layer use (cost shifts the output), and cost-sensitivity (directional and magnitude) - plus a paraphrase-invariance constraint (Section 2, Desiderata). The headline claims: (1) patient risk (probe AUC 0.81-0.83, Appendix D Table 1) and cost direction are linearly recoverable in all four models; (2) only 2 of 12 model-phrasing cells pass both mirror and coherence tests; (3) magnitude responses are predominantly direction-agnostic and cost directions never invert across sides. These are clearly scoped, falsifiable claims, and the evidence is largely proportionate to them.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
The paper's central finding is that four open-weight LLMs (Qwen-2.5-7B/32B and Llama-3.1-8B/70B) internally encode patient risk and the direction of false-negative vs false-positive cost asymmetry in linearly recoverable ways, but they rarely use this encoding when producing decisions. Out of 12 model-phrasing combinations, only 2 behaved cost-correctly under both mirror and coherence tests, and cost-magnitude responses were mostly direction-agnostic.
Which models handled cost tradeoffs correctly?
Llama-3.1-70B and Qwen-2.5-32B passed the behavioral mirror and coherence tests, and only with the original cost phrasing (not two paraphrases). Both smaller models (Qwen-2.5-7B, Llama-3.1-8B) failed under every phrasing, and both larger models failed under both paraphrases.
What data was used in the study?
The public Pima Indians Diabetes dataset from the UCI repository: 768 female patients aged 21+, 8 numeric predictors (glucose, BMI, age, etc.), and a binary diabetes label. Each patient was serialized as structured text and the models answered HIGH or LOW.
Why does this matter for clinical AI deployment?
Any model emitting a HIGH/LOW recommendation carries an implicit operating point. If a model ignores stated error costs, institutions cannot align it with their own risk tolerance, and single-threshold benchmarks cannot detect the gap. The authors argue clinical evaluations should add tradeoff-conditioning tests, phrasing-sensitivity checks, and default operating-point reporting alongside AUROC-style metrics.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures (Figures 1-4, Appendix D Figures 1-8) are images; their plotted content could not be inspected, so figure-based judgments rest on captions and the fully tabulated appendix tables.
- Appendix B/C formulas show minor extraction artifacts; mathematical substance was readable.
- Code repository is only promised upon acceptance, so implementation-level reproducibility could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.