arXiv paper2026-09-11· medical AI evaluation (clinical informatics)· method paper + benchmark

Scaling Clinical Judgment to Evaluate Medical AI

Thomas A. Buckley, Zahir Kanjee, Peter G. Brodeur, Byron Crowe, Anthony M. Pettinato, Aashna P. Shah, Adrian D. Haimovich, Liam G. McCoy, et al.

View original paper ↗arXiv:2609.12822

cs.AI

Quick verdict

LoRA-tuned 32B judge reaches physician-level agreement on 3/5 tasks and reproduces 5 landmark studies; valuable benchmark, but 'across tasks' claim overshoots.

Summary

The paper introduces PrecepTron, a 32-billion-parameter model (Qwen3-32B base) fine-tuned with LoRA on a small number of physician-scored examples to grade open-ended clinical AI responses like a physician panel, and releases GRAND-ROUNDS, a benchmark of 9,217 physician scores across six clinical tasks from seven studies by 11 graders. Zero-shot frontier judges (GPT-5, Claude Opus 4.6, Gemini 3.1 Pro) disagree with physicians and with each other. Trained on only 2-46 cases per task, PrecepTron reaches physician-level agreement on three of five tasks (e.g., 92% vs 92% on NEJM CPCs) but falls short on Landmark Diagnostic (61% vs 67%) and Grey Matters management (80% vs 95%) cases, and outperforms few-shot and GEPA prompt calibration. It reproduces the headline findings of five published studies without new human grading, and a token-by-token experiment shows GPT-5 solving 42% of CPC cases from 50 or fewer tokens under top-10-differential credit. Main limitations: the judge inherits each panel's scoring biases, the five-study reproduction relies on judges trained on the same physician panels, two tasks have very few training cases, and kappa declined on one task despite higher accuracy.

Review complete

Assessment at a glance

medical AI evaluation (clinical informatics)

risk flags identified

● high 0

● medium 3

● low 2

Reproducibility readinesshigh

Code & data availableyes

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducibleNeeds GPU

4 strengths · 4 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Zero-shot frontier LLM judges given only published rubrics disagree with physician scores and with each other (mean score spreads up to 17 points on Landmark Diagnostic Cases).
LoRA fine-tuning of Qwen3-32B on 2-46 physician-scored cases per task reaches physician inter-rater agreement on three of five tasks and outperforms few-shot and GEPA prompt calibration.
PrecepTron reproduces the headline findings of five published studies (JAMA, Science, Nature Medicine lineage) without any new human grading.
GPT-5 solves 42% of NEJM CPC cases with 50 or fewer tokens under top-10 differential credit and reaches 80% of rubric maximum on all four Landmark cases within 50 tokens, implying current benchmarks reward early pattern matching.

Strengths

Large, openly released physician-annotated benchmark
GRAND-ROUNDS: 9,217 physician scores over 5,250 response-rubric entries from 11 graders and seven studies; case text for Landmark Diagnostic and Grey Matters released for the first time on Hugging Face (Methods, Code and Data Availability).
Calibration recipe validated by real ablations, not anecdotes
Balanced resampling prevents majority collapse (NEJM Healer QWK 0.72 to 0.51 without it, Supplementary Table 1); retraining the Landmark adapter on human-only responses (Supplementary Fig. 3) tests and rules out AI-style shortcutting; three-seed variance reported.
Right yardstick and useful negative results on prompting
Downstream science enabled by cheap grading

Weaknesses

Abstract overstates the headline result
'Reproduction' of five studies is partly circular
Thin case-level diversity on two tasks
Lenient primary metric and unexplained metric divergence

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

This method-and-benchmark paper targets the reliability bottleneck of "LLM-as-a-judge" in medicine. It contributes (1) GRAND-ROUNDS, a released benchmark of 9,217 physician scores over 5,250 response-rubric entries, 11 graders, seven source studies, six clinical tasks (five used in judge experiments; the sixth, CPC testing plan, excluded for heavy score skew); and (2) PrecepTron-32B, a LoRA fine-tune of Qwen3-32B with per-rubric adapters, deployable on one 80GB GPU. Four claims are tested: frontier judges disagree with physicians and with each other; small-scale fine-tuning reaches physician-level agreement; five published studies are reproduced without new human grading; and token-by-token case reveal shows near-immediate performance saturation.

Methodology Assessment

Design choices are mostly sound and unusually transparent. Train/test splits are by case, correctly preventing memorization of a case's responses. The yardstick — judge agreement compared against physician-physician inter-rater agreement rather than raw accuracy — is the right comparison, reported per task (Table 1, Fig. 3). Two ablations have real teeth: removing balanced resampling collapses NEJM Healer quadratic-weighted kappa from 0.72 to 0.51 despite similar accuracy (majority-class collapse, Supplementary Table 1), and retraining the Landmark adapter on human-on

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 4 strengths & 4 weaknesses, analyzed in detail
5 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is PrecepTron?

A 32-billion-parameter language model (based on Qwen3-32B) fine-tuned with LoRA on a small number of physician-scored clinical examples so it can grade open-ended medical AI responses like a physician panel; it runs locally on a single 80GB GPU.

What is the GRAND-ROUNDS benchmark?

A publicly released dataset of 9,217 physician scores covering 5,250 clinical AI responses across six tasks (e.g., NEJM CPC diagnosis, diagnostic reasoning, management reasoning, consultation notes, ER triage), assembled from seven published studies with 11 physician graders; available on Hugging Face.

Did PrecepTron match human physicians?

On the paper's accuracy metric it matched or exceeded physician inter-rater agreement on three of five tasks (e.g., 92% vs 92% on NEJM CPCs), but fell short on Landmark Diagnostic Cases (61% vs 67%) and Grey Matters management cases (80% vs 95%).

What did the token-by-token experiment find?

When shown only the first 10-50 tokens of a case, GPT-5 still solved 42% of NEJM CPC cases under lenient top-10 differential credit and reached 80% of the rubric maximum on all Landmark diagnostic cases within 50 tokens, suggesting current benchmarks reward early pattern matching rather than reasoning over a full case.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

Figure images are not part of the text extraction; Fig. 5 saturation values cited in this review are those quoted in the text and captions, not read from the curves.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper