Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality
Jiaju Huang, Hao Yang, Xinyu Ma, Xinglong Liang, Kunyan Cai, Junqiang Ma, Shaobin Chen, Yue Sun, et al.
cs.CL · cs.AI
Quick verdict
Proprietary judge Jev tracks expert error counts at low cost; evaluation is careful and candid, but a closed model, small benchmarks, and no code limit trust.
Summary
The paper evaluates Jev, a proprietary 'System One' decision model from TypeSafe AI, as a low-cost judge of factual agreement between AI-generated and physician-written radiology reports. The pipeline decomposes both reports into atomic findings with a local Qwen3.5-9B model, asks one support question per statement against the opposing report, and aggregates contradiction and non-coverage bidirectionally. The single-question configuration (OneQ) reaches Kendall τb of 0.573 with expert total-error counts on RadEvalX (n=100) and 0.398 on RadEvalExpert (n=624), clearly outperforming a DeBERTa NLI judge under identical decomposition and aggregation. OneQ matches a seven-question configuration at 43–45% fewer judgment input tokens, costing under three US cents per hundred report pairs (judgment API only, excluding decomposition). On the controlled ReXErr benchmark, Jev detects false negation with AUROC 0.977 but is near-blind to typos and repetition. Notably, the authors report that local RadMatch agrees better on clinically significant errors, that a mere finding-count baseline reaches τb 0.29/0.26, and that raw probabilities need isotonic calibration (ECE 0.12→0.008). Limitations: a closed judge with unreleased analysis code, small expert benchmarks, no multiple-testing adjustment, and baseline judges rerun on a substitute local model.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A single support question per statement (OneQ) reaches Kendall τb 0.573 on RadEvalX and 0.398 on RadEvalExpert against expert total-error counts, outperforming an open DeBERTa NLI judge under identical decomposition and aggregation.
- One question per statement retains expert agreement close to the seven-question Full configuration while cutting judgment input tokens by 43–45%, at under three US cents per hundred report pairs for the judgment API alone.
- On controlled ReXErr errors, Jev detects false negation with AUROC 0.977 and factual errors broadly (0.9680 on the factual subset), but is largely insensitive to typos and repetition (AUROC 0.5617 and 0.4848).
- Local RadMatch agrees better with clinically significant errors in both expert datasets, so Jev is positioned as a cheap judgment component rather than a full evaluator.
- Benchmark agreement partly reflects report size and error definitions: a finding-count baseline reaches τb 0.288/0.261, and count aggregation (0.320) far outperforms soft F1 (0.111) on the shared RadEvalExpert subset.
Strengths
Properly matched judge comparison
The open NLI baseline shares the same L1 atomic decomposition and Eq. 2 aggregation as Jev, isolating the judge itself; OneQ beats it by +0.236 [0.131, 0.345] on RadEvalX and +0.075 [0.043, 0.106] on RadEvalExpert total errors (Section 5.1, Table 1).
Unusual candor about unfavorable results
The paper reports that NLI edges Jev over all ReXErr error types (AUROC 0.8969 vs 0.8896, Table 2), that Jev collapses on typos (AUROC 0.5617) and repetition (recall 0.0026), and that a content-blind finding-count baseline already reaches τb 0.288/0.261 (Section 5.5).
Honest treatment of the stronger competitor
Concrete cost and repeatability accounting
Calibration treated as a requirement, not an afterthought
Weaknesses
The judged artifact is a black box
No released code or aggregate results
Small benchmarks, unadjusted contrasts
Baseline comparability weakened by substitute backends
Headline cost excludes a mandatory GPU step
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether Jev (jev-1.13.0), a proprietary 'System One' decision model from TypeSafe AI that returns typed support/contradiction/not-addressed probabilities, can serve as a low-cost judge of factual agreement between generated and reference radiology reports. The protocol follows the established decompose-then-entail pattern (RadFact-style): L1 atomic decomposition with a local Qwen3.5-9B, one support question per statement against the opposing report (OneQ), and bidirectional soft discrepancy aggregation (Eq. 2). Central claims: (1) OneQ reaches Kendall τb 0.573 on RadEvalX (n=100) and 0.398 on RadEvalExpert (n=624) against expert total-error counts, beating an open DeBERTa NLI judge under identical decomposition and aggregation (+0.236 [0.131, 0.345] and +0.075 [0.043, 0.106]); (2) one question matches seven (Full) at 43–45% fewer judgment input tokens, under $0.03 per 100 report pairs; (3) false-negation AUROC 0.977 on ReXErr; (4) local RadMatch stays better on clinically significant errors, so Jev is a component, not a replacement.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Jev?
Jev (jev-1.13.0) is a proprietary 'System One' decision model sold by TypeSafe AI. Instead of generating free text, it returns typed answers with probabilities — for example, whether a report supports, contradicts, or does not address a given clinical statement. The paper tests whether this interface works as a cheap judge of radiology report factuality.
How well does Jev agree with radiologists when judging reports?
Its one-question configuration correlates with expert error counts at Kendall τb = 0.573 on RadEvalX (100 report pairs) and 0.398 on RadEvalExpert (624 pairs). On controlled injected errors it detects false negation with AUROC 0.977, but it barely notices typos or repeated sentences, which mainly affect writing quality rather than medical meaning.
How much does it cost to evaluate radiology reports with Jev?
At the documented price of $0.042 per million input tokens, the judgment API costs roughly 2.3–2.6 US cents per 100 report pairs. This excludes the local decomposition step, which requires running a separate language model (Qwen3.5-9B) on your own hardware.
Is the code or data for this paper available?
The evaluation datasets are public (RadEvalX and ReXErr via PhysioNet; RadEvalExpert and RaTE-Eval from their distributors), and the exact prompts are printed in the paper's appendix. However, the analysis scripts and aggregate results are stated to be 'not yet publicly available', and Jev itself is a closed commercial API, so the numbers cannot currently be independently verified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 and A1-A2 are available only as captions and in-text numbers; visual content (calibration curves, category-allocation heatmap) could not be independently inspected.
- The supplied metadata lists 8 authors, but the full text shows 9, including corresponding author Tao Tan; the metadata appears stale.
- Several comparators (CRIMSON, VERT, RadMatch, RadSEM, AtomiMed) are 2026 preprints that could not be verified offline.
- Text extraction is essentially complete through Appendix G, with only minor citation-formatting artifacts; confidence is correspondingly high.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.