VeriDx: Earning the Right to Diagnose with Disease-Centric Verification
cs.AI · cs.CL
Quick verdict
VeriDx checks LLM diagnostic reasoning against disease-profile obligations; promising cross-model gains, but LLM-judge circularity and no significance tests.
Summary
The paper introduces VeriDx, a disease-centric verification framework for medical LLM reasoning. Premise: once a model proposes a disease hypothesis it incurs clinical obligations — confirming the core phenotype, ruling out important mimics, ordering discriminative tests, resolving contradictions, and justifying closure — which answer-centric benchmarks ignore. VeriDx links free-form reasoning trajectories to structured disease profiles (core phenotype, supportive and opposing findings, mimics, critical examinations, closure conditions) derived from 2,563 respiratory guidelines (~3.93 GB, physician-reviewed), labels each obligation satisfied, unresolved, or violated, and applies bounded patch-style revisions. Evaluation uses a separately developed longitudinal pulmonary benchmark: 188 expert-annotated real-world cases, 1,455 images, three-stage progressive disclosure, and 1,128 task instances scored across six dimensions (D1–D6). Frontier models drop from complete-information settings (oracle-intermediate accuracy up to 0.89 for GPT-5.5) to stepwise reasoning (0.29–0.38 for most frontier models; Table 1). In the multimodal setting, VeriDx raises GPT-5.5's dynamic-revision score from 0.297 to 0.682 and investigation planning from 0.241 to 0.638 (Table 3), with consistent patterns across GPT, Gemini, and Claude families; specialist gold profiles only marginally outperform LLM-extracted ones (Table 2). Limitations: extraction and judging are both LLM-based, the verifier edits the very artifacts later scored, no significance testing, safety-event rates are defined but unreported, and validation is respiratory-only.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Current medical LLM benchmarks evaluate cases, steps, or final answers but do not verify the clinical obligations induced by each disease hypothesis the model raises.
- VeriDx converts diffuse reasoning defects into machine-checkable violations of disease-specific commitments by linking free-form trajectories to guideline-derived disease profiles.
- Frontier LLMs degrade sharply from complete-information to sequential staged diagnosis (GPT-5.5: oracle-intermediate accuracy 0.89 vs stepwise accuracy 0.30).
- Verification-guided bounded revision improves process dimensions D1–D6 consistently across GPT, Gemini, and Claude families, with DDx-verification and exam-verification contributing complementary gains.
- Many diagnostic errors are broken commitments made earlier in the reasoning process rather than isolated final-answer mistakes.
Strengths
Novel evaluation unit: hypothesis-induced obligations
Reframes audit from case/step/answer level to per-hypothesis commitments (Sec. 1, 4.2–4.4), grounded in the diagnostic-error literature (Croskerry 2002; Graber 2005; Schiff 2009); six violation types (missing critical test, unresolved mimic, ignored contradiction, unsupported claim, premature closure, unsafe treatment) are clinically interpretable and machine-checkable.
Substantial new resource with evidence automation works
Profile library extracted from 2,563 respiratory guidelines (~3.93 GB) with physician review (Sec. 3, App. C); Table 2 shows specialist gold profiles beat LLM-extracted ones only marginally (D2 0.71 vs 0.68, D5 0.87 vs 0.84), supporting scalable profile construction.
Cross-model consistency of the intervention pattern
Carefully engineered evaluation protocol
Candid limitations and ethics documentation
Weaknesses
Verifier–judge circularity threatens the headline gains
No inferential statistics anywhere
Unexplained Table 1 anomaly
Safety events motivated but never quantified
No external verification baselines
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
VeriDx (arXiv 2609.14018) reframes clinical LLM evaluation around "hypothesis-induced obligations": once a model proposes a disease, it must confirm the core phenotype, rule out important mimics, order discriminative tests, resolve contradictions, and justify closure. The authors contribute (1) a disease-profile library auto-extracted from 2,563 respiratory guidelines (~3.93 GB, physician-reviewed), with fields for core phenotype, supportive/opposing findings, mimics, missing information, critical examinations, and closure conditions (§4.3); and (2) an evaluation on a separately developed longitudinal pulmonary benchmark — 188 expert-annotated real-world cases, 1,455 images, three-stage progressive disclosure, 1,128 task instances scored across six dimensions D1–D6 (§3, Appendix C). Claims: existing benchmarks miss these obligations; frontier models degrade sharply in sequential settings; verifier-guided revision improves D1–D6 and transfers across model families.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is VeriDx?
VeriDx is a verification framework for medical LLM reasoning. It checks whether a model's diagnostic reasoning satisfies the clinical obligations created by each disease hypothesis it proposes — confirming the phenotype, ruling out important mimics, ordering critical tests, resolving contradictions, and justifying closure — using structured disease profiles built automatically from 2,563 respiratory guidelines and reviewed by physicians. Each obligation is labeled satisfied, unresolved, or violated.
Does VeriDx improve final diagnostic accuracy?
Its largest gains are on process quality, not the final label. In the paper's multimodal experiments, GPT-5.5's dynamic differential-revision score roughly doubles (0.297→0.682) and investigation planning more than doubles (0.241→0.638); final-diagnostic precision (D4) rises more modestly from 0.601 to 0.711. No statistical significance tests are reported, and validation covers only respiratory cases.
What data does the VeriDx paper use?
Two resources: (1) a disease-profile library extracted from 2,563 respiratory guidelines and reference documents (~3.93 GB), reviewed by physicians; (2) a longitudinal benchmark of 188 de-identified real-world respiratory cases with 1,455 medical images, disclosed in three stages, yielding 1,128 scored task instances across six dimensions (D1–D6). Ethics approval: CAMS&PUMC-IEC-2024-067.
Is the VeriDx code or dataset publicly available?
The paper does not state any code release. It says the clinical benchmark is not intended for unrestricted public release unless data-governance and ethics requirements permit, and access to the disease-profile library is not clearly committed. Judge and extractor model identities are also not disclosed, which limits independent reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–3 are schematics/plots; Fig. 3 bar values were cross-checked against Table 3 numbers, but figure-level detail is not independently verifiable from extracted text.
- Safety-event rates (B.10) are defined but no safety-event results appear in the extracted text — possibly truncated tables or omitted results.
- Judge and extractor model identities are not stated in the provided text; Appendix B prompts are only partially included.
- The benchmark's provenance beyond the stated IRB protocol (CAMS&PUMC-IEC-2024-067) and single-setting case mix cannot be verified from the text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.