Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection
cs.CL
Quick verdict
Claims single-pass medical hallucination benchmarks undercount errors; multi-pass annotation study supports this, but constructed references are circular.
Summary
The authors present MDHJudgments, 541 English chatbot responses to cystic-fibrosis and pediatric infectious-disease questions, annotated through a multi-pass pipeline: first-pass (FP) annotators from three backgrounds (8 medical experts, 6 students, 6 AI researchers), a GPT-5 LLM-as-a-Judge (LaJ), and two adjudication tracks (Prolific-certified medical experts and fact checkers) applied to an 834-section disagreement pool. FP annotators reach only about 20-21% recall against adjudicated references (precision 75-87%), missing errors adjudicators validate; the LaJ achieves 71-80% recall but misses errors annotators catch; and the two adjudication styles themselves disagree (xRR 0.18; medical-expert Krippendorff's alpha as low as 0.05). Against the FP-consensus reference containing only six positive sections, the LaJ shows 83% recall but 2% precision. Transplanting the pipeline to MedExpert, 78% of LaJ-flagged candidates absent from that benchmark's labels, within a 124-section disagreement subset, were adjudicated as errors. The authors conclude that single-pass, singly annotated medical hallucination benchmarks likely undercount factual errors, while inferences remain sensitive to annotator judgment and evidence. Limitations: references are partly built from the scored detectors' own disagreements (acknowledged in section 4.3), adjudication covers only a bias-selected pool, the MedExpert case study uses a broader factuality definition than the original guidelines, and the LaJ is a single prompt and model.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- First-pass annotators miss most factual errors that adjudicators validate: individual FP recall is about 20-21% against both adjudicated references (Table 5).
- LLM-as-a-Judge improves candidate discovery (recall 71-80%) but complements rather than replaces human annotators, since it misses errors annotators catch (Tables 4 and 5).
- Adjudicators themselves disagree substantially (ME-FC xRR 0.18; medical-expert alpha as low as 0.05), so adjudication provides a lens rather than canonical ground truth (Tables 3-4).
- The under-annotation pattern generalizes: in a 124-section disagreement subset of MedExpert, 78% of LaJ-flagged candidates absent from its labels were adjudicated as errors (Section 5, Table 10).
- Holistic response-level accuracy correlates only moderately with section-level error frequency (r = -0.39), so response-level judgments can mask detailed factual errors (Section 4.5).
Strengths
Multi-perspective design isolates complementary failure modes
The pipeline jointly measures what each signal uniquely contributes: Table 2 shows 389 sections found only by the LaJ versus 8 found only by FP annotators in the 834-section pool, with 402 FP-FP disagreements, directly instantiating the TREC-style pooled-adjudication framing of section 3.4.
Unusually honest reliability reporting
Tables 3 and 4 report Krippendorff's alpha as low as 0.05 for medical-expert adjudicators (CI -0.12 to 0.27) and xRR 0.18 between the two adjudication styles, and the authors explicitly conclude adjudication should be treated as an additional lens rather than a canonical source of truth (section 4.1).
Sensitivity analysis guards the headline finding against pooling artifacts
External evidence that under-annotation generalizes
Release quality supports protocol replication
Weaknesses
Recall/precision estimates are conditional, not population-level
The adjudicator anchor is statistically fragile
The FP:AG contrast rests on six positive sections
MedExpert case study partially conflates definition mismatch with missed errors
Narrow, deliberately unbalanced generation design
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.03953, USC Information Sciences Institute) asks how annotation procedure shapes medical hallucination benchmarks. It contributes MDHJudgments: 541 English chatbot responses to cystic-fibrosis and pediatric infectious-disease questions, annotated by first-pass (FP) annotators from three backgrounds (8 medical experts, 6 students, 6 AI researchers per Table 13), a GPT-5 LLM-as-a-Judge (LaJ), and two adjudication tracks recruited through Prolific (21 medical experts at $130/hr; 156 fact checkers at $30/hr). Central claims: (1) single-pass FP annotation has very low recall (20-21%) against adjudicated references; (2) the LaJ raises recall to 71-80% but misses errors humans catch; (3) adjudicators disagree with each other, so adjudication is a lens, not ground truth; (4) the MedExpert case study (78% validation of 65 LaJ-flagged candidates absent from its labels, within a 124-section disagreement subset) suggests the pattern generalizes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the MDHJudgments dataset?
A dataset of 541 English chatbot responses to medical questions (cystic fibrosis and pediatric infectious diseases), annotated at both response and section level by first-pass annotators, a GPT-5 LLM-as-a-Judge, and two adjudication tracks (medical experts and fact checkers recruited via Prolific). Machine-readable data and tools are released at github.com/isi-vista/mdhjudgments.
Do single-pass hallucination benchmarks undercount errors?
This paper argues they likely do in the settings examined: individual first-pass annotators caught only about 20% of the factual errors later validated by adjudicators, and a case study of the MedExpert benchmark adjudicated 78% of LLM-flagged candidates missing from its labels as real errors. The authors caution that these estimates are conditional on their adjudication setup and do not claim hallucination benchmarks are uninformative.
Can an LLM-as-a-Judge replace human annotation for medical factuality?
Not in this paper's findings: the GPT-5-based judge improved recall (71-80% versus about 20% for single first-pass annotators) but still missed errors humans caught, and scored only 2% precision against the human-consensus reference (which contained just six error sections), so it is positioned as a complementary discovery tool that feeds adjudication, not a replacement.
How reliable were the expert adjudicators?
Moderately at best: the medical-expert and fact-checker styles agreed 73% of the time with cross-replication reliability of 0.18, and medical-expert Krippendorff's alpha on the small doubly adjudicated subset was 0.05 (CI -0.12 to 0.27). The paper therefore treats adjudication as an additional lens on the data rather than a canonical source of truth.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's heatmap cell values are garbled by PDF extraction (axis labels and counts interleave); the review relies on the prose-reported correlation (-0.39) and section descriptions rather than reconstructing cell counts from the figure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.