CLEAR: Cross-Source Evidence Adjudication for Large Language Models in Medicine
Shuai Wang, Yize Zhao, Qingyu Chen
cs.AI · cs.CL
Quick verdict
Agentic cross-source adjudication for medical LLMs: large gains on weak backbones (up to +13.7pp), yet no significance tests or naive multi-source baseline.
Summary
The paper introduces CLEAR, a training-free agentic framework that adjudicates between three knowledge sources for medical LLMs: parametric knowledge, a locally curated corpus (MedCorp with BM25), and dynamically retrieved web evidence. Three pathway-specific candidates are generated independently; an aggregation verifier builds an evidence bundle with provenance and source-quality labels and routes to consensus, keep-parametric, keep-local, accept-dynamic, or needs-more-search. An override guard prevents weak web evidence from overturning parametric-local consensus, while a challenge audit lets credible new evidence revise outdated consensus; unresolved conflicts trigger bounded targeted follow-up search. Evaluated with o3-mini and Qwen-3.5-9B on ten benchmarks, CLEAR is competitive where baselines are strong (MedQA, PubMedQA) and gains most where they are weak: with Qwen-3.5-9B, best configurations improve MedQA by 9.74, NEJM-QA by 11.30, and MedRBench by 13.69 percentage points over the strongest baseline. On HealthBench with o3-mini, default CLEAR underperforms Direct inference (0.2422 vs 0.3334), which the authors attribute to an already-strong parametric baseline. Ablations show the full framework beats any single source (88.34 vs 87.48/86.99/83.20 aggregate) and follow-up search adds 0.91 points (12 of 123 triggered searches accepted). A manual audit found no direct benchmark-item leakage, though upstream source documents are retrieved for 42-78% of PubMedQA/MedRBench items. Main limitations: no statistical significance testing, no naive multi-source (concatenation) baseline to isolate adjudication benefits, and no cost analysis.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Explicit cross-source adjudication (aggregation verifier + override guard + challenge audit + bounded follow-up search) beats direct inference and single-source RAG where parametric knowledge is weak: +9.74pp MedQA, +11.30pp NEJM-QA, +13.69pp MedRBench with Qwen-3.5-9B over the strongest baselines.
- Retrieved evidence should not be trusted by default: dynamic-only retrieval degrades o3-mini accuracy on MedQA (87.82 vs 92.38 Direct) and NEJM-QA (78.32 vs 84.12).
- The full three-source framework outperforms any individual evidence source in aggregate (88.34 vs 87.48/86.99/83.20 on o3-mini MCQs, Table 4).
- Targeted follow-up search is selective and beneficial: 12 accepted revisions out of 123 triggers yield +0.91pp aggregate with Qwen-3.5-9B (Table 5).
- Manual audit found zero direct benchmark-item leakage in retrieval traces; upstream source-document retrieval (42-78% on PubMedQA/MedRBench) reflects effective retrieval rather than contamination.
Strengths
Honest reporting of unfavorable results
HealthBench with o3-mini: default CLEAR scores 0.2422 vs Direct 0.3334 (§3.2), and PubMedQA never beats MedRAG-BM25 (82.00); both are reported, tabulated, and analyzed rather than hidden (§4.1).
Exemplary retrieval-leakage audit
§3.4 and Tables 6-7: every retrieval trace manually audited; zero direct benchmark-item leakage across all benchmarks/backbones, with upstream-source retrieval transparently quantified (PubMedQA 42.4/42.6%, MedRBench 60.4-77.6%) and the same checks applied to MedRAG baselines.
Mechanism-level ablations with usage statistics
Controlled, matched-backbone design across 10 benchmarks
Conservative two-sided adjudication design
Weaknesses
No inferential statistics anywhere
Missing naive multi-source baseline
Internal inconsistency in Table 5
Unreported cost and unspecified components
Cross-model confound in the dynamic pathway
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CLEAR (Sections 1-2) addresses a real, well-motivated problem: LLM parametric knowledge is frozen while medical evidence evolves, and naive RAG can inject irrelevant or conflicting evidence. The framework generates three independent candidates (parametric; local MedCorp+BM25; dynamic web search), aggregates them into an evidence bundle with provenance and categorical source-quality labels, routes via an aggregation verifier (consensus / keep-parametric / keep-local / accept-dynamic / needs-more-search), and adjudicates conflicts through an override guard and challenge audit, with bounded targeted follow-up search (Section 2.3). Claims in the abstract are mostly proportionate ("competitive performance", "largest gains where baselines weaker"). The headline numbers all come from the Qwen-3.5-9B backbone: +9.74pp MedQA (83.58 dual-models vs 73.84 MedRAG-BM25), +11.30pp NEJM-QA (78.17 QPIS vs 66.87), +13.69pp MedRBench (74.74 vs 61.05 Direct, which is itself the strongest baseline there). With o3-mini, MedQA/PubMedQA/NEJM differences are marginal (92.69/81.60/85.04 vs Direct 92.38/79.40/84.12), and HealthBench degrades (0.2422 vs 0.3334).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CLEAR and what problem does it solve?
CLEAR is a training-free agentic framework for medical LLMs. It generates candidate answers independently from three knowledge sources — the model's internal parametric knowledge, a locally curated corpus, and live web search — then explicitly adjudicates agreement and conflict between them (via an aggregation verifier, an override guard, a challenge audit, and bounded follow-up search) before answering. The motivation is that retrieved evidence can be irrelevant, outdated, or contradictory, so blindly trusting it can hurt accuracy.
How much does CLEAR improve performance over standard RAG?
It depends on the backbone. With the open-weight Qwen-3.5-9B, the best CLEAR configurations beat the strongest evaluated baselines by 9.74 points on MedQA, 11.30 on NEJM-QA, and 13.69 on MedRBench. With o3-mini, gains are marginal on MedQA (+0.31) and PubMedQA, but MedRBench improves from 71.30 to 82.38. On HealthBench with o3-mini, the default CLEAR configuration actually underperforms plain direct inference (0.2422 vs 0.3334).
Was the evaluation contaminated by benchmark leakage?
The authors manually audited every retrieval trace: no retrieved document contained the benchmark question or answer itself. However, upstream source articles were retrieved for 42.4-42.6% of PubMedQA and 60-78% of MedRBench items, and possible memorization of public benchmarks during backbone pretraining cannot be ruled out — both are acknowledged limitations.
Is the code and data available?
The paper states the implementation code will be released at github.com/Yale-BIDS-Chen-Lab/Agentic-Adjudication-for-Medical-Reasoning, and all ten benchmarks plus the MedCorp corpus are public. Note that the dynamic pathway relies on GPT-4o's web search (Tavily in one subset experiment), which is nondeterministic, so exact published numbers may not be reproducible later.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are not visually analyzable; Figure 2/3 values are embedded as chart text and were cross-checked against Tables 1-2 (they agree).
- The 'Overall' column of Table 1 appears dropped in extraction; NEJM figures were verified against Table 2 and the numbers quoted in §3.1/§4.1.
- Appendix A prompting details and some decision-rule specifics are referenced but not included in the extracted text.
- The promised code repository could not be verified as actually released (paper says 'will be available').
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.