Learning to Fuse LLMs with Ontology Rankers for Rare-Disease Diagnosis
Zhaoyang Jiang, Zhizhong Fu, Yunsoo Kim, Zicheng Li, Xuanqi Peng, Fei Teng, Jiacong Mi, Honghan Wu
cs.CL
Quick verdict
Proposes leakage-corrected eval and behavior-gated LLM/ontology fusion for rare-disease diagnosis; rigorous, honest about limits, gains from low baselines.
Summary
The paper asks whether an LLM can improve an HPO-based ontology ranker for rare-disease diagnosis without sacrificing its inspectable evidence trail. It first identifies publication-source overlap: Phenopacket Store cases and HPOA annotations are frequently curated from the same paper. Removing source-exclusive gold-disease annotations (leave-one-publication-out) drops Phenomizer Recall@1 from 0.4481 to 0.1217 on 2,227 test cases (32.64 pp; LIRICAL falls similarly), with matched-deletion and profile-survival controls ruling out generic annotation loss. The authors then train a small shared scorer (3,121 parameters, 39 behavioral features per ranking) that assigns case-level weights before weighted reciprocal-rank fusion of the Phenomizer top-100 and the LLM's top-10, name-mapped to OMIM. Under a family-held-out protocol across eight open LLMs, fusion reaches macro Recall@1 0.2002 versus 0.1515 for CombMNZ, beating every component and baseline; a gate trained without DeepSeek-V4-Flash still lifts test-only Recall@1 from 0.1657 (LLM) and 0.1217 (Phenomizer) to 0.2176. On RAMEDIS, fusion gains 20.18 pp over Phenomizer yet is statistically indistinguishable from parameter-free RRF. 90.8% of correct fused diagnoses retain candidate-level ontology evidence. Limitations: low absolute accuracy, phenotype-only inputs, no direct Exomiser comparison, and transfer demonstrated only across the studied model families.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Publication-source overlap between Phenopacket Store and HPOA materially inflates ontology-ranker accuracy: removing source-exclusive gold-disease annotations cuts Phenomizer Recall@1 by 32.64 pp (0.4481→0.1217) and LIRICAL by 31.43 pp.
- A behavior-gated fusion of Phenomizer's top-100 with an LLM's top-10 reaches macro Recall@1 0.2002 vs 0.1515 for CombMNZ under backbone-family holdout across eight open LLMs, beating all components and baselines.
- The gate transfers to unseen model lineages without target labels: test-only DeepSeek-V4-Flash rises from 0.1657 (LLM alone) and 0.1217 (Phenomizer) to 0.2176.
- 90.8% of correct fused diagnoses retain candidate-level ontology evidence inside Phenomizer's stored top-100.
- The learned gate's advantage over parameter-free fusion disappears on sparse inputs: on RAMEDIS it is statistically indistinguishable from RRF (0.3573 vs 0.3562).
Strengths
Benchmark-leakage correction with unusually strong causal hygiene
Section 6.1: LOPO filtering cuts Phenomizer Recall@1 from 0.4481 to 0.1217 (−32.64 pp) and LIRICAL from 0.4868 to 0.1724; Table 2's matched-deletion control (−0.67 pp for equal matched deletions vs −22.95 pp for source deletions), profile-survival split (21.43 pp among 1,717 retained-profile cases), single-case-publication restriction, and dose-response analysis jointly rule out generic annotation loss.
Credible transfer protocol for the fusion gate
Backbone-family holdout with conservative lineage grouping (Qwen2.5 grouped with Qwen3-based HuatuoGPT), fixed labeled budgets, plus a test-only DeepSeek-V4-Flash arm (0.1217/0.1657 → 0.2176) that mirrors the practical deploy-a-new-model scenario without any target labels.
Thorough baseline and control coverage
Honest negative and boundary results
High reproducibility disclosure
Weaknesses
Low absolute accuracy under the corrected protocol
Learned gate's value is corpus-dependent and per-arm fragile
Headline RAMEDIS number rests on a very wide interval
Unexamined hyperparameter and residual-leakage questions
No Exomiser comparison
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes two linked contributions. First, a benchmark-validity result: Phenopacket Store cases and HPOA disease annotations frequently share source publications (publication-source overlap: 74.6% of 10,348 HPOA-represented cases; on average 33.6% of the gold profile rests solely on the case's paper, Table 10). A leave-one-publication-out (LOPO) filter removing source-exclusive gold-disease relations collapses Phenomizer Recall@1 from 0.4481 to 0.1217 on 2,227 test cases (−32.64 pp, Table 1), with the independent LIRICAL 2.4.1 executable replicating the drop (−31.43 pp) and matched-deletion plus profile-survival controls (Table 2) ruling out generic annotation loss. Second, a method: a 3,121-parameter shared scorer converts 39 behavioral features per ranking into case-level fusion weights, combining Phenomizer's top-100 with an LLM's top-10 (OMIM-mapped) via weighted reciprocal ranks. Headline claims: macro Recall@1 0.2002 vs 0.1515 for CombMNZ under backbone-family holdout across eight open LLMs (Table 3); transfer to unseen DeepSeek-V4-Flash (0.1657→0.2176); 90.8% of correct fused predictions retain candidate-level ontology evidence.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is publication-source overlap in rare-disease AI benchmarks?
It means the benchmark patient cases and the knowledge-base annotations used to score them are curated from the same published papers. The paper measures that 74.6% of Phenopacket Store cases have at least one gold-disease annotation in HPOA citing the case's own source publication, so the ontology ranker can match evidence from the very paper the test patient came from — a test-set leakage pathway at inference time.
How much did correcting this leakage change Phenomizer's reported accuracy?
After removing HPOA annotations supported only by the test case's publication, Phenomizer Recall@1 fell from 0.4481 to 0.1217 on the 2,227-case test set (−32.64 points). The independent LIRICAL tool fell similarly (0.4868→0.1724), and matched-deletion controls show the drop is not explained by generic annotation loss.
Does fusing an LLM with Phenomizer keep the diagnosis explainable?
Mostly yes: 90.8% of correct fused predictions still place the disease inside Phenomizer's top-100, where matched HPO terms and p-values remain available for clinician inspection. Predictions promoted from outside that prefix are less reliable (10.3% correct) and lack stored ontology evidence.
Can this fusion model be used with a new LLM without retraining?
That is the design goal: the gate reads only ranking behavior (list shape, agreement, ontology support), never model identity. Trained only on eight open LLMs with whole backbone families excluded, it lifted test-only DeepSeek-V4-Flash Recall@1 from 0.1657 to 0.2176 without any DeepSeek training labels.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-2 convey no machine-readable values beyond caption text; all numbers cited in this review come from tables and inline text.
- The linked code repository is anonymized (Anonymous-Awesome-Submissions/PhenoGate), so it could not be inspected for this review; availability is as stated by the authors.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.