Large Language Model Agents for Evidence Based Genetic Disease Severity Classification
Tohid Ghasemnejad, Ahmadreza Argha, Mark Grosser, John Wang, Min Yang, Thantrira Porntaveetus, Tony Roscioli, Nigel H. Lovell, et al.
q-bio.GN · cs.AI · cs.CL
Quick verdict
ReAct+RAG LLM agent classifies 10,211 HPO terms at 93.6% expert-label accuracy; promising, but small selection set and ad hoc gene-level thresholds.
Summary
The paper presents an autonomous LLM agent combining ReAct reasoning with retrieval-augmented generation to classify the severity of 10,211 Human Phenotype Ontology (HPO) terms under ACMG-endorsed severity tiers and ACOG quality-of-life criteria. Grounded in PubMed evidence, GPT-4o (temperature 0.2, chosen from six models using 20 terms) reached 93.55% accuracy (MCC 0.9237) on a 941-term expert-annotated validation set matched to the population distribution, with 82.6-91.4% of claims supported by direct evidence or valid inference and 11.4% flagged for human review. Aggregating 8,738 gene-disease pairs yielded 79.5% Profound/Severe, and external comparison showed 95.2% concordance with the Mackenzie's Mission gene list and 99.4% with ACMG secondary findings genes. Main limitations: model selection rested on only 20 terms; the gene-level aggregation thresholds lack sensitivity analysis; no non-agent baseline was tested at validation scale; the production prompt is available only under a material transfer agreement; the same chi-square statistic is quoted for two different comparisons; and the HPO release date is inconsistent between January 2024 and January 2025.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 4
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A ReAct+RAG LLM agent classifies 10,211 HPO terms into ACMG-derived severity tiers with 93.55% accuracy (MCC 0.9237) against blinded expert annotations on a 941-term validation set.
- 82.6-91.4% of the agent's claims are supported by direct evidence or valid inference, with an independent verification agent flagging 11.4% of classifications for human review.
- Aggregation over 8,738 gene-disease pairs yields 79.5% Profound/Severe genes, including 3,283 of 3,873 autosomal recessive pairs, with 95.2% concordance to the Mackenzie's Mission gene list.
- The pipeline costs about $0.102 per phenotype and enables standardized, evidence-linked carrier and newborn screening panel design.
Strengths
Massive, evidence-grounded classification at low unit cost
10,211 HPO terms classified with mandatory citation of retrieved sources; mean $0.102/phenotype and 6.73 reasoning steps; an independent SourceCheckup-derived verifier reports 82.6-91.4% claim support across tiers and flags 11.4% for review (Figure 4, Operational Efficiency).
Well-designed primary validation
The 941-term validation set was PSI-matched to the population distribution (PSI 0.0288), annotated blindly by two geneticists per term with structured consensus; results reported as MCC 0.9237, kappa 0.9223, bootstrap 95% CI 91.9-95.2%, ordinal QWK 0.9164, within-1-tier 98.21%.
Layered external validation with discrepancy analysis
Transparency artifacts and cost disclosure
Honest limitation reporting
Weaknesses
Model selection on only 20 terms
No baseline at validation scale
Ad hoc gene-level aggregation
Statistical reporting inconsistencies
Reproducibility caveats despite artifacts
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This method paper targets a documented bottleneck in reproductive carrier and newborn screening panel design: severity of genetic conditions is judged subjectively, and commercial panels range from 41 to 1,792 conditions with minimal overlap (refs 16, 23). The authors build an autonomous ReAct+RAG agent that classifies all 10,211 HPO terms carrying gene-phenotype associations into Lazarin-derived severity tiers (plus three non-tiered labels), assigns PedsQL-based quality-of-life domains, age of onset, and management profiles, then aggregates phenotype tiers to gene-level Profound/Severe/Moderate/Mild over 8,738 gene-disease pairs. Central claims: 93.55% phenotype-level accuracy (MCC 0.9237) against expert labels; 82.6-91.4% of claims evidence-supported with 11.4% flagged for review; 95.2% concordance with Mackenzie's Mission; $0.102 per classification.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What accuracy did the AI agent achieve for genetic disease severity classification?
On a 941-term expert-annotated validation set of Human Phenotype Ontology terms, the ReAct+RAG agent built on GPT-4o reached 93.55% accuracy, MCC 0.9237, and 98.21% within-one-tier accuracy, with 82.6-91.4% of its claims supported by direct literature evidence or valid inference.
How many HPO terms and genes did the system classify?
It classified all 10,211 HPO phenotype terms with documented gene-phenotype associations and aggregated severity across 8,738 gene-disease pairs; 79.5% of pairs were Profound or Severe, and 3,283 of 3,873 autosomal recessive pairs fell in those combined categories.
Is the code and data for this HPO severity classifier publicly available?
Partly: code is on GitHub (T0hid/hpo-classification-agent) with a Code Ocean capsule and a Zenodo deposit containing ground truth, hyperparameter results and the gene-level severity table; however, the full production prompt is provided only under a material transfer agreement with UNSW Sydney.
What are the main limitations of the study?
Key limitations are model selection on only 20 terms, unvalidated gene-aggregation thresholds, absence of a non-agent baseline at validation scale, panel concordance that mixes severity with carrier-frequency criteria, a duplicated chi-square statistic in the QoL analyses, and inconsistent HPO release dates (January 2024 vs 2025).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- All figures and supplementary tables (S1-S4, Figure S1-S3) are referenced but not analyzable from the extracted text; per-category detail relies on the authors' reported numbers.
- The input HPO release is stated inconsistently (January 2025 in Methods, January 2024 in Data Availability), so the exact 10,211-term set cannot be verified from the text alone.
- The production prompt and full verification outputs are not in the manuscript (MTA-gated), so prompt-level fidelity of the pipeline cannot be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.