DIASENTINEL: An Auditable Multi-Agent System for Guideline-Grounded Diabetes Risk Screening
Yung Wei Shueh, Zhi-Jie Chen, Chia-Hsuan Hsu, Hsin-Ling Hsu, Donghua Zhang, Chenwei Wu, Jun-En Ding, Tongze Zhang, et al.
cs.CL
Quick verdict
Well-engineered auditable LLM screening demo with honest reporting, but private data, no code, and XGBoost beating it on AUPRC cap real-world evidence.
Summary
DIASENTINEL is an on-premise, LangGraph-orchestrated multi-agent system, presented as a system demonstration, for one-year type 2 diabetes (T2DM) risk screening and guideline-grounded report generation from EHRs. A LoRA fine-tuned Qwen2.5-14B converts token-level log-probabilities into risk scores, Platt-scaled to match the ~6% observed incidence; deterministic agents extract clinical signals and longitudinal trends; bge-m3 dense retrieval fused with a cross-encoder via RRF grounds recommendations in 382 chunks of the ADA Standards of Care 2026; and a verification layer of four deterministic checks plus an LLM entailment judge flags errors in an append-only audit log without altering reports. On 4,982 held-out records the risk model attains AUROC 0.737 (95% CI 0.694-0.773), Brier 0.054, and AUPRC 0.158, with well-calibrated tiers (high tier: 0.115 predicted vs 0.117 observed); notably, XGBoost achieves higher AUPRC (0.211 vs 0.146) with overlapping AUROC confidence intervals. On a 50-question retrieval set (Cohen's kappa=0.76), RRF reaches Recall@5 0.745 and Chapter@5 0.939. Verification is evaluated only on 24 synthetic cases (100% sensitivity/specificity) and 40 entailment pairs (80%/100%). Main limitations: private single-hospital data, no public code, entirely synthetic verification evaluation, no prospective or clinician study, and unassessed subgroup fairness.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A LoRA fine-tuned Qwen2.5-14B with Platt scaling yields calibrated one-year T2DM risk probabilities (AUROC 0.737, 95% CI 0.694-0.773; Brier 0.054) whose triage tiers match observed incidence.
- RRF fusion of dense retrieval (bge-m3) and cross-encoder reranking outperforms either alone on the ADA guideline corpus (Recall@5 0.745 vs 0.704/0.697), mitigating the reranker's verbosity bias.
- A hybrid verification layer (four deterministic checks plus LLM entailment) detects all 12 injected errors with zero false alarms on 12 clean reports, and its annotate-only design preserves clinician authority.
- The system runs fully on-premise, keeping patient data inside the hospital while supporting real-time batch screening and cited, guideline-grounded report generation.
Strengths
Unusually honest self-assessment
The paper openly reports that XGBoost beats its LLM on AUPRC (0.211 vs 0.146, Table 3) with overlapping AUROC CIs, admits the verification evaluation 'does not establish real-world clinical reliability' (end of §4.4), and Table 7 candidly states 'most positive predictions were false positives'.
Rigorous, clinically meaningful calibration
Platt scaling (a=1.0049, b=-1.5693) corrects mean predicted risk from 0.213 to the observed 0.060 (Fig. 6), and - rare at demo venues - per-tier calibration is verified (Table 4: high tier 0.115 predicted vs 0.117 observed; medium 0.048 vs 0.052; low 0.024 vs 0.022).
Auditable-by-design architecture
Concrete retrieval ablation with error analysis
Reproducible judge behavior
Weaknesses
Verification evidence is entirely synthetic
Simpler baseline wins on the paper's own priority metric
Tiny, non-public evaluation artifacts
Operating points scattered across sections
No end-to-end report quality or prospective evaluation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a system demonstration paper (cs.CL) presenting DIASENTINEL, an on-premise, LangGraph-orchestrated multi-agent pipeline built with Far Eastern Memorial Hospital (Taiwan) for (i) batch one-year new-onset T2DM risk screening and (ii) on-demand, ADA-guideline-grounded patient report generation. Three central claims: (1) a calibrated LLM risk predictor (LoRA-tuned Qwen2.5-14B + Platt scaling) supporting clinically meaningful stratification; (2) a two-stage RRF retrieval pipeline over the ADA Standards of Care 2026 that mitigates reranker verbosity bias; (3) an in-the-loop verification layer (four deterministic checks plus LLM entailment) that audits reports without modifying them. Claims are scoped honestly - the paper repeatedly positions itself as a demonstration, not a clinical validation, and the limitations section is unusually candid.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is DIASENTINEL?
DIASENTINEL is a fully on-premise multi-agent system, published as an academic system demonstration, that screens hospital EHR patients for one-year type 2 diabetes risk and generates patient reports whose recommendations are cited to the ADA Standards of Care 2026. A verification layer flags possible errors in an audit log without modifying reports, and physicians can accept or override each flag.
How accurate is DIASENTINEL's diabetes risk prediction?
Its fine-tuned Qwen2.5-14B model achieves AUROC 0.737 (95% CI 0.694-0.773) and Brier 0.054 on 4,982 held-out records, and its high/medium/low risk tiers are calibrated so predicted probabilities match observed incidence (e.g., 0.115 vs 0.117 in the high tier). However, a standard XGBoost baseline achieves a higher AUPRC (0.211 vs 0.146) with overlapping AUROC confidence intervals.
Is DIASENTINEL's code or data publicly available?
No. The paper states the production system and source code are not released under a public software license. The online demo runs only on synthetic patient data, and the underlying EHR dataset is private, single-hospital data, so the reported numbers cannot be independently re-derived.
Has DIASENTINEL been validated with real clinicians or patients?
No prospective evaluation or clinician study is reported. The verification layer was tested only on 24 synthetically constructed error cases and 40 built claim-evidence pairs; the authors explicitly state this does not establish real-world clinical reliability, and subgroup fairness (e.g., by sex) remains unevaluated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-7 are present only as captions in the text extraction; their visual content (reliability diagram shape, dashboard layout, flagged-panel example) cannot be assessed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.