REFINE: LLM Refinement over Budgeted Text-Attributed Graphs for Personalized Medical Concept Representation
Mohsen Nayebi Kerdabadi, Arya Hadizadeh Moghaddam, Dongjie Wang, Zijun Yao
cs.LG · cs.AI
Quick verdict
RL-budgeted patient-specific KG context + frozen-LLM soft prompts for EHR codes; MIMIC gains look solid, but no fold variance or significance tests.
Summary
REFINE is a plug-in medical concept encoder for EHR prediction that personalizes how much knowledge-graph context each observed code receives. It builds a global text-attributed KG from co-occurrence and next-visit transition statistics with LLM-induced relations, extracts per-patient graphs, and trains an actor-critic policy that sequentially selects an expansion budget (depth and per-hop fanout) per code, penalized for graph cost. Each patient graph is processed in parallel by a heterogeneous GNN and a frozen Llama-3.2-1B refined via mixed hard/soft prompts (code and KG-context soft tokens), then gated-fused for next-visit diagnosis prediction. On MIMIC-III and MIMIC-IV (five folds), REFINE reports 47.41 and 50.24 AUPRC versus 45.35 and 48.28 for the strongest prior baseline (MedCo), improves five backbones as a plug-in, and ablations show the personalized GNN matters most (50.24 to 47.35 on MIMIC-IV). Main limitations: results are fold means with no standard deviations or significance tests, several ablation deltas are small (e.g., 49.72 without KG soft tokens), the strongest baselines and the reused KG pipeline come from the same group, and only very small frozen LLMs (1B/0.5B) are evaluated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A sequential RL policy selecting code- and patient-specific KG expansion budgets outperforms fixed expansions and visit-level value-of-information selection (RQ4, Fig. 4).
- REFINE achieves the best results among 11 baselines on MIMIC-III and MIMIC-IV next-visit diagnosis prediction, with AUPRC of 47.41 and 50.24 versus MedCo's 45.35 and 48.28 (Table 1).
- REFINE acts as a plug-in concept encoder that consistently improves five diverse backbones, including under reduced training data (Fig. 3, Appendix A.6).
- The personalized GNN branch is the largest contributor (AUPRC 50.24→47.35 on MIMIC-IV without it), while budgeted soft-prompt LLM refinement adds gains beyond global code representations (Table 3).
Strengths
Novel budget formulation with verified personalization behavior
Sec. 2.5 casts per-code KG context size as sequential macro-actions (depth <=2, fanout <=10) with a cost-penalized terminal reward (Eqs. 9-15); the extracted Fig. 5 percentages (e.g., 17.0% for (2,(2,1)), 4.3% for (2,(5,5))) show the learned policy does not collapse to a single global budget.
Leakage controls unusually explicit
Appendix A.1 computes all TKG co-occurrence/temporal statistics from the training split of each fold only; Appendix A.5 states VoI selection is label-free at validation/test; the LLM runs locally on concept-level data only (Sec. 8).
Claims cross-check against tables
Practical two-stage training and honest backbone sensitivity
Large margins over externally developed baselines
Weaknesses
No dispersion or significance testing anywhere
Self-referential baseline ecosystem
Undefined aggregation makes Table 1 hard to audit
Only very small frozen LLMs evaluated
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
REFINE (arXiv:2609.04415, ACL-formatted, University of Kansas) targets next-visit diagnosis prediction on MIMIC-III and MIMIC-IV. Its claims: (1) medical concept representations should be patient-personalized rather than static; (2) a sequential RL policy can select per-code KG expansion budgets; (3) a frozen LLM with graph-aware soft prompts refines code semantics; (4) gains generalize across five backbones and data regimes. The paper explicitly extends the authors' MedCo line (global KG-LLM concept embeddings) toward per-patient, per-code conditioning.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is REFINE in one sentence?
REFINE is a plug-in encoder for EHR prediction that, per patient and per medical code, uses reinforcement learning to decide how much knowledge-graph context to retrieve, then combines a heterogeneous GNN with a frozen small LLM (via soft prompts) to produce personalized code representations for next-visit diagnosis prediction.
How much does REFINE improve over prior methods?
On MIMIC-III and MIMIC-IV next-visit diagnosis prediction (five-fold averages), the paper reports AUPRC of 47.41 and 50.24 versus 45.35 and 48.28 for the strongest prior baseline MedCo, and 39.44 and 43.56 for a plain Transformer backbone (Table 1).
Does REFINE need a large language model?
No. The LLM stays frozen and small: Llama-3.2-1B by default, and Appendix A.7 shows swapping in the even smaller Qwen2.5-0.5B only moderately lowers AUPRC (47.41 to 46.92 on MIMIC-III; 50.24 to 49.85 on MIMIC-IV).
Is the code and data available?
The paper links a GitHub repository (github.com/mohsen-nyb/REFINE) and uses the credentialed-but-public MIMIC-III and MIMIC-IV datasets; the knowledge-graph statistics are computed per fold from training data only, which supports reproducible evaluation.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3, 4 and 6 are charts whose data values were not present in the extracted text (only axis tick labels extracted), so exact per-backbone plug-in gains, RL-vs-VoI-vs-fixed AUPRC numbers, and data-insufficiency curve values could not be verified numerically; Figure 5 percentages and Figure 7 values did extract and were used.
- The extraction runs complete from the abstract through the references and appendices to Algorithm 1 with no truncation; the sole extraction limits are the chart-embedded values noted above.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.