CliniCIRCA: A Modular LLM Framework for Constructing Longitudinal Mental Health Patient Journeys from Raw EHR Narratives
Aiwei Ivy Zhang, Nimra Ishfaq, Mohit Chandra, Santiago Alvarez Lesmes, Adam Coscia, Khatiya Chelidze Moon, Xiaohan Ding, Munmun De Choudhury
cs.CL · cs.AI · cs.IR · cs.LG
Quick verdict
An LLM pipeline turning MIMIC-III psychiatric notes into calendar-anchored timelines; clinician-validated and honest, but the modular design is never ablated.
Summary
The paper introduces CliniCIRCA, a three-stage LLM framework that reconstructs longitudinal mental health patient journeys from unstructured discharge summaries. Stage 1 extracts atomic clinical events, Stage 2 tags each event with an ISO date and one of four certainty labels anchored to admission, discharge, and birth dates, and Stage 3 renders the tagged timeline into a date-grouped summary. From 14,882 MIMIC-III psychiatric-coded admissions, the authors benchmark 11 models on 52 discharge summaries (15,891 events), select Gemini 2.5 Pro, and exhaustively correct 629 errors with clinicians to build a gold reference. The corrected timelines drive summarization achieving 1.52× compression. Scaling to 1,000 silver timelines, instruction tuning generally improves five open-weight models over zero/few-shot prompting on event extraction (e.g., Mistral-24B rises from 0.284 to 0.847 RM-F1 under SILVER→GOLD), temporal tagging, and summarization, and silver data outperforms training on the 52 gold examples alone. Limitations: the modular design is never compared against a single-pass baseline; SILVER→SILVER evaluation is circular since labels come from the same pipeline; the gold set is small (52 notes) with all annotation by authors, and the two clinicians who authored the Stage 3 specification also evaluated it; the LLM judge shows only moderate agreement on rare errors (positive agreement 0.669) and over-reports Stage 3 errors 6.7-fold.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CliniCIRCA is the first framework to temporally classify clinical events across unstructured discharge summaries without event-level timestamps, using calendar anchors and four certainty tags.
- A calibrated LLM judge (Gemini 3.1 Pro) agrees strongly with clinicians on correct events but misses most clinically coherent temporal errors (169 of 175 over-specified pre-admission tags), so exhaustive expert adjudication is required for gold references.
- Instruction tuning on 1,000 pipeline-generated silver timelines generally improves five open-weight models over zero- and few-shot prompting on event extraction, temporal tagging, and summarization, evaluated against clinician-verified gold.
- At the evaluated scale, 750 silver documents provide stronger supervision than the 52 clinician-verified examples alone, without implying the silver labels are of higher quality.
Strengths
Novel problem formulation with clear operationalization
Reconstructs open-vocabulary, calendar-anchored, uncertainty-aware timelines from raw discharge summaries using four certainty tags ([EXACT]/[APPROX]/[PRE ADM]/[INDETERMINATE]) anchored only to DOB, admission, and discharge dates (Sections 1, 3; Table A6) — no event-level timestamps or predefined schemas required.
Exhaustive clinician-in-the-loop gold construction
All 15,891 Stage-2 events across 52 summaries were human-reviewed over 6 sessions with 629 errors corrected (Sections 5.2, G.2); agreement reporting handles rare-error class imbalance properly (κ=0.655 alongside Gwet's AC1=0.972, negative agreement 0.986, positive agreement 0.669; Table A11).
Unusually honest auditing of the LLM judge
Disciplined downstream statistics
Strong practical artifact and disclosure
Weaknesses
No end-to-end single-pass baseline or stage ablation
Circular SILVER→SILVER evaluation
Gold set and evaluator independence are thin
Decorative power analysis and no multiple-comparison control
Modest demonstrated clinical utility
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CliniCIRCA (arXiv 2609.19585) reframes patient-journey recovery as open-vocabulary, calendar-anchored, uncertainty-aware temporal reconstruction from a single discharge summary. Three sequential LLM stages produce two linked outputs: Output A, an event-level timeline of (ISO date/range, certainty tag, atomic event) tuples, and Output B, a date-grouped prose summary. Four tags ([EXACT], [APPROX], [PRE ADM], [INDETERMINATE]) encode temporal certainty; structured MIMIC-III fields (DOB, admission, discharge) serve only as anchors and cohort filters. The authors claim three things: (1) this is the first temporal classification of clinical events across unstructured discharge summaries without event-level timestamps; (2) calibrated LLM judging plus exhaustive clinician adjudication yields a trustworthy gold reference; (3) 1,000 pipeline-generated silver timelines are effective training data for open-weight models. Claims (2) and (3) are well supported within scope; claim (1) rests on literature comparison rather than any head-to-head experiment, and the implicit claim that modular decomposition beats end-to-end generation is never tested.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does CliniCIRCA do?
CliniCIRCA is a three-stage LLM pipeline that converts raw psychiatric discharge summaries from the MIMIC-III database into calendar-anchored patient timelines: it extracts atomic clinical events, assigns each an ISO date and a certainty tag (exact, approximate, pre-admission, or indeterminate) using admission, discharge, and birth dates as anchors, and then renders the timeline as a date-grouped prose summary with about 1.5× compression of the original note.
How accurate is CliniCIRCA?
On a benchmark of 52 clinician-reviewed discharge summaries containing 15,891 events, exhaustive human review found 3.96% of events erroneous and corrected 629 of them to build a gold standard. Models trained on 1,000 automatically generated silver timelines reached up to 0.847 relaxed-match F1 for event extraction and 0.754 ROUGE-L for summarization when evaluated on the clinician-verified gold set.
Is the CliniCIRCA code and data available?
The paper links an anonymous code repository and fully reproduces all prompts and evaluation rubrics in its appendices. However, the source data is MIMIC-III, which requires a PhysioNet credential and training, and the teacher pipeline uses paid Gemini and Claude API access, so full replication is gated behind those credentials.
What are the main limitations of the paper?
Key limitations acknowledged by the authors: the modular design was never compared against a single-pass end-to-end baseline; silver-data evaluations are circular because the same pipeline produced the labels; the gold set is only 52 notes annotated entirely by the authors, with the Stage 3 specification authors also serving as evaluators; and the Stage 3 summarization excludes affect and routine exam findings, so psychiatric symptom trajectories may be missing from the final readable summary.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several passages were elided in text extraction (parts of Section 5, the tail of Appendix B, and fragments of Appendices E-I); judgments rest on the preserved tables, rubrics, and appendices, which contain the key numbers.
- Figures (Fig. 1, A1-A3) could not be analyzed visually; Figure A2's error-type counts were taken from the accompanying text description rather than the figure itself.
- Model names include very recent systems (Gemini 2.5/3.1, Claude Sonnet 4.6, Gemma 4, Qwen 3.6) whose relative standings could not be cross-checked against external leaderboards within this review.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.