arXiv paper2026-09-01· clinical NLP / LLM evaluation· benchmark

ClinTraceBench: Source-Verifiable Longitudinal Clinical Reasoning over EHR-Derived Dialogues

Q: What is ClinTraceBench?

A benchmark of 385 multi-visit doctor-patient dialogues synthesized from MIMIC-IV structured data, with 6,271 questions across nine tasks (lab retrieval, trends, attribution detection, contradiction, cross-patient comparison, treatment recall, treatment response, abstention). Each gold answer is traceable to a specific dialogue turn via event IDs. It tests whether compact history representations — dense retrieval, structured timelines, LLM summaries, agentic memory — preserve longitudinal clinical signal compared with feeding the full chart.

Q: What are the main findings of ClinTraceBench?

Full context gives the best pooled accuracy (63.0–70.6% across four backbones), with BGE-M3 retrieval within 1–4 points. Even when an attribution sentence is present before a representation is built, Mem0, A-Mem and LLM summaries recover only 0–5.3% of the injected positives. Compressed representations lose multi-visit trends and cross-patient comparisons, and abstention degrades non-monotonically as context grows (last-visit-only beats full-context on every backbone). On cost-quality, Claude Haiku 4.5 under full-context ($25.76) dominates Sonnet 4.6 ($106.21).

Q: Is the ClinTraceBench dataset publicly available?

Partially. The question set with event-ID provenance and gold answers, all 200,672 predictions, the scoring harness, and the injection-probe generator are on GitHub (github.com/HathyHuimin/ClinTraceBench). The dialogues themselves are not redistributed because they inherit MIMIC-IV's PhysioNet license restrictions; regenerating them requires PhysioNet credentialing and MIMIC-IV access.

Q: Can ClinTraceBench results be trusted given the dialogues are synthetic?

The six-gate pipeline (deterministic L0–L4 checks plus a 370-question human audit at 98.92% agreement) establishes that gold answers are correct and source-traceable within the synthetic dialogues. However, the authors themselves state that conversational realism — the ambiguity, redundancy and missing documentation of real clinical text — is untested, and they explicitly do not claim that the strategy ordering transfers to real clinical conversations.

Huimin Wang, Zhengyi Zhao, Yutian Zhao

View original paper ↗arXiv:2609.01111

cs.CL

Quick verdict

Claims source-verifiable 9-task EHR dialogue benchmark; rigorous validation, honest caveats, but headline probe uses 19 positives and dialogues are synthetic.

Summary

The paper introduces ClinTraceBench, a benchmark of 385 multi-visit dialogues synthesized from de-identified MIMIC-IV structured data, built to test whether compact history representations preserve the longitudinal signal clinical reasoning needs. Nine tasks (T1–T9) cover lab retrieval, trend classification, attribution detection, retrospective evidence, contradiction, cross-patient comparison, treatment recall, treatment response, and abstention, with event-ID provenance and six validation gates (deterministic L0–L4 plus human audit at 98.92% agreement). Eight history strategies — blind floor, last-visit-only, full-context, BGE-M3 retrieval, structured timeline, LLM summary, Mem0, and A-Mem — are evaluated on four backbones over 6,271 questions (200,672 predictions). Key findings: even when the injected attribution sentence is present before construction, compressed and agentic-memory representations recover only 0–5.3% of injected positives; compressed strategies pay an aggregation tax on multi-visit trends and cross-patient comparisons; blind-to-full gains range from +29.8pp (GPT-4o-mini) to +62.7pp (Haiku); abstention varies non-monotonically with context length. On the cost-quality frontier, Haiku under full-context ($25.76) dominates Sonnet ($106.21). Main limitations: dialogues are synthetic and realism on real clinical text is untested; T4 (53 questions) and T8 (134) are under-powered; the headline preservation probe rests on only 19 injected positives.

Review complete

Assessment at a glance

clinical NLP / LLM evaluation

risk flags identified

● high 0

● medium 2

● low 2

Reproducibility readinessmedium

Code & data availableyes

Review confidencehigh

AI text signal0/20 passages flagged

Deep research viableCode-reproducible

5 strengths · 0 weaknesses · 5 author questions

AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.

Levels reflect verification complexity — not a quality score. Evolve never rates your paper.

Key claims under review

Compressed and agentic-memory representations lose the finding–diagnosis relation even under equal input, recovering only 0–5.3% of injected T3 attribution positives (Table 3).
Full-context is the accuracy ceiling on every backbone (pooled 67.2–70.6%), with BGE-M3 dense retrieval within 1–4pp and structured timelines strikingly backbone-invariant (±0.6pp).
How much a backbone extracts from given context (blind-to-full gap, +29.8pp GPT-4o-mini to +62.7pp Haiku) varies more than overall accuracy, so backbone choice is not one capacity dimension.
Abstention is non-monotonic in context length: last-visit-only beats full-context on T9 on every backbone (0.924 vs 0.763 pooled, McNemar p = 1.7e-38).
The cost-quality Pareto frontier inverts 'biggest backbone wins': Haiku 4.5 under full-context ($25.76, 0.706) dominates Sonnet 4.6 ($106.21, 0.698).

Strengths

Six-gate validation with event-level provenance
L0–L4 deterministic checks plus an L5 audit of 370 questions (366/370 = 98.92% agreement); golds independently recomputed with <0.5% mismatch (Appendix A); every answer is traceable to a specific dialogue turn — a property the paper correctly notes is absent from LOCOMO/LongMemEval.
Controlled pre/post-construction injection probe
The equal-input setting (Table 3) separates preservation from staleness: compressed strategies recover ≤1/19 injected positives pre-construction, while post-construction the four upstream-built strategies hit the 0.500 constant-No floor across all 16 cells (Figure 5) — a methodology generalizable beyond clinical settings.
Statistical hygiene and power honesty
Confounds actively investigated, not waved away
Full cost transparency and artifact release

Full Review Report

$1.99 · one-time

Unlock the complete preliminary analysis Evolve produced for this paper.

Report preview

Scope & Claims

ClinTraceBench asks whether the compact history representations used to scale clinical LLM assistants — dense retrieval, structured timelines, LLM summaries, agentic memory (Mem0, A-Mem) — preserve the longitudinal signal that multi-visit reasoning needs. The substrate is 385 dialogues synthesized from de-identified MIMIC-IV structured fields (400 patients sampled across 4 diseases × complexity tertiles; 15 dropped by the verifier), carrying per-event event_uid provenance. Nine tasks (T1–T9) span lab-value retrieval, trend classification, controlled attribution detection, retrospective evidence, controlled contradiction, cross-patient comparison, treatment recall, observed treatment response, and abstention. The evaluation grid is 8 strategies × 4 backbones (DeepSeek-V3, GPT-4o-mini, Haiku 4.5, Sonnet 4.6) on a fixed 6,271-question set: 32 cells, 200,672 predictions, $527.85 stated in §1 (Table 4 totals $527.87). Four signal findings (SP1–SP4) plus a cost claim (Haiku full-context $25.76/0.706 dominates Sonnet $106.21/0.698) are put forward.

Methodology Assessment

The validation chain is the paper's strongest asset: L0 schema, L1 anchor-in-covered-events, L2 independent gold recomputation (<0.5% mismatch), L3 no-leakage, L4 distribution checks, plus L5 human audit of 370 questions (366/370 = 98.92%; Appendix A). The controlled T3 injection probe is genuinely clever: injecting the attribution sentence before versus after representation construction separates signal preservation from staleness. Under equal input (Table 3), Mem0 and A-Mem recover 1/19 positives at best and llm-summary 0/19; post-construction, the four upstream-bu

The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.

All 5 strengths & 0 weaknesses, analyzed in detail
4 risk flags with severity, section & reviewer questions
Methods & statistical evidence assessment
5 questions a peer reviewer would ask
Full six-dimension evaluation report & recommendations

One-time purchase · this paper only

One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.

Frequently asked questions

What is ClinTraceBench?

A benchmark of 385 multi-visit doctor-patient dialogues synthesized from MIMIC-IV structured data, with 6,271 questions across nine tasks (lab retrieval, trends, attribution detection, contradiction, cross-patient comparison, treatment recall, treatment response, abstention). Each gold answer is traceable to a specific dialogue turn via event IDs. It tests whether compact history representations — dense retrieval, structured timelines, LLM summaries, agentic memory — preserve longitudinal clinical signal compared with feeding the full chart.

What are the main findings of ClinTraceBench?

Full context gives the best pooled accuracy (63.0–70.6% across four backbones), with BGE-M3 retrieval within 1–4 points. Even when an attribution sentence is present before a representation is built, Mem0, A-Mem and LLM summaries recover only 0–5.3% of the injected positives. Compressed representations lose multi-visit trends and cross-patient comparisons, and abstention degrades non-monotonically as context grows (last-visit-only beats full-context on every backbone). On cost-quality, Claude Haiku 4.5 under full-context ($25.76) dominates Sonnet 4.6 ($106.21).

Is the ClinTraceBench dataset publicly available?

Partially. The question set with event-ID provenance and gold answers, all 200,672 predictions, the scoring harness, and the injection-probe generator are on GitHub (github.com/HathyHuimin/ClinTraceBench). The dialogues themselves are not redistributed because they inherit MIMIC-IV's PhysioNet license restrictions; regenerating them requires PhysioNet credentialing and MIMIC-IV access.

Can ClinTraceBench results be trusted given the dialogues are synthetic?

The six-gate pipeline (deterministic L0–L4 checks plus a 370-question human audit at 98.92% agreement) establishes that gold answers are correct and source-traceable within the synthetic dialogues. However, the authors themselves state that conversational realism — the ambiguity, redundancy and missing documentation of real clinical text — is untested, and they explicitly do not claim that the strategy ordering transfers to real clinical conversations.

About this review

This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.

The abstract as supplied in the source metadata is cut mid-word ('inve'), but the complete abstract appears in the embedded full text; nothing else at the true end of the text is missing (references and all appendices A–I extracted in full).
Numeric labels embedded inside figures (e.g., the Figure 2 pipeline bands) extracted with OCR noise; every number cited in this review comes from tables and prose that extracted cleanly, including Tables 1–14.

Go deeper on this paper

Wrote a paper? Stress-test it before reviewers do.

Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.

Scan your paper