MMTClinic: Multimodal, Multilingual Time Series Question Answering and Reasoning Benchmark for Clinical Domain
Sourav Malakar, Harshit Nigam, Akash Ghosh, Sriparna Saha, Amlan Chakrabarti, Saptarsi Goswami, Priti Singh
cs.CL · cs.AI
Quick verdict
Claims a 30K multilingual clinical time-series LLM benchmark; promising resource, but binary-scored reasoning, accuracy-only single-run eval, no baselines.
Summary
The paper introduces MMTClinic, a benchmark of 30,000 question–answer pairs (15,000 MCQ, 15,000 open-ended reasoning) for testing whether LLMs can reason over clinical time series. It is built semi-automatically from the 2012 PhysioNet/CinC ICU dataset (2,000 patients, first 48 hours, six vital signs as CSV plus line-plot images): GPT-4o few-shot prompting generates English questions, Google Translate plus native-linguist review produces Hindi, Bengali, Marathi and Tamil versions, and four medical experts rate about 70% of samples (means 4.2–4.5/5). Three tasks are covered: in-hospital mortality prediction, heart-rate forecasting, and SOFA-score estimation. Ten text-only and four multimodal models were evaluated zero-shot, few-shot, and with CoT; DeepSeek-R1 leads (about 65–66% zero-shot text+time-series MCQ, up to 98.4% multimodal MCQ under few-shot/CoT), multimodal models score only 26–46% on zero-shot multimodal MCQ, and the weakest text models fall below the 25% chance level. Key limitations: the open-ended reasoning track is actually scored as binary 1/0 answers; accuracy is the only metric, from single-run inference with no significance tests; no chance or majority-class baselines; internal inconsistencies (three vs four tasks, 13 vs 14 models); several incorrect citations; dataset release deferred to paper acceptance. A valuable first step whose evaluation methodology needs substantial tightening.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MMTClinic is the first benchmark to unify clinical text, time series, and images in both MCQ and reasoning formats across five languages.
- Current LLMs show large performance differences across tasks, languages, and modalities, with persistent weaknesses in vision-language reasoning and regional languages.
- DeepSeek-R1 consistently outperforms all 13 evaluated models with minimal reliance on chain-of-thought prompting.
- Expert review by four physicians (~70% of samples, mean 4.2–4.5/5) validates the clinical quality of the LLM-generated questions.
Strengths
Genuinely new dimension combination
Table 6 positions MMTClinic as the only listed benchmark combining text + time-series + images, MCQ + reasoning formats, and 5 languages; prior entries (ECG-QA, Q-Heart, EHRNoteQA, EHRXQA) cover at most two modalities and one language.
Independent ground truth
Answers are taken directly from PhysioNet outcome/signal files, not from the LLM (Sec 3.3 Step 3, FAQ Q2) — a clean decoupling of question generation from answer supervision, a common flaw in LLM-built benchmarks.
Layered quality control
Broad evaluation grid
Weaknesses
The 'open-ended' reasoning track is binary
No chance/majority baselines or class statistics
Accuracy-only, single-run statistics
Internal contradictions and citation errors
Headline claims contradicted by own tables
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MMTClinic (arXiv 2609.04842) claims to be the first benchmark unifying clinical text, multivariate time series, and images in both MCQ and open-ended formats across five languages (EN/HI/BN/MR/TA). Three ICU tasks — in-hospital mortality prediction, 48-hour heart-rate forecasting, and SOFA-score estimation — span 30,000 QA pairs (15k MCQ + 15k 'reasoning') generated from 2,000 patients of the 2012 PhysioNet/CinC Challenge. The authors evaluate 13 models (Table 5 actually lists 14) across zero-shot, few-shot, and CoT, concluding that proprietary models excel in multimodal settings, open-source models handle text+time-series well, and vision-language and regional-language performance drop 'consistently' (Conclusion).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the MMTClinic benchmark?
A benchmark of 30,000 question–answer pairs (15,000 multiple-choice and 15,000 'reasoning' items scored as binary 1/0) for testing how well large language models handle clinical ICU time-series data, combining patient text, CSV vital-sign series, line-plot images, and five languages (English, Hindi, Bengali, Marathi, Tamil) across three tasks: mortality prediction, heart-rate forecasting, and SOFA-score estimation.
What data is MMTClinic built from?
The public 2012 PhysioNet/CinC Challenge ICU dataset: 2,000 patients' first 48 hours of stay, six vital signs resampled hourly and provided both as CSV files and matplotlib line-plot images. Ground-truth answers come directly from PhysioNet outcome files rather than from the question-generating LLM, and questions were produced by GPT-4o few-shot prompting then translated into four Indian languages.
Which models performed best on MMTClinic?
In the reported results DeepSeek-R1 leads: roughly 65–66% average accuracy on zero-shot text+time-series multiple-choice questions and 96–99% on multimodal multiple-choice items under few-shot/chain-of-thought prompting, while multimodal models scored 26–46% in the zero-shot multimodal setting and the weakest text models fell below the 25% random-choice level. Notably, Qwen3-235B-A22B beat DeepSeek-R1 on the text+time-series reasoning track.
Is the MMTClinic dataset publicly available?
Not at the time of this preprint: the paper states the dataset will be made available 'on successful acceptance', and the manuscript's access link appears as a placeholder. Prompts used for generation and evaluation are reproduced in the paper's appendix, but no code repository is stated.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The sampled excerpt omits middle sections marked as sampling gaps: zero-shot per-language result tables, Appendix 9.6 inter-annotator-agreement values, and all figure contents (Figures 1, 2, 12) are not in the provided text; judgments that would depend on them are hedged.
- The dataset access link appears as the literal phrase 'this link' with no URL in the extracted text; whether a live URL exists in the PDF could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.