ECGQuest: Benchmarking and Fine-Tuning Language Models for Electrocardiography
Mohammadsina Hassannia, Matthew A. Reyna, Reza Sameni
cs.CL · cs.IR
Quick verdict
GPT-4o-built 10.9k-pair ECG true/false benchmark; LoRA-tuned 14B models beat GPT-5 on it, but labels lack expert validation and gaps lack significance tests.
Summary
The authors introduce ECGQuest, a True/False question benchmark for electrocardiography knowledge, generated by a GPT-4o pipeline from 23 reference documents (textbooks, clinical guidelines, device manuals, tutorials) plus Computing in Cardiology proceedings 2003-2025. Every item is source-linked and paired with a minimally edited opposite-label counterpart, yielding 10,904 balanced pairs (21,808 questions). They evaluate 23 models zero-shot on a 1,050-question test split: accuracy spans 49.5-74.4%, GPT-5 leads, general-purpose models beat medical-specialized ones, and BERT/BiomedBERT encoder baselines sit near chance. LoRA fine-tuning of five 7-14B open models adds 6.5-14.1 points; DeepSeek-R1-Distill-Qwen-14B reaches 76.3% and a five-model voting ensemble 78.5%, nominally exceeding GPT-5 (74.4%). External conversion of ECG-topic MedMCQA/MedQA items to True/False shows gains transfer only partially: weak or class-biased models improve while strong balanced models (Llama-3.1-8B, DeepSeek-R1-14B) drop. Main limitations: labels are GPT-4o-generated and GPT-4o-verified with no expert over-read (authors call them weak labels); the headline margin over GPT-5 (~2 points, ~20 of 1,050 questions) carries no significance testing; external sets are small (147 MedQA questions) and filtered by an unvalidated GPT-4o topic classifier with acknowledged leakage risk; and the dataset/models are only promised upon acceptance, so results are not yet independently reproducible.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- ECGQuest provides 10,904 source-linked, label-balanced True/False question pairs (21,808 questions) generated by GPT-4o from 23 ECG reference documents and Computing in Cardiology proceedings 2003-2025, each paired with a minimally edited opposite-label counterpart.
- Zero-shot accuracy of 23 evaluated models spans 49.5% to 74.4% with GPT-5 best (74.4%), general-purpose models outperforming all medically specialized models, and BERT/BiomedBERT encoder baselines near chance (52.1%/54.0%).
- LoRA fine-tuning improves all five 7-14B open models by 6.5-14.1 points; DeepSeek-R1-Distill-Qwen-14B reaches 76.3% and a five-model voting ensemble 78.5%, nominally exceeding GPT-5's 74.4% on the held-out ECGQuest test set.
- On external exam-key ECG subsets of MedMCQA and MedQA, fine-tuning primarily benefits weak or class-biased models and reduces accuracy for already strong balanced models, indicating a generalization gap.
- The labels are weak — GPT-4o-generated, machine-verified, and never expert-over-read — so reported scores measure agreement with benchmark labels rather than clinical correctness.
Strengths
Paired-negation design blocks label shortcuts
Section 3.1 pairs every question with a minimally edited opposite-label counterpart from the same source text, verified programmatically (identical source_text, flipped label, distinct question); this eliminates True/False prevalence or topic priors as answering shortcuts, a hygiene step most LLM-generated QA benchmarks skip.
Full traceability chain and published prompts
Sources are MD5-hashed and title-anonymized, every item retains source MD5 + page number + generation mode, source_text must be verbatim, near-duplicates removed at Jaccard >=0.90, and all three prompts (generation, vision, label-balancing) plus 638 rejection examples with reasons are given in Supplement C — unusually reproducible construction documentation.
Broad, multi-metric model panel
Independent-label external validation, honestly reported
Clear bias diagnostics via accuracy/macro-F1 gaps
Weaknesses
Label circularity: generator = verifier = evaluated model family
Zero statistical inference despite headline ranking claims
Fine-tuning gains are benchmark-confined
Reproducibility deferred and partially impossible
Undocumented tuning and an unexplained duplicate row
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This preprint (arXiv 2608.30893, cs.CL/cs.IR; Emory + Georgia Tech) introduces ECGQuest and reports a benchmark-plus-fine-tuning study. Five claims are made: (1) a GPT-4o pipeline converted 23 ECG reference documents plus Computing in Cardiology proceedings (2003-2025) into 10,904 source-linked True/False pairs (21,808 questions), each with a minimally edited opposite-label counterpart; (2) 23 models evaluated zero-shot on a 1,050-item test split span 49.5-74.4% accuracy, with GPT-5 best and every medically specialized model beaten by general-purpose ones; (3) LoRA fine-tuning of five 7-14B open models yields +6.5-14.1 points, with DeepSeek-R1-Distill-Qwen-14B at 76.3% and a five-model voting ensemble at 78.5% versus GPT-5's 74.4%; (4) on external ECG subsets of MedMCQA/MedQA converted to True/False with official exam keys, fine-tuning gains transfer only partially; (5) labels are weak — machine-generated, machine-verified, never expert-audited. The abstract headlines claims 1-3; claims 4-5 live in Section 5 and Supplement B.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is ECGQuest?
ECGQuest is a benchmark of 21,808 True/False questions (10,904 balanced pairs) about electrocardiography knowledge, generated by GPT-4o from 23 ECG textbooks, clinical guidelines, device manuals, tutorials, and Computing in Cardiology proceedings from 2003-2025. Each question is grounded in a quoted source passage and paired with a minimally altered opposite-label twin, which prevents models from guessing answers from True/False base rates.
Can small language models beat GPT-5 on ECG questions after fine-tuning?
On ECGQuest's own 1,050-question test set, yes nominally: a LoRA-tuned DeepSeek-R1-Distill-Qwen-14B scored 76.3% and a five-model voting ensemble 78.5%, versus GPT-5's 74.4%. But the margins (~2 points, roughly 20 questions) come with no statistical significance tests, and on external MedQA/MedMCQA-derived sets the fine-tuned strong models actually scored lower than their base versions — so the 'beats GPT-5' result is benchmark-specific rather than a general victory.
Are ECGQuest's answer labels medically verified?
No. All labels were generated by GPT-4o and automatically verified by another language-model pass against the quoted source text. The authors explicitly state no expert over-read has occurred and call the labels 'weak labels'. Model scores on ECGQuest therefore measure agreement with GPT-4o's reading of the sources, not clinical correctness.
Where can I download the ECGQuest dataset and the fine-tuned models?
According to the paper, the Q&A set and the fine-tuned models will be made public, but only 'upon manuscript acceptance'; no URL or repository is provided in this preprint version, so as of now they cannot be downloaded and the reported results cannot be independently reproduced.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The metadata abstract supplied alongside the paper ends mid-sentence ('reached 76'), but the complete abstract is present at the start of the extracted full text, so no content was actually lost.
- Figure 1's diagram labels and the Figure 2 pairwise-kappa matrix extracted as run-on streams of names and numbers; individual matrix cells were not reliably readable, so all kappa values cited in this review come from body text (Section 4.3).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.