Benchmarking Large Language Models for Biomedical Relation Extraction
Claudiu Creanga, Teodor Marchitan, Liviu P. Dinu
cs.CL
Quick verdict
Benchmarks LLMs, MLMs and a hybrid on SNPPhenA; O1 few-shot and fine-tuned Gemini 2.0 Pro lead all three tasks, but SOTA margins lack significance tests.
Summary
The paper benchmarks 18+ model configurations on the SNPPhenA corpus for extracting SNP-phenotype associations, across three tasks: sentence-level classification (positive/negative/neutral), abstract-level classification, and association strength classification (weak/moderate/strong). Systems include fine-tuned masked language models (DeBERTa v3, XLM-RoBERTa, ModernBERT) trained with a two-stage frozen-then-unfrozen regimen, a hybrid character-CNN + BiLSTM-CRF architecture borrowed from NER, and causal LLMs — open-source (Mistral-7B, Qwen2.5, DeepSeek-R1 distill) and proprietary (Gemini 2.0 family, GPT-4-Turbo/4o, o1-mini/o3-mini/o1) — under zero-shot, few-shot (1 or 5 examples), and fine-tuning (150 balanced examples) regimes, at temperature 0 with three repetitions. OpenAI O1 with one-shot prompting attains the best sentence-level macro-F1 (0.89), claimed as SOTA among non-finetuned models, and a new abstract-level SOTA (0.82 vs. the authors' prior 0.80 with fine-tuned Gemini 1.0 Pro). Fine-tuned Gemini 2.0 Pro leads association strength (F1 0.60 vs. 0.45 bag-of-words prior), the first LLM evaluation of that task. Main limitations: no significance tests or variance reporting despite 0.01–0.02 F1 margins; epoch selection explicitly used test-set performance; the MLM head is described as binary sigmoid/BCE despite 3-class tasks; open-source comparisons rely on <=14B or 2-bit quantized models; no code release is mentioned.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- OpenAI O1 with one-shot prompting achieves state-of-the-art sentence-level SNP-phenotype classification among non-finetuned models (macro-F1 0.89, Table 3).
- OpenAI O1 establishes a new abstract-level SOTA of 0.82 macro-F1, surpassing the prior 0.80 of fine-tuned Gemini 1.0 Pro (Table 4).
- Fine-tuned Gemini 2.0 Pro is the best model for association strength classification (F1 0.60), in the first LLM evaluation of this task (Table 5).
- Proprietary LLMs significantly outperform MLMs, the hybrid architecture, and the tested open-source models across all tasks (Sections 5-6).
- Fine-tuning does not always help: fine-tuned Gemini 2.0 Flash (0.78) slightly underperforms its few-shot variant (0.79) at sentence level (Section 5.1).
Strengths
Breadth and currency of model coverage on an underused corpus
Tables 3-5 span 18+ model-setting combinations across MLMs, a hybrid CNN-BiLSTM, and 2024-2025-era LLMs (Gemini 2.0 family, OpenAI o-series, Qwen2.5, Mistral, DeepSeek-R1 distill), applied to SNPPhenA, which the authors note has only four prior studies (Section 3).
First LLM evaluation of the association-strength task
Section 5.3 / Table 5 give the first non-BoW results for weak/moderate/strong grading: fine-tuned Gemini 2.0 Pro reaches 0.60 macro-F1 vs. the 0.45 bag-of-words baseline from the original corpus paper [2], while few-shot O1/Gemini manage only 0.34-0.38 — useful evidence the task is far from solved.
Meaningful abstract-level improvement and prompt transparency
Controlled decoding protocol
Candid reporting of nuanced/negative findings
Weaknesses
No statistical support for SOTA claims
Model selection on the test set
Contradictory or missing methodology details
Handicapped open-source comparison
Incomplete result reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the best model for extracting SNP-phenotype associations from text?
According to this benchmark on the SNPPhenA corpus, OpenAI O1 with one-shot prompting achieved the best sentence-level macro-F1 (0.89) and the best abstract-level F1 (0.82), while a fine-tuned Gemini 2.0 Pro was best at grading association strength (F1 0.60).
What is the SNPPhenA corpus?
A manually annotated corpus (Bokharaeian et al., 2017) of 360 biomedical abstracts containing 2,625 sentences, of which 483 include SNP-phenotype pairs labeled positive, negative, or neutral; positive pairs additionally carry a graded association strength of weak, moderate, or strong.
Do LLMs beat fine-tuned BERT models on biomedical relation extraction?
In this study, proprietary reasoning LLMs used few-shot (OpenAI O1, Gemini 2.0 Pro) matched or exceeded fine-tuned encoder models, while the tested small open-source models (7B-14B, one 2-bit quantized) trailed the best score by 0.15-0.27 F1 at sentence level.
How well can AI classify the strength of SNP-phenotype associations?
It remains an unsolved problem: the best result in this paper was macro-F1 0.60 from a fine-tuned Gemini 2.0 Pro, versus only 0.34-0.38 for few-shot LLMs and about 0.45 for the earlier bag-of-words baseline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (token-distribution histograms, fine-tuning and architecture diagrams) are described in the text but cannot be visually inspected from the extraction.
- Per-task test-split sizes and per-class results are absent from the extracted text (and appear absent from the paper itself), limiting statistical assessment.
- Few-shot and fine-tuning prompt/templates for the LLM settings other than the printed zero-shot template are not included, so prompt sensitivity cannot be judged.
- The arXiv stamp (2609.19071, dated Jul 2026) postdates the KES 2025 proceedings banner in the text; version provenance was not verifiable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.