Fine-Tuning Models for Biomedical Relation Extraction
Claudiu Creanga, Liviu P. Dinu, Liviu P. Dinu, Daniela Gifu
cs.CL
Quick verdict
Claims fine-tuned Gemini 1.0 Pro beats SNPPhenA SOTA (0.89 sentence, 0.80 abstract F1); plausible, but no significance tests or code temper confidence.
Summary
This paper evaluates fine-tuned and prompted language models for extracting variant–phenotype relations from biomedical literature using the SNPPhenA corpus, where the task is to classify each SNP–phenotype pair as positive, neutral, or negative. The authors fine-tune four BERT-style masked language models (DeBERTa v3, BioBERT, BERT, DistilBERT) with a two-step procedure that trains a classification head first, then unfreezes only the last pre-trained layer. DeBERTa v3 reaches 0.85 macro-F1 at sentence level, close to the published BioBERTGRU state of the art (0.881). They also test causal LLMs zero-shot and few-shot (SOLAR 10.7B, Mistral 7B, Llama-2 7B, Gemini 1.0/1.5 Pro), all of which underperform, then fine-tune Gemini 1.0 Pro via Google's tuning API on a balanced 50-examples-per-class subset, reporting 0.89 macro-F1 at sentence level and 0.80 at abstract level versus a 0.645 prior abstract-level SOTA. Per-class tables show the negative class is weakest (F1 0.831 sentence-level; 0.630 abstract-level, precision 0.523). Main limitations: no statistical significance testing despite a 0.009 sentence-level margin on a 365-item test set with only 29 negative examples; a potential confound because BERT models were not fine-tuned on the same balanced subset; no code or exact prompts/decoding settings released; and the dependence on a proprietary, versioned tuning API limits reproducibility.
Review complete
Assessment at a glance
0
risk flags identified
● high 0
● medium 0
● low 0
4 strengths · 5 weaknesses · 0 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Fine-tuning small BERT-based models, especially DeBERTa v3, approaches the SNPPhenA sentence-level state of the art (0.85 vs 0.881 macro-F1).
- Gemini 1.0 Pro fine-tuned on a balanced 50-examples-per-class subset sets a new sentence-level SOTA of 0.89 macro-F1.
- The same fine-tuned model sets an abstract-level SOTA of 0.80 macro-F1, versus a previous best of 0.645.
- Open-source causal LLMs underperform in zero-/few-shot settings (best: SOLAR 10B at 0.67), below even an older CNN-LSTM baseline.
- Training on a balanced 50-per-class subsample beats fine-tuning on the full imbalanced dataset (0.89 vs 0.72 for Gemini 1.0 Pro).
Strengths
Focus on an under-served benchmark
SNPPhenA is the only curated SNP–phenotype corpus with only three prior papers ([4], [15], [21]); the paper targets exactly this neglected sub-domain rather than the crowded ChemProt/DDI space (Section 3.1).
Systematic MLM ablation
Table 3 layer ablation (last layer 0.85 vs -2: 0.80 vs [-1,-3]: 0.83) plus the full-model fine-tuning collapse to 0.66 (Section 3.4) document a coherent small-data overfitting story and a practical recipe (freeze, then unfreeze last layer at LR 2e-5).
Breadth across paradigms with transparent prompting
Honest error analysis and one large-margin result
Weaknesses
No statistical support for the sentence-level SOTA claim
Confounded headline comparison
Low reproducibility
Related-work and editorial hygiene problems
Baselines not re-run under identical conditions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Fine-Tuning Models for Biomedical Relation Extraction
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 0 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 0 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the SNPPhenA corpus?
SNPPhenA (Bokharaeian et al., 2017) is a manually curated corpus of biomedical abstracts that annotates associations between single-nucleotide polymorphisms (SNPs) and phenotypes. Each SNP–phenotype pair is labeled positive, neutral, or negative, and it is described in the paper as the only dedicated dataset for this variant–phenotype relation type.
Does fine-tuned Gemini beat BERT models on biomedical relation extraction?
On SNPPhenA, the paper reports that Gemini 1.0 Pro fine-tuned on a balanced subset of 50 examples per class reaches 0.89 macro-F1 at sentence level (previous best BioBERTGRU: 0.881) and 0.80 at abstract level (previous best: 0.645). Fine-tuned DeBERTa v3 reaches 0.85 at sentence level. Caveat: the sentence-level margin of 0.009 is reported without statistical significance testing.
Is the code or the fine-tuned model available?
No. The paper publishes verbatim prompts for the zero-/few-shot settings but no repository, no decoding settings, and no checkpoints. The headline result also depends on Google's proprietary Gemini 1.0 Pro tuning API, so exact replication is currently impractical. The SNPPhenA corpus itself is publicly available.
What was the best small model in the study?
DeBERTa v3, fine-tuned with a two-step recipe (classification head first, then only the last pre-trained layer unfrozen), scoring 0.85 macro-F1 at sentence level — close to the 0.881 state of the art.
What is the difference between sentence-level and abstract-level tasks in this paper?
Sentence-level: only the target sentence is given to the model. Abstract-level: the entire abstract is processed, so relations must be found across multiple sentences. Fine-tuned Gemini 1.0 Pro scored 0.89 vs 0.80 macro-F1 on these two variants respectively, and the abstract-level gap over the prior SOTA (0.80 vs 0.645) is the paper's largest claimed gain.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–4 (architecture, confusion matrices) appear only as captions in the extracted text; matrix cell values could not be verified.
- Table 5/7 scores are rounded to two decimals; exact F1 values and per-run variance are unavailable.
- The arXiv stamp (2609.20169, Jul 2026) postdates the KES 2024 Procedia version, creating minor version-provenance ambiguity.
- Prompt templates contain extraction artifacts ([INS]/[\INS]); the exact production prompts may differ slightly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.