Analysis of Prompt Engineering for Drug Toxicity Prediction
Mia MacGregor, Aakash Welgamage Don, Mark Bartlett
cs.AI
Quick verdict
Claims LLM variance dwarfs prompt tuning and RDKit beats LLM features for toxicity prediction; plausible but lacks significance tests — preliminary.
Summary
The paper examines whether prompt phrasing matters when large language models (LLMs) are used in a drug toxicity prediction pipeline. In a first stage, five prompt variants (Table 1), each crossed with four job roles, were given to five LLMs (Gemma3 4B, Deepseek-r1-Distill-Qwen-8B, Llama3.2 3B, Mistral 7B, Gemini-2.5-flash) to elicit fifteen toxicity-relevant features; only seven features appeared in over 25% of lists, with LogP universal. In a second stage, 18 datasets were built by asking LLMs to compute feature values for 2,294 SMILES (789 toxic, 1,505 non-toxic) from a PubChem cytotoxicity assay (AID 489025), then training six ML algorithms (Random Forest, Decision Tree, MLP, SVM, Naive Bayes, XGBoost) with SMOTE, scaling and grid search on an 80/20 split, evaluated by AUC. Repeated identical prompts, LLM swaps, and role changes were compared. The central findings: inherent LLM stochasticity produced variance comparable to changing prompt or model; RDKit-based chemoinformatic extraction substantially outperformed LLM-generated feature values; no job role was consistently best. Key limitations: conclusions rest on visual AUC comparisons without significance testing (acknowledged by the authors as future work), the model set is small, the ground truth is a single cell-line assay, and seeds/temperature for LLM generation are unreported. Code is released on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 4
● low 0
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 18 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Natural stochastic variance in LLM outputs outweighs the effect of prompt fine-tuning on downstream toxicity prediction accuracy.
- Chemoinformatic (RDKit) feature extraction yields substantially higher AUC than LLM-generated feature values (4cL Calc vs 4cL LLM, Fig. 2).
- Changing the LLM had little impact on downstream AUC, and no job-role prompt variant was consistently best.
- The proposed prompt-analysis methodology generalizes to a wide range of prompt types across bioinformatics.
Strengths
Factorial design cleanly isolates variance sources
The dataset naming scheme (Fig. 2 caption) separates factors: repeated identical prompts (1aL1-1aL3) isolate stochasticity, 3aD/3aM/3aL/3aG/3aF isolate the LLM factor, 5aL-5dL isolate job roles, and the 4cL Calc vs 4cL LLM pair isolates extraction method with feature lists held fixed.
Honest negative result with actionable recommendation
The finding that inherent LLM variance rivals prompt/model choice (Sec. 2.3, Fig. 2) challenges routine prompt-tuning practice, and the RDKit-vs-LLM contrast gives practitioners a concrete default: compute descriptors deterministically.
LLM outputs verified against a deterministic reference
Prompts and code are published
Weaknesses
No statistical significance testing
Model identity inconsistency
Ground truth from a single assay
Missing classic featurization baseline
Reproducibility gaps in stochastic and tuning stages
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This CIBB 2026 paper (arXiv:2609.03635) asks a practical question: does prompt phrasing matter when LLMs are embedded in a drug toxicity prediction pipeline? The authors (Robert Gordon University) run a two-stage factorial study. Stage 1 elicits fifteen toxicity-relevant features from five LLMs under 20 prompt variants (5 prompt structures x 4 job roles, Table 1). Stage 2 builds 18 datasets by having LLMs compute feature values for 2,294 SMILES (789 toxic, 1,505 non-toxic) from PubChem assay AID 489025, then trains six ML algorithms to predict toxicity, evaluated by AUC on an 80/20 split. Four central claims: (1) inherent LLM output variance outweighs prompt fine-tuning; (2) RDKit-based chemoinformatic extraction substantially outperforms LLM-generated values; (3) swapping LLMs had little impact and no job role was consistently best; (4) the analysis methodology generalizes across bioinformatics prompt types.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does prompt phrasing or assigning a job role matter when using LLMs for toxicity prediction?
According to this study, no systematic benefit was found: random variation between identical prompts produced downstream variance comparable to changing the prompt text or the model, and no job role (Biochemist, Biologist, Chemist, none) was consistently best.
Can LLMs reliably compute chemical descriptors like LogP or molecular weight?
The paper's RDKit comparison found that LLM-generated feature values led to substantially worse toxicity-prediction AUC than deterministic chemoinformatic calculation, so for descriptor values the practical recommendation is to use tools like RDKit rather than asking an LLM.
What data did the study use?
2,294 SMILES strings (789 toxic, 1,505 non-toxic) from PubChem bioassay AID 489025, a THP-1 cell cytotoxicity assay. Six standard ML models were trained on 80% and evaluated by AUC on 20%, with SMOTE for class balancing.
Is the code or data for this paper available?
Yes. The paper links a public GitHub repository (CIBB-Analysis-of-Prompt-Engineering-for-Drug-Toxicity-Prediction) and prints all five prompt templates verbatim in Table 1; the underlying compounds come from the public PubChem database.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 are images in the PDF; exact AUC values, feature-frequency percentages, and importance magnitudes are available only via captions and body text, so numeric magnitudes could not be independently verified from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.