Agentic BAIM-LLM Evaluation (ABLE): Benchmarking LLM Use of Protein Design Tools
Bryce Cai, Geetha Jeyapragasan, Samira Nedungadi, Jake Yukich, Seth Donoughe
cs.AI · cs.CY · q-bio.QM
Quick verdict
ABLE benchmarks LLM agents on BAIM protein-design workflows; well-designed and timely, but single-pathogen, redacted, model-graded, and statistically thin.
Summary
ABLE (Agentic BAIM-LLM Evaluation) is a benchmark from SecureBio assessing whether LLM agents can use biological AI models such as ProteinMPNN and AlphaFold3 in a dual-use viral protein design workflow. The workflow is decomposed into one planning task (ABLE0) and eight execution tasks (ABLE1-8) covering structure retrieval, structure preparation, critical-residue identification, design strategy, tool selection, variant generation, validation-tool selection and variant ranking, each scored 0-1 with automated grading (ABLE0 uses a rubric graded by Claude Sonnet 4). Fifteen frontier models ran N=10 times per task; seven closed-weight models refused all tasks. Among the eight non-refusing models, Claude Sonnet 4 and Gemini 3 Pro lead, and Gemini 3 Pro is the only model with at least one perfect score on every execution task. Eleven human experts completed three tasks: models beat humans on planning (0.88 vs 0.55) and variant selection (0.90-0.92 vs 0.70), while experienced humans beat all models on ProteinMPNN execution (0.94). Main limitations: a single redacted pathogen and workflow; prompts, rubric and scoring withheld for biosecurity, blocking independent replication; no statistical tests; non-uniform exclusion of refusals and early-stopping runs; and a participant model serving as ABLE0 grader.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Seven of fifteen frontier closed-weight models (four Claude, three GPT) refuse every ABLE task, apparently reflecting developer safety filtering of dual-use protein design workflows.
- Among the eight non-refusing models, Claude Sonnet 4 and Gemini 3 Pro score highest, and Gemini 3 Pro alone achieves at least one perfect score on each of the eight execution tasks.
- Frontier models outperform 11 human experts on plan generation (0.88 vs 0.55) and variant selection (0.90-0.92 vs 0.70), while experts with hands-on BAIM experience beat all models on ProteinMPNN execution (0.94 vs 0.76-0.86).
- LLM-BAIM integration lowers expertise barriers for computational protein design, which the authors argue warrants extending governance oversight to computational pathogen design.
Strengths
Fills a genuine evaluation gap with a well-chosen decomposition
ABLE1-8 spans the full dual-use workflow (retrieval, preparation, residues, strategy, tool selection, generation, validation, ranking) with partial credit and per-task prompts (Sec 2.2-2.3, Table 1), going beyond knowledge-only biosecurity tests like VCT.
Algorithmic grading for the execution tasks
ABLE1-8 are scored by deterministic comparison to reference files/lists/metrics rather than LLM judging (Sec 2.3.2), removing judge noise from 8 of 9 tasks; the model-graded exception (ABLE0) is explicitly disclosed.
Verified expert human baseline with survey data
Responsible disclosure and refusal transparency
Failure-mode analysis beyond raw scores
Weaknesses
No inferential statistics
Irreplicable by design, yet proposed as a governance instrument
Participant-as-judge for ABLE0
Counterfactual exclusion rows and shifting denominators
Single-workflow validity and internal numerical inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
ABLE (Agentic BAIM-LLM Evaluation) comes from SecureBio (Cambridge, MA; funded by Sentinel Bio and the Survival and Flourishing Fund) and measures whether LLM agents can wield biological AI models (BAIMs) - here ProteinMPNN and AlphaFold3 - inside a dual-use workflow: computationally redesigning a redacted human-virus protein for enhanced function while preserving structural stability. The workflow is decomposed into ABLE0 (plan generation, rubric-graded) and eight sequential execution tasks ABLE1-8 (structure retrieval, structure preparation, critical-residue identification, design strategy, design-tool selection, variant generation, validation-tool selection, variant ranking), each scored 0-1 (Sec 2.2, Table 1). Central claims: (1) 7 of 15 frontier models refuse all tasks; (2) among non-refusers, Claude Sonnet 4 and Gemini 3 Pro lead, with Gemini 3 Pro the only model reaching a perfect score at least once on every execution task; (3) models beat 11 human experts on planning and interpretation but lose to hands-on experts on tool execution; (4) hence BAIM-LLM integration lowers expertise barriers and warrants capability-threshold governance (Sec 4.3).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the ABLE benchmark?
ABLE (Agentic BAIM-LLM Evaluation) is a benchmark by SecureBio that tests whether LLM agents can operate biological AI models (ProteinMPNN, AlphaFold3) across a nine-task viral protein design workflow: plan generation plus eight execution steps (structure retrieval, preparation, residue identification, design strategy, tool selection, variant generation, validation-tool selection, variant ranking), scored 0-1, mostly by automated checks.
Which LLMs performed best on ABLE?
Among the eight models that did not refuse, Claude Sonnet 4 and Gemini 3 Pro scored highest. Seven closed-weight models (Claude Sonnet 4.5, Claude Opus 4/4.1/4.5, GPT-5/5.1/5.2) refused all tasks. Gemini 3 Pro was the only model to score a perfect 1.0 at least once on every one of the eight execution tasks.
Are LLMs better than human experts at protein design tasks?
On the three ABLE tasks baselined with 11 human experts, models beat humans at writing design plans (0.88 vs 0.55) and picking the best variants from AlphaFold3 outputs (0.90-0.92 vs 0.70), but experts with hands-on tool experience beat all models at actually running ProteinMPNN (0.94 vs 0.76-0.86).
Is ABLE open source or publicly available?
No. The paper withholds the full prompts, the ABLE0 rubric, the pathogen identity and the scoring mechanism for biosecurity reasons, and states no code or benchmark release; only third-party components it uses (Inspect AI, ProteinMPNN) are open source.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Tables arrive flattened by PDF extraction; Grok 4's Table 2 row (ABLE0 refused) and Table 4's columns were mapped by position and cross-checked against Table 3, so minor misalignment inside Table 4's detail cells cannot be fully excluded.
- No truncation observed: the provided text runs complete through Section A.6 and the references.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.