PFArena: Benchmarking Language Models for Protein Modification
Yawen Ouyang, Xinbo Zhang, Ziyuan Ma, Yixin Wu, Wenbin Liao, Feiran Zhang, Wenjie Li, Lihao Wang, et al.
cs.AI · q-bio.BM
Quick verdict
Well-built 4-task benchmark: PLMs lead open single-mutant search, LLM agents lead evidence-rich ranking; single-run data and contamination limit certainty.
Summary
PFArena is a benchmark for model-assisted protein modification that evaluates six zero-shot protein language models (ESM-2, ProGen2-base, ProSST-2048, S3F, S3F-MSA, VenusREM), six frontier LLMs (GPT-6 Astra, Claude Opus 5, Gemini 3.1 Pro, Kimi K3, GLM-5.2, DeepSeek-V4-Pro) and five LLM-based agents (Biomni- and AMixKit-based) across four tasks: single-mutant generation (T1) and three multi-mutant ranking tasks with increasing evidence (T2 measurement-free, T3 anchor-informed, T4 single-mutant-informed). Data derive from 202 public assays (ProteinGym, MaveDB, MegaScale, FLAb, Human Domainome, CombinGym) with documented harmonization, quality control and order-leakage checks; metrics cover global (Spearman), top-weighted (NDCG), peak (NMS@K) and coverage (Recall@K) performance. PLMs lead T1 (VenusREM Recall@40 0.0427 vs 0.0254 for the best LLM), while agents dominate T2–T4, most clearly in T4 (AMixKit+Claude Spearman 0.6462 vs ~0.35 for PLMs). Analyses attribute the LLM gains largely to additive use of single-mutant scores (performance–additivity correlation ρ=0.733 for LLMs vs 0.378 for PLMs) and show collapse on non-additive assays (LLM Spearman 0.015 on 7/29 assays); all families degrade with mutation depth and search-space size. Main limitations: potential pretraining contamination of the public assays, single-run evaluation without variance estimates, refusal handling that replaces missing outputs with a random baseline, and in-house AMixKit/AMix-2.1 details not yet released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- No single model family dominates: PLMs lead open-ended single-mutant generation (T1) via protein-specific priors, while LLM-based agents lead multi-mutant ranking, most clearly when component-level fitness data are provided (T4).
- The LLM/agent advantage in T4 derives largely from additive use of measured single-mutant scores — performance correlates with an additivity diagnostic (ρ=0.733 LLMs vs 0.378 PLMs) and collapses on non-additive assays (LLM Spearman 0.015 on 7/29 assays).
- All model families degrade with larger search space and mutation depth: T1 top-40 recovery is under 5% even for the best method, and ranking accuracy falls with mutation count (ρ=−0.430, P=4.75e−9).
- Feedback-guided multi-round generation and test-time scaling improve LLM inference: multi-round lifts mean NMS@40 by 0.0222 and Recall@40 by 0.0150, and RRF aggregation reaches 0.7847 NMS@40 vs 0.7696 sample averaging at n=16.
Strengths
Task interfaces mirror real experimental decisions
§2.1/Fig 2 define four settings along an action-space × evidence grid (open generation vs fixed-pool ranking; none → anchor → full single-mutant context), instantiated on 123/74/67/29 assays — a design that maps directly onto design–build–test–learn workflows rather than arbitrary splits.
Above-average data construction and QC discipline
§2.3 harmonizes seven public sources into unified DMS_score semantics with documented transformations (−log10(Kd), log-ratio expression, per-assay z-scoring, sign reversal); §2.3.2 runs three QC groups including an explicit order-leakage check (monotonicity/rank correlation between row position and DMS_score) and excludes tie-heavy instances.
Multi-metric evaluation avoids single-number collapse
Mechanistic diagnostics go beyond the leaderboard
Transparent failure handling and matched-query sensitivity analysis
Weaknesses
Structural contamination risk is unaddressed beyond acknowledgment
Single-run leaderboard with no uncertainty
Refusal fallback conflates policy with ability
Conflict-of-interest shape: in-house agent stack tops the board with unpublished components
Compressed peak metric and thin T4 sample
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PFArena (arXiv 2609.28921, v1, 24 Sep 2026) positions itself between two evaluation traditions: fixed-list mutation-effect prediction (ProteinGym, FLIP) and end-to-end agentic design (BioDesignBench). Its four task interfaces (§2.1, Fig 2) cross an action-space axis (open-ended generation vs fixed-pool ranking) with an evidence axis (none → one measured anchor → complete single-mutant context): T1 single-mutant generation (123 assays, 594,695 measured mutants), T2 measurement-free ranking (74 assays, 6,288 candidates), T3 anchor-informed ranking (67 instances), T4 single-mutant-informed ranking (29 assays, 2,638 candidates). The headline claims are explicitly dual: (1) divergent strengths — PLMs win T1, agents win T2–T4 with the gap widest in T4; (2) convergent bottlenecks — all families degrade with search-space size and mutation depth. Both are supported by the data presented, with the caveats below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PFArena?
PFArena (arXiv 2609.28921) is a public benchmark for evaluating AI systems on protein modification. It defines four tasks — proposing top single amino-acid mutations (T1), and ranking multi-mutation candidates with no measurements (T2), one measured anchor (T3), or complete single-mutation fitness data (T4) — built from 202 real deep-mutational-scanning assays drawn from ProteinGym, MaveDB, MegaScale, FLAb, Human Domainome and CombinGym. It compares protein language models, general-purpose LLMs, and LLM-based agents under shared, macro-averaged metrics.
Which model performs best on PFArena?
It depends on the task. On open-ended single-mutant generation (T1), the protein language model VenusREM leads (Recall@40 0.0427). On multi-mutant ranking (T2–T4), LLM-based agents lead — AMixKit+Claude Opus 5 and Biomni+GPT top most metrics, with the clearest agent advantage in T4 (Spearman 0.6462 vs ~0.35 for PLMs). Leaders vary by metric: e.g., in T3, Biomni+Claude wins Spearman (0.4511) while ProSST-2048 wins peak metrics (NMS@5 0.8620).
Do LLMs beat protein language models at mutation prediction?
Not uniformly. PFArena finds PLMs are better when no target-specific fitness data exist (open-ended single-mutant generation, T1), while LLMs and agents pull ahead once measured single-mutant fitness values are supplied (T4). The LLM advantage is largely additive reuse of those values: on the 7 of 29 T4 assays where combinations are non-additive, LLM Spearman drops to 0.015 versus 0.242 for PLMs.
Are PFArena results reliable given data contamination?
Partial reliability caveats apply. All 202 assays are public and may appear in LLM pretraining data, so memorization cannot be excluded — the authors state this in the Limitations section. Each LLM/agent was also run only once (no variance estimates), and refused or unparseable outputs were replaced by a deterministic random baseline. The code and benchmark suite are stated to be released, enabling independent checks.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction contains omitted sections (Tables 7, 8, 12; parts of §4.2.2, §5.2.2–5.2.3, Appendix B.2); T2/T3 per-metric numbers are cited from Figure 1, §4.2.3 prose, and §5.1.2 aggregates rather than full result tables.
- Figures are available as text only; curve shapes and distributions (Figs 4, 7–11) are judged from captions and inline statistics.
- Model names and versions (GPT-6 Astra, Claude Opus 5, etc.) are as stated in the paper; leaderboard standings will drift with API updates, so this review reflects arXiv v1 (24 Sep 2026).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.