Biology-in-the-loop: Amortized Adaptive Hit Discovery in CRISPR Screens
Carl Edwards, Edward De Brouwer, Xiner Li, Namkyeong Lee, Ehsan Hajiramezanali, Anne Biton, Sara Mostafavi, Gabriele Scalia
q-bio.QM · cs.AI · cs.CL · q-bio.GN
Quick verdict
Introduces 1,389-screen benchmark + AssayLoop (5.67x random EF) for adaptive CRISPR hit discovery; strong, though 20 filtered test screens limit generality.
Summary
The paper introduces AssayBench-Loop, a benchmark of 1,389 historical CRISPR screens across five phenotype categories, split temporally into 1,349 training, 20 validation and 20 test screens, enabling both evaluation and cross-screen training of acquisition policies for sequential hit discovery. On this resource the authors build AssayLoop: AssayFormer, a transformer-based amortized policy trained on historical screens (BPMF-initialized gene embeddings, supervised fine-tuning, then GRPO reinforcement learning with a context-delta reward), is combined with an LLM warm start — e.g., Gemini-3.1-Pro nominates the first batches before handing off to AssayFormer. A post-trained LLM variant, AssayLLM (Qwen3.6-27B with SFT and GRPO), reaches EF 3.69 standalone. On the 20 temporally held-out genome-wide test screens, AssayLoop attains 5.67-fold enrichment over random selection and recovers 27.7% of hits within roughly 5% of the library, surpassing the strongest standalone LLMs (Gemini 4.71, GPT-5.6 Sol 4.81), AssayFormer alone (4.83), and adaptive-design baselines (BioBO 2.59, Probability-of-Hit 2.41). Performance grows with training-screen count without plateau and transfers under leave-one-phenotype-out training (EF 4.59 vs 4.83). Main limitations: test screens were filtered for genome-wide coverage, signal strength and a minimum LLM performance floor; headline comparisons rest on 20 screens with no variance or significance reporting in the provided text; and validation is retrospective, without a live experimental loop.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- AssayBench-Loop is the first large-scale (1,389-screen, five-phenotype) benchmark enabling both evaluation and cross-screen training of amortized acquisition policies for adaptive CRISPR hit discovery.
- AssayLoop, combining an LLM warm start with the AssayFormer amortized policy, achieves 5.67-fold enrichment over random selection and recovers 27.7% of hits at ~5% library coverage on temporally held-out screens, outperforming standalone LLMs, adaptive-design methods, and AssayFormer alone.
- Acquisition capability amortizes across experiments: performance improves monotonically with the number of historical training screens (no plateau over 1-1,349) and transfers to phenotype categories excluded from training (LOPO EF 4.59 vs 4.83 full training).
- Frontier LLMs do use experimental feedback in context, but their adaptation is limited relative to a policy explicitly trained for feedback conditioning; task-specific post-training (AssayLLM, EF 3.69, +39% over base) narrows but does not close the gap.
Strengths
Benchmark scale genuinely enables amortization
1,389 screens with a 1,349-screen training split is two orders of magnitude beyond GeneDisco (four screens), which is precisely what allows training a shared, history-conditioned policy (Section 2.1-2.2) rather than fitting per-screen surrogates like BioBO or Probability-of-Hit.
Leakage-aware temporal evaluation with measured novelty
The >2021 test holdout is validated by median test-to-nearest-training-screen hit-set Jaccard of 0.064 versus 0.64 within training (Fig. 2E), and the primary EF metric is missing-gene aware, penalizing hallucinated genes and unfilled slots while not punishing valid out-of-library proposals (Fig. 2D, Appendix H Eq. 19).
Unusually broad baseline coverage
Honest negative and limiting results
Mechanistic analysis beyond leaderboard numbers
Weaknesses
Test-set selection is partly model-dependent
No uncertainty quantification on headline margins
Phenotype imbalance between training and evaluation
AssayLLM's standalone value is below cheaper baselines
Possible LLM pretraining contamination of the temporal holdout
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes two coupled contributions. (1) **AssayBench-Loop**: a retrospective benchmark of 1,389 historical CRISPR screens (five phenotype categories; temporal split 1,349 train / 20 validation / 20 test), built on AssayBench, with eight metrics headed by a missing-gene-aware hit enrichment factor (EF) that penalizes hallucinated genes and unfilled slots but not valid out-of-library proposals (Fig. 2D, Appendix H). (2) **AssayLoop**: an amortized sequential-design framework in which AssayFormer — a 4.44M-parameter transformer trained across the 1,349 training screens (BPMF-initialized gene embeddings, supervised fine-tuning, then GRPO with a context-delta reward) — is warm-started by an LLM (Gemini-3.1-Pro or GPT-5.6 Sol) before handoff. AssayLLM (Qwen3.6-27B, SFT on GLM-5.1 traces plus GRPO) tests whether the same principle fits inside an LLM. Headline claims: 5.67x enrichment over random and 27.7% of hits recovered at ~5% library coverage on temporally held-out screens; data scaling without plateau; transfer to unseen phenotypes.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is AssayBench-Loop?
A benchmark of 1,389 historical CRISPR screens across five phenotype categories for evaluating and training adaptive hit-discovery policies. Screens are split by publication date (1,349 training before 2021, 20 validation in 2021, 20 test after 2021) so methods are evaluated on experiments newer than their training data, under a fixed budget of 10 rounds x 100 genes per screen.
What is the difference between AssayFormer, AssayLoop, and AssayLLM?
AssayFormer is a small transformer trained across historical screens to rank which genes to test next given the experiment's description and the results observed so far. AssayLoop lets a general-purpose LLM pick the first batches using biological prior knowledge, then hands off to AssayFormer. AssayLLM is a 27B LLM itself fine-tuned on acquisition trajectories and reinforced with GRPO to act as the policy directly.
How much better is AssayLoop than random selection or an LLM alone?
On the 20 held-out test screens it reaches 5.67-fold enrichment over random selection and recovers 27.7% of hits using about 5% of the gene library, compared with 4.71 for the best standalone LLM (Gemini-3.1-Pro), 4.83 for AssayFormer alone, 1.04 for random, and 2.59 for BioBO, a prior adaptive-design method.
Has this been validated in a real laboratory loop?
No. All results are retrospective: the benchmark replays historical screens with fixed binary hit labels. The authors state that prospective validation in a live experimental loop, richer readouts, and modeling of experimental noise across replicates remain future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt: sampling gaps cover parts of Section 2.2 (handoff specification), Section 2.6, the Discussion opening, Methods 4.4-4.6, parts of Appendix D.4 and E, and the tail of Appendix H (Vendi and pathway-overlap definitions); claims depending on those regions are qualified.
- The absence of statistical tests is asserted only for the visible Table 1 and main text; tests could exist within a sampling gap or in supplementary material not fully included here.
- The handoff round k used in headline AssayLoop results is not visible in the provided excerpt, as the relevant Section 2.2/4.6 passage falls in a sampling gap.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.