Same Encoder, Different Winner: A Paired-View Framework for Cell Painting Encoder Evaluation
Tim Treis, Nikita Moshkov, Johan Fredin Haslum, Shantanu Singh, Fabian J. Theis
cs.CV · cs.LG · q-bio.QM
Quick verdict
Paired-view benchmark CP-BG-Bench shows Cell Painting encoder rankings depend on the protocol; convincing design, but single-run and artifacts unreleased.
Summary
The paper introduces CP-BG-Bench, a paired-view evaluation framework for Cell Painting vision encoders. Every single-cell tile is provided in four matched views sharing the same cell centre: raw crop (C), segmented cell (S), and density-augmented variants (CD, SD), so background context and cell density can be ablated or added as controlled interventions. The framework is instantiated on three datasets (JUMP-CP compounds in U2OS, RxRx1 siRNA in HEPG2, RxRx3-core CRISPR knockouts in HUVEC) with three encoders (DINOv3 ViT-B/16, OpenPhenom, SubCell), each LoRA fine-tuned under a multi-positive contrastive objective and evaluated under four protocols: replicate mAP, scIB batch integration, CellProfiler-feature prediction, and cross-batch perturbation recall. The four protocols produce systematically different rankings of the same encoders, and the disagreements decompose along three axes (cell vs background, morphology vs context, within-study vs across-batch). Key results: on RxRx3-core, SubCell with segmented views retains 94% of crop replicate mAP but only 32% of crop R@10; density augmentation recovers 84% of the within-study gap but only 8% of the cross-batch gap; on two of three datasets segmented views predict CellProfiler features better than crops; the C-to-S gap is set by dataset design rather than the encoder. Limitations: a single training run per configuration without seed variance, three datasets with confounded factors, one training recipe, and release links not yet provided.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Four community-standard evaluation protocols produce systematically different rankings of the same Cell Painting encoders, with disagreements decomposing along cell-vs-background, morphology-vs-context, and within-vs-across-batch axes.
- Background signal inflates within-study replicate mAP but largely fails to transfer across batches: SubCell segmented views on RxRx3-core retain 94% of crop replicate mAP but only 32% of crop R@10.
- Density augmentation explains the within-study C-to-S gap (84% recovery on RxRx3-core) but not the cross-batch gap (8% recovery).
- Segmented views predict CellProfiler features as well as or better than crops on JUMP-CP and RxRx3-core, inverting the replicate-mAP ranking and showing the crop advantage is not classical morphology.
- The magnitude of the C-to-S replicate-mAP gap is set by experimental design (0.03 on JUMP-CP to ~0.5 on RxRx1) and is similar across encoders within each dataset.
Strengths
Genuinely controlled intervention design
Holding the central cell fixed across four matched views (C, S, CD, SD; Section 3.2) isolates background and density as manipulable factors — a contrast existing datasets cannot support, directly addressing the shortcut confound raised by Seal et al. [12] and the alternative interpretation of Moshkov et al. [8].
Multi-protocol triangulation with interpretable decomposition
Four protocols (Section 3.6) are mapped onto three axes, and Table 5 gives a reusable catalogue of what each protocol measures, assumes, and how it fails — durable beyond the specific encoders tested.
Headline numbers are internally consistent and robust to batch correction
Unusual transparency about limitations
Licence-aware release plan
Weaknesses
Single training run per configuration, no variance estimates
Abstract/table numeric inconsistency on the central C-to-S gap
Held-out batch doubles as early-stopping set and cross-batch test set
All conclusions conditional on one pipeline
Narrow encoder family
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a methodological benchmark paper. CP-BG-Bench provides every single-cell tile in four matched views sharing the same cell centre — raw crop (C), segmented cell (S), and density-augmented variants (CD, SD), with density encoded as corner-patch intensity proportional to FOV cell count (Sections 3.2, A.1). This is a deliberate controlled intervention on the input, motivated by two prior observations: Seal et al. [12] showing cell count alone predicts bioactivity benchmarks, and Moshkov et al. [8]'s cells-in-context result being equally consistent with exploitation of batch-correlated background. The central claims: (i) four community-standard protocols — replicate mAP, scIB batch integration, CellProfiler-feature prediction, cross-batch recall — rank the same three encoders (DINOv3 ViT-B/16, OpenPhenom, SubCell) differently across three datasets (JUMP-CP, RxRx1, RxRx3-core); (ii) the disagreements decompose along three interpretable axes; (iii) the C-to-S replicate-mAP gap is set by experimental design rather than encoder identity.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CP-BG-Bench?
A paired-view evaluation framework for Cell Painting image encoders: every single-cell crop is provided in four matched versions sharing the same cell centre — raw crop, segmented cell, and two density-augmented variants — so that background context and cell density can be ablated or added as controlled interventions, then scored under four standard protocols (replicate mAP, scIB batch integration, CellProfiler feature prediction, cross-batch perturbation recall).
Why do segmented cells score worse than crops on replicate mAP?
On single-facility datasets (RxRx1, RxRx3-core), background pixels are correlated with plate layout and acquisition conditions, so crops carry perturbation-discriminative shortcut signal. Removing background via segmentation drops replicate mAP substantially (e.g., DINO on RxRx1 falls from 0.756 to 0.251), and that signal mostly fails to transfer to unseen imaging batches.
Which metric should I use to choose a Cell Painting encoder?
It depends on deployment: replicate mAP fits within-study screens such as mechanism-of-action discovery; cross-batch perturbation recall is the most conservative choice when generalising to new plates or sites. The paper's core message is that rankings disagree across protocols, so a single metric should be treated as tentative.
Is the CP-BG-Bench code and data available?
The authors commit to releasing the paired-view JUMP-CP dataset under CC0, deterministic reconstruction pipelines for RxRx1 and RxRx3-core, 36 trained checkpoints, aggregated embeddings, and the full evaluation suite — but in this preprint version the download links are not yet present (footnote 1 says links will be added in a future revision).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Bar-level values in Figures 2, 3, A1-A3 exist in the extraction only as captions and axis labels; all numeric judgments above rely on the paper's tables (1-5, A1-A5), which extracted cleanly.
- The abstract supplied in the task metadata is cut mid-sentence, but the complete abstract appears at the start of the extracted paper text, so no content is missing.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.