Discover, Falsify, Revise: Auditing Input-Use Claims from Source Code to Predictive Contribution in Agent-Discovered Cell Models
Mengran Li, Bo Li, Chengyang Zhang, Yang Yan, Jinfeng Xu, Zhenchao Tang
cs.LG · q-bio.QM
Quick verdict
CELLAUDIT tests whether agent-discovered cell models truly use perturbation inputs; rigorous and honest, but effects are tiny and sci-Plex gains underpowered.
Summary
The paper introduces CELLAUDIT, a framework for auditing whether AI-agent-discovered cellular-response predictors actually use the perturbation inputs their designs claim. It separates three non-equivalent questions: source consumption (can the input enter the cited computation), fitted dependence (do predictions change under matched input replacement on frozen checkpoints), and target-relevant contribution (does that dependence improve prediction). On BBBC047, an agent-selected predictor reaches Global PCC 0.3153 on held-out Fold 5 versus 0.3142 for a control-profile-only predictor, yet is exactly invariant to compound replacement; source inspection localizes a singleton key-value attention pathway unable to transmit compound identity, and invariance persists after refitting with physically disjoint control wells. A stratified audit of 48 candidates finds 47 with fitted dependence but only 20 with positive target-loss intervals on both folds. Falsification-guided residual revisions recover positive mean compound contributions with small predictive gains. Matched sci-Plex searches with audit-enriched feedback show higher mean performance and contributions, but paired intervals over five trajectories span zero. LINCS-to-LKCP refits show predictive generalization need not imply claim generalization. Main limitations: very small effect sizes, an underpowered discovery comparison (five trajectories, single LLM backbone), and self-auditing of the authors' own agent policy.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Held-out predictive performance does not establish use of supplied perturbation information: an agent-selected BBBC047 model with Global PCC 0.3153 on held-out Fold 5 is exactly invariant to compound replacement.
- Source consumption, fitted dependence, and target-relevant contribution diverge: 47/48 audited candidates change predictions under compound replacement on both folds, but only 20 show positive target-loss intervals on both.
- Falsification-guided residual revisions recover positive mean compound contributions while retaining small predictive gains over a control-profile-only baseline.
- In matched sci-Plex searches, audit-enriched feedback yields higher mean held-out performance and larger mean compound and dose contributions, but paired intervals across five trajectories span zero.
- Refitting fixed LINCS designs on independently acquired LKCP shows predictive generalization need not imply claim generalization: dose contribution persists while compound-identity support does not.
Strengths
Principled three-level decomposition, empirically instantiated
Source consumption vs fitted dependence vs target-relevant contribution is demonstrated, not just asserted: Table 1 shows the contradicted singleton-attention source with Ecmpd exactly 0.0000 while Ectrl is 0.2004 (Fold 5), and §6.1 reports 47/48 candidates with fitted dependence but only 20/48 with positive target-loss intervals on both folds.
Pre-registration-style experimental hygiene
Folds, selection rules, budgets and 32 fixed replacement maps are fixed before search; selected sources/checkpoints are frozen before held-out evaluation. The shared-vs-disjoint control-well intervention (Tables 6-7) isolates the physical-overlap confound: the score model stays exactly invariant in both arms (80/80 zero compound coordinates) while the guided model shows 40/40 positive coordinates.
Rare statistical candor
Validated sensitivity and cost transparency
Weaknesses
Marginal effect sizes throughout
Underpowered central discovery claim
Self-audit and backbone monoculture
Positive means that fail the paper's own registered criterion
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CELLAUDIT targets the prediction-claim gap in agentic model discovery for AI virtual cells: do executable predictors generated by LLM agents actually use the perturbation inputs (compound, dose) their registered designs claim? The framework asks three questions the authors argue are non-equivalent: source consumption (can the input enter the cited computation?), fitted dependence (do predictions change under matched input replacement on frozen checkpoints?), and target-relevant contribution (does that dependence improve prediction?). Central claims: (1) a high-scoring agent-selected BBBC047 model (Global PCC 0.3153, Fold 5) is exactly compound-invariant, statistically matching a control-profile-only predictor at 0.3142; (2) of 48 stratified candidates, 47 show fitted dependence but only 20 show positive target-loss intervals on both folds; (3) falsification-guided revisions restore positive mean compound contributions with small predictive gains; (4) audit-enriched sci-Plex feedback yields higher means, but paired intervals over five trajectories span zero; (5) LINCS-to-LKCP refits show predictive generalization without claim generalization.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CELLAUDIT?
CELLAUDIT is an auditing framework for AI-agent model discovery. It checks whether a generated cell-response predictor actually uses the perturbation inputs its design claims, via three steps: inspecting the source code for a pathway the input can enter, testing whether trained predictions change when that input is replaced, and testing whether that dependence improves prediction of the observed response.
What is the prediction-claim gap the paper describes?
It is the mismatch between what a model's registered design claims about input use and what the implementation or fitted model actually does. Flagship example: a BBBC047 predictor scoring Global PCC 0.3153 whose predictions are exactly unchanged when the compound input is replaced — statistically matching a control-profile-only baseline at 0.3142, meaning the compound information was never used.
Which datasets does the paper use?
Public perturbation datasets: BBBC036 and BBBC047 (paired Cell Painting morphology plus L1000 transcriptomics), sci-Plex3, LINCS Pilot 1 with refitting on LKCP Batch 2, and the Norman et al. CRISPRa double-perturbation dataset.
Does audit-enriched feedback actually improve model discovery?
In matched sci-Plex searches, audit-enriched feedback produced higher mean held-out performance and larger mean compound/dose contributions, but the paired confidence intervals across only five trajectories included zero. The authors themselves call this evidence suggestive rather than conclusive.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction omits parts of §6.3-6.6 (sci-Plex detail and transfer analysis), Figure 2, and several appendices; the 10/50 Fold-5 transfer count comes from a truncated fragment and should be re-checked against the PDF.
- Appendix B's reference threshold (used for Table 1 Above/Below/Across counts) is not in the extracted text and could not be verified.
- No explicit code/data availability statement appears in the extracted text — only a project-page URL — so codeAvailable is set to false.
- Figures could not be analyzed; all numeric judgments rely on tables and body text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.