Rethinking Class Imbalance for Single-Cell Foundation Models: A Systematic Benchmark Across Architectures and Long-Tail Loss Functions
cs.LG
Quick verdict
First systematic loss-level benchmark for class imbalance in single-cell foundation models; rigorous controlled design, but statistics thin and only 3 seeds.
Summary
The paper presents the first systematic, loss-level benchmark of class-imbalance handling for single-cell foundation models. Six losses (cross-entropy, weighted CE, class-balanced, focal, LDAM, logit-adjusted softmax) are evaluated on three backbones (scGPT, scBERT, Geneformer) across three datasets (Multiple Sclerosis, Zheng68K, human Pancreas) in 162 controlled runs (three seeds per configuration), using each backbone's recommended fine-tuning protocol and default loss hyperparameters. Under plain cross-entropy, accuracy exceeds Macro-F1, which exceeds rare-class recall, in all nine (backbone, dataset) settings. Class-balanced loss (mean rank 2.44) and LDAM (2.56) are the most consistent choices; logit adjustment trades rare-class precision for recall. Among 'recoverable' rare classes, the reweighting gain over cross-entropy correlates negatively with absolute training-set size (r=−0.77, p<0.001, n=19; robust across exclusions, r=−0.53 to −0.78). Two MS classes with only three training cells (oligodendrocyte C, phagocyte) fail under every loss and backbone; confusion patterns, neural-collapse ratios (NC1), and nearest-neighbor analyses attribute this to embedding geometry, and linear probes (F1 0.175–0.465) show discriminative signal survives unused in the frozen representations. Limitations: only three seeds with sparse significance testing and no confidence intervals, three human datasets and three backbones, a scBERT reproduction scoring 0.639 versus 0.858 Macro-F1 in prior work, and default hyperparameters that may under-serve some losses. Code is stated available on GitHub.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The gap between overall accuracy, Macro-F1, and rare-class recall under plain cross-entropy is consistent across all nine (architecture, dataset) settings, driven by dataset structure rather than pretraining.
- Rare-class failure splits into two regimes with distinct embedding-geometry signatures: loss-sensitive classes that the right loss can recover, and severely entangled classes (three training cells) that fail under every evaluated loss and architecture.
- Among recoverable classes, reweighting benefit is predicted by absolute training-set size (r=-0.77, n=19), not by relative frequency share or dataset imbalance ratio.
- Class-balanced loss and LDAM are the most consistent choices across all nine settings (mean ranks 2.44 and 2.56), while logit adjustment trades rare-class precision for recall rather than improving both.
Strengths
Genuinely controlled loss-axis design
162 runs with fixed train/test splits, a shared top-3000 HVG gene panel per dataset across backbones, and loss terms computed from fixed precomputed training statistics (Section 3); each backbone uses its own recommended fine-tuning protocol rather than a forced uniform one, isolating the loss axis cleanly.
Mechanistic diagnosis beyond leaderboard metrics
The two-regime finding is triangulated: phagocyte errors are absorbed into oligodendrocyte A and a mixed glial type for 91-97% of cases across all three backbones (Section 5.2), NC1 trace ratios separate the entangled classes (oligodendrocyte C 1.19-1.79 vs phagocyte 0.17-0.32), and non-foundation 5-NN/RF baselines fail identically on these classes (Section 5.1), ruling out an FM-specific artifact.
Robustness analysis for the headline correlation
Quantified trade-offs and calibration nuance
Linear-probe result localizes the bottleneck
Weaknesses
Thin inferential statistics
Unvalidated scBERT configuration divergence
Default hyperparameters may under-serve some losses
Pooled correlation across datasets
Narrow external validity and a preprocessing approximation
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: Rethinking Class Imbalance for Single-Cell Foundation Models
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Which loss function works best for imbalanced single-cell cell-type classification?
In this benchmark spanning scGPT, scBERT, and Geneformer on three public datasets, class-balanced loss (average rank 2.44 of 6) and LDAM (2.56) were the most consistent choices across all nine architecture-dataset settings. Logit adjustment raised rare-class recall but lowered precision. Classes with very few training cells (about 3) failed under every loss tested.
Can loss reweighting rescue rare cell types that a foundation model keeps misclassifying?
Only partly. The paper finds rare classes split into two regimes: loss-sensitive classes gain from reweighting - more so when they have fewer absolute training cells (r=-0.77) - while severely entangled classes with roughly three training examples fail under all six losses and all three backbones. Embedding-geometry checks (the NC1 trace ratio and nearest-neighbor confusion patterns) can flag such classes before any loss is chosen.
What data and how many experiments are behind this paper?
162 controlled fine-tuning runs: 3 foundation models (scGPT, scBERT, Geneformer) x 3 public datasets (Multiple Sclerosis, Zheng68K, human Pancreas) x 6 long-tail loss functions x 3 seeds, evaluated with accuracy, Macro-F1, rare-class recall/precision, rare-class AUPRC, plus embedding-geometry and linear-probe diagnostics.
Is the code and data for this benchmark available?
The paper states that implementation and experiment configurations are available at github.com/jz890/sc-imbalance, and all three datasets are public (MS from Schirmer et al. 2019, Zheng68K from Zheng et al. 2017, Pancreas from Baron et al. 2016). Experiments ran on NVIDIA RTX 4090 GPUs with bfloat16 mixed precision.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are only partially recoverable from text extraction; figure-level details were inferred from captions and in-text numbers, not from the rendered plots.
- Code repository availability (github.com/jz890/sc-imbalance) is as stated in the paper and was not verifiable in this offline review.
- The abstract text appears truncated mid-sentence in the extracted text ('...trades rare-class precision for re'); the body text appears complete, so impact on the review is minimal.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.