Self-Supervised Pretraining of Molecular Graph Encoders with LeJEPA
Michał Kulczykowski, Rafał Łabędzki
q-bio.QM · cs.LG · stat.ML
Quick verdict
Claims LeJEPA pretraining improves molecular representations but not finetuning; rigorously honest nulls, though gains rest on small shifts and one benchmark.
Summary
The paper adapts LeJEPA — a predictor-free joint-embedding architecture regularised by Sketched Isotropic Gaussian Regularisation (SIGReg) — to molecular graphs, pretraining GPS and D-MPNN encoders on ~2.9M ChEMBL molecules and evaluating on the Wong et al. antibiotic dataset and ogbg-molhiv under multi-seed, bootstrap-based protocols. A frozen probe on the pretrained embeddings clearly beats random initialisation on both tasks (molhiv ROC-AUC 0.788 vs 0.665), reaching the published self-supervised band, but this edge does not robustly survive finetuning: the significant antibiotic scaffold-split gain (+0.041 AUPRC, p=0.010) fails to replicate across five partitions (pooled +0.013, p=0.095), and finetuning is null on the random split, molhiv, and the D-MPNN backbone. A width-controlled analysis shows the embedding saturates at ~16–32 effective dimensions; truncating it via PCA and concatenating with a 1024-bit Morgan fingerprint raises molhiv ROC-AUC from 0.803 to 0.830 (p=0.014), while an untrained-encoder control gains nothing — but the benefit appears only on shifted test scaffolds, rests on 130 test actives, and cannot be selected on validation. Descriptor-only random forests also beat the published graph baseline. Limitations include modest absolute scores below published self-supervised methods, a single benchmark for the headline combination result, and the unresolved question of why finetuning fails to exploit the representation. Code and checkpoints are released.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LeJEPA pretraining of molecular graph encoders yields a measurably better representation than random initialisation, shown by frozen probes on two tasks (molhiv ROC-AUC 0.788 vs 0.665; antibiotic linear probe AUPRC 0.159 vs 0.082).
- This representational edge does not robustly convert into finetuning gains: the only significant finetuning result (+0.041 AUPRC on the canonical antibiotic scaffold partition, p=0.010) fails to replicate across five partitions (pooled +0.013, p=0.095), and finetuning is null on the random split, ogbg-molhiv, and the D-MPNN backbone.
- Embeddings saturate at ~16–32 effective dimensions, and a PCA-32 truncation concatenated with a 1024-bit Morgan fingerprint raises ogbg-molhiv test ROC-AUC from 0.803 to 0.830 (p=0.014), with an untrained-encoder control gaining nothing.
- The combination gain reflects robustness to scaffold shift (validation-to-test degradation −0.017 vs −0.042 for fingerprint alone), not extra in-distribution fit.
- Descriptor-only tree ensembles match or exceed graph-based Chemprop on the antibiotic task, establishing that a graph network is not necessary there.
Strengths
Replication-minded statistics on the fragile result
The single significant finetuning gain (canonical antibiotic scaffold split, ΔAUPRC +0.041, p=0.010, Table 4) is deliberately re-tested on four further leakage-free scaffold partitions; pooled effect +0.013 (95% CI [−0.006, +0.032], p=0.095) and per-partition spread −0.033 to +0.054 (mean +0.010±0.038, t=0.60) are reported as non-replicating rather than buried.
Strong simple baselines that reframe the task
Section 3.1 / Table 1: a descriptor-only RandomForest reaches pooled AUPRC 0.453, significantly beating both retrained Chemprop (0.407; Δ+0.046, p=0.006) and the published Wong et al. Chemprop (0.364), establishing that the graph network is not necessary on the antibiotic task before any pretraining claims are made.
Causal controls on the positive headline
Width-controlled, split-aware representation analysis
Full release
Weaknesses
Headline positive rests on one narrow shift
Internal numeric inconsistency between abstract and body
No same-protocol comparison against established molecular SSL
No sensitivity analysis on the pretraining recipe
Compact encoder limits external validity
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper adapts LeJEPA — Balestriero and LeCun's predictor-free JEPA regularised by SIGReg — to molecular graphs, reportedly for the first time, and asks the transfer question directly: does pretraining on ~2.9M ChEMBL molecules improve downstream molecular property prediction? Two encoder backbones (GPS graph transformer, ~2M params; Chemprop-style D-MPNN), two downstream tasks (the Wong et al. 39,312-compound S. aureus antibiotic dataset with 1.3% actives; ogbg-molhiv with its fixed scaffold split), and a multi-seed, bootstrap-based protocol. Central claims: (1) frozen probes show a real representational gain (molhiv 0.788 vs 0.665 ROC-AUC; antibiotic linear probe 0.159 vs 0.082 AUPRC); (2) that gain does not robustly survive finetuning — the one significant result (+0.041 AUPRC, canonical antibiotic scaffold split) fails across five partitions; (3) the edge is recoverable at the feature level: PCA-32 truncation plus a 1024-bit Morgan fingerprint lifts molhiv ROC-AUC 0.803→0.830 (p=0.014).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Does LeJEPA pretraining improve molecular property prediction?
At the representation level, yes: a frozen probe on the pretrained embedding clearly beats the same architecture at random initialisation on both tasks tested (ogbg-molhiv ROC-AUC 0.788 vs 0.665). But it does not robustly improve finetuned models — finetuning gains were null on the random split, on ogbg-molhiv, and with a D-MPNN backbone, and the one significant scaffold-split gain (+0.041 AUPRC) did not replicate across five scaffold partitions (pooled +0.013, p=0.095). The practical value appears when the embedding is truncated and concatenated with a Morgan fingerprint instead of finetuning.
What is LeJEPA and SIGReg?
LeJEPA is a self-supervised joint-embedding predictive architecture (JEPA) proposed by Balestriero and LeCun that removes the usual anti-collapse machinery (momentum teacher, stop-gradients, predictor network) and instead regularises the embedding distribution toward an isotropic Gaussian using SIGReg — Sketched Isotropic Gaussian Regularisation, which compares 1024 random one-dimensional projections of the embedding batch to a standard normal via an Epps–Pulley statistic. By the Cramér–Wold theorem this drives the full embedding distribution toward N(0, I), making collapse impossible.
What ROC-AUC does this paper achieve on ogbg-molhiv?
Several numbers under different configurations: a frozen pretrained embedding plus random forest reaches 0.788±0.003 (vs 0.665±0.004 for random init, and 0.807±0.004 for a 2048-bit Morgan fingerprint); a finetuned pretrained GPS reaches 0.722±0.010 vs 0.717±0.014 from scratch (not significant); the strongest configuration, a 1024-bit Morgan fingerprint plus a 32-dimensional PCA truncation of the embedding, reaches 0.830±0.004 vs 0.803±0.004 for the fingerprint alone (Δ+0.027, p=0.014), with the gain appearing only on shifted test scaffolds.
Is the code and data for this paper available?
Yes. The authors state that all pretraining code, configurations, and final checkpoints are released, and both evaluation datasets are public: ogbg-molhiv from the Open Graph Benchmark and the Wong et al. 2024 Nature antibiotic-activity dataset (39,312 compounds, 512 active against S. aureus).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 are available only through their captions in the extracted text; visual claims (e.g., the Figure 5 embedding-manifold organisation by LogP and ring count) could not be inspected directly.
- The abstract quoted in the task metadata is cut mid-sentence ('raises ogbg-'), but the complete abstract is present in the full paper text, so no content was lost.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.