Are You Learning Biological Signal or Shortcuts? Auditing and Mitigating Bias in Protein-Protein Interaction Datasets
Judith Bernett, Anton Spannagl, Joel Ås, Markus List, David B. Blumenthal
q-bio.MN · cs.LG
Quick verdict
Claims PPI datasets carry exploitable biases and offers ILP splitting/sampling to remove them; convincing audit, but only an RF baseline evaluated.
Summary
The paper argues that much of the reported performance of sequence-based PPI prediction reflects dataset-construction artifacts rather than learned biology. Using the G-AUDIT detectability/utility framework, the authors jointly audit eight shortcut attributes — topology, self-interactions, taxonomy, sequence and embedding similarity, and GO-BP/MF/CC overlap — across nine datasets: HIPPIE, IntAct, five STRING human-physical subsets, PINDER, and a newly constructed PDB-Dimers. A Random Forest on mean-pooled ESM-2/ProtT5 embeddings serves as a canary expected to sit at chance on unbiased data: with random splitting, balanced AUPRC reaches 0.81-0.98, driven by degree/topology shortcuts and, in PDB-Dimers, self-interactions (80% of positives). Two integer linear programs are introduced: a data-loss-minimizing similarity-aware splitter and a negative sampler matching positives' self-interaction counts, per-protein degrees, taxon-pair distribution, and mean GO-BP Jaccard. Combined, they reduce most datasets to balanced AUPRC 0.53-0.58 (chance 0.5), while PDB-Dimers and PINDER remain at 0.77 because negative self-interactions cannot be matched. Sampling negatives from high-confidence non-interactors backfires on STRING subsets (AUPRC 0.62-0.66) since such pairs rarely share GO terms. Limitations: two embedding models, linear detectability probe, ILPs on subsampled pools, no deep-learning models evaluated, single runs without significance testing.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Random train/test splitting of PPI data lets a trivial Random Forest baseline reach near-perfect scores (balanced AUPRC 0.81-0.98) by exploiting degree/topology shortcuts available only through train-test protein overlap.
- Even after leakage-free, similarity-aware splitting, datasets retain shortcut structure from self-interactions, taxonomic identity, and GO functional relatedness, with prevalence depending on the source database.
- The proposed ILP split plus bias-minimizing ILP negative sampling drives seven of nine datasets to near chance (balanced AUPRC 0.53-0.58, realistic 0.08-0.13), while PDB-derived datasets remain at 0.77 because negative self-interactions cannot be matched.
- Sampling negatives from a high-confidence non-interactor pool amplifies the functional-relatedness shortcut on STRING subsets (balanced AUPRC 0.62-0.66), since confirmed non-interacting pairs rarely share GO-BP annotations.
- PDB-derived structural datasets encode a fundamentally different, homo-dimer-heavy interaction population, limiting their use for augmenting assay-based PPI datasets with structural information.
Strengths
First joint multi-shortcut, multi-source audit
Prior studies examined one bias on one dataset (Park & Marcotte 2012 topology; Hamp & Rost 2015 similarity; Hallee et al. 2026 taxonomy). Section 2.2 quantifies eight attributes via G-AUDIT across nine datasets spanning assay-based (HIPPIE, IntAct, STRING channels) and structure-derived (PINDER, new PDB-Dimers) sources, revealing source-dependent shortcut dominance (e.g., GO overlap highest in STRING-900/database/textmining).
Principled ILP formulations replace ad hoc procedures
The split ILP (Table 3) minimizes discarded cross-cluster PPIs under explicit split-size constraints, fixing two documented drawbacks of Bernett et al. (2024)'s KaHIP k=3 scheme (no size control, heavy data loss; Figure 3A). The sampling ILP (Table 4) matches four bias statistics with per-term worst-case normalization, log-weighted dynamic weights, and a degree cap (C2), with proven optimality per solved pool (Section 2.4).
Counterintuitive, actionable findings
Honest, unusually complete limitation section
Balanced vs realistic test-set design
Weaknesses
No deep PPI model evaluated
No uncertainty quantification anywhere
Weak detectability probe
Pipeline fails exactly where bias is worst
Adoption friction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper tackles a question previously studied only piecemeal — Park & Marcotte (2012) on topology, Hamp & Rost (2015) on sequence similarity, Hallee et al. (2026) on taxonomy — namely: which shortcuts infest which PPI data sources, how do they interact, and can dataset construction be fixed in a principled way? The authors audit eight candidate shortcut attributes on nine datasets: dedicated interaction databases (HIPPIE v3.0; IntAct, 1,082,410 PPIs of which 618,796 human), five slices of STRING v12.0 human-physical (full, score>=900, experimental/database/textmining channels), and two structure-derived sets (a re-deduplicated PINDER, 2,319,564 -> 135,398 interactions; newly constructed PDB-Dimers, 97,344 -> 38,758 two-chain assemblies). Central claims: (1) random splitting makes near-perfect baseline scores achievable via degree/topology shortcuts; (2) leakage-free splitting still leaves self-interaction, taxonomy, and GO-overlap shortcuts whose dominance varies by source; (3) ILP splitting plus bias-matched negative sampling drives seven of nine datasets to chance; (4) high-confidence negatives amplify the functional-relatedness shortcut; (5) PDB-derived data describe a homo-dimer-heavy interaction population unsuitable for augmenting assay data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Why do PPI prediction models report such high accuracy — are they really learning binding?
Often not. This paper shows that random train/test splits let models recognize proteins seen during training, and randomly sampled negatives differ from positives in graph degree, species, self-interactions, and functional similarity. Controlling these artifacts drops a simple Random Forest baseline from balanced AUPRC 0.81-0.98 to 0.53-0.58 (chance is 0.5), so much published performance reflects bias exploitation rather than learned binding mechanisms.
What are high-confidence negatives in PPI datasets, and why can they hurt model training?
They are protein pairs experimentally tested together at least three times but never reported to interact (here: 25 million human IntAct pairs). Intuitively cleaner labels, but such pairs rarely share biological-process annotations (only ~5% have any GO-BP overlap), so training on them teaches the model that functionally unrelated proteins do not interact — amplifying a functional-similarity shortcut and inflating balanced test performance from 0.53-0.58 to 0.62-0.66 AUPRC on STRING-derived data.
Is the code and data from this PPI bias paper available?
Yes. The splitting and bias-analysis pipeline is an open Nextflow workflow at github.com/bionetslab/ppi-splitting-pipeline, and all datasets, splits, and results are on figshare (DOI 10.6084/m9.figshare.33407437). Caveat: full replication requires a Gurobi academic license, GPU embedding with ESM-2/ProtT5, and roughly 3.5 days of cluster compute.
Do these results mean PPI prediction is a hopeless task?
No. The authors deliberately use a weak Random Forest baseline that should sit at chance on truly unbiased data, so residual performance above chance measures bias, not the task's ceiling. Whether modern deep learning models can learn genuine binding on these de-biased splits is explicitly left to future work.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure panels (Figures 2-5, A1-A2, C1) extracted with interleaved/garbled numeric annotations; figure-level readings were cross-checked against the prose and Supplementary Table A1, but some per-panel value attributions remain approximate.
- Supplementary Table A1 row labels interleave with metric values in the PDF extraction; per-row attributions were verified against in-text quotes (e.g., the 0.53-0.58 balanced AUPRC range and the 0.62-0.66 HCN range).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.