CoMPASS: Collaborative Molecular Property Prediction via Adaptive Small-Large Model Synergy
Wentao Li, Jiangjie Qiu, Yijun Li, Leyi Zhao, Xiaonan Wang
cs.LG · cs.AI
Quick verdict
Improves its own GAT anchor on 8 benchmarks via retrieval calibration plus a bounded LLM gate; credible ablations, but unpaired baselines temper the headline.
Summary
CoMPASS is a molecular property prediction framework in which a graph attention network (GAT) remains the calibrated predictor and a large language model contributes only bounded, evidence-conditioned corrections. For each molecule, the system retrieves K analogues from the training split (combining Morgan fingerprint, graph-embedding, and attention-overlap similarity), blends a similarity-weighted label estimate with the GAT output using a validation-fitted weight, and prompts the LLM with the target, attention evidence, and retrieved cases. The LLM's numeric proposal passes through a deterministic agreement-aware gate driven by uncertainty, retrieval conflict, attention consistency, and directional agreement, which caps the correction. Across eight MoleculeNet-style benchmarks, CoMPASS improves its paired GAT anchor everywhere, with ROC-AUC gains of +0.096 on BACE and +0.080 on BBBP but only +0.003 on CYP450 and +0.009 on Tox21; molecule-bootstrap intervals exclude zero on six of the eight datasets. Ablations attribute most of the gain to retrieval calibration plus gated fusion rather than prompting alone. Key limitations: external baselines (GIMLET, Graphormer, MolRAG) are copied from prior papers rather than retrained under the same split, the GAT anchor may be under-tuned, results average only three seeds, and the gate's hyperparameters are not fully specified.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CoMPASS improves its paired GAT anchor on all eight benchmarks while limiting LLM intervention in high-confidence regimes (Tables 3-5, Fig. 3).
- The gains arise from validation-calibrated retrieval plus bounded gated fusion rather than prompting alone (Table 6: LLM+GAT without retrieval moves BACE ROC-AUC only 0.756→0.759).
- The collaboration interface generalizes across four LLM backbones (Table 7) and to a GraphGPS anchor (Table 8: BACE +0.098).
- LLM reasoning should augment calibrated predictors through evidence-grounded, capped corrections rather than direct output replacement, as illustrated by the success/failure analysis (Fig. 4).
Strengths
Honest, anchor-relative claim discipline
The Results section explicitly states CoMPASS does not claim universal dominance: CYP450 is framed as low-headroom (+0.003, CI [-0.000, 0.006]), Graphormer stays ahead on Tox21 (0.759 vs 0.744), and Figure 4 shows failure cases where misleading retrieval pulls correct GAT predictions across the threshold.
Clean evaluation hygiene
Retrieval is restricted to the training split, the query is excluded by SMILES match, K ∈ {4,6,8} and the blend weight λ are selected on validation only (Eq. 10), and test labels are used solely for final evaluation (Experimental Setup).
Ablations directly support the mechanism claim
Auditable inference trail
Interface robustness checks
Weaknesses
External baselines are not paired
Possibly under-tuned anchor inflates headroom
Thin statistical reporting
Gate is under-specified and partly circular
Crude multi-task correction
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CoMPASS addresses molecular property prediction by keeping a graph attention network (GAT) as the calibrated anchor and allowing a large language model only bounded, evidence-conditioned corrections. The pipeline (Fig. 1, Table 1) has four stages: GAT inference producing prediction, confidence, and attention evidence; retrieval of K training-split analogues via combined Morgan fingerprint / graph-embedding / attention-overlap similarity (Eq. 5); a validation-fitted blend of the GAT output with a similarity-weighted label estimate (Eqs. 9-10); and a deterministic agreement-aware gate (Eq. 16) that interpolates between the calibrated base and the LLM proposal. The claims are deliberately anchor-relative: CoMPASS improves its paired GAT anchor on all eight benchmarks while restricting LLM influence in high-confidence regimes, and the ablations attribute gains to retrieval calibration plus gated fusion rather than prompting (Table 6).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CoMPASS in molecular property prediction?
CoMPASS is a framework where a graph attention network (GAT) makes the calibrated prediction, a retriever finds similar molecules from the training set, and a large language model proposes corrections that pass through an agreement-aware gate which caps how much the final prediction can move — the LLM augments rather than replaces the graph model.
Does CoMPASS beat Graphormer, GIMLET, or MolRAG?
Not everywhere. Against numbers copied from prior papers, CoMPASS reports higher ROC-AUC than Graphormer on BACE (0.852 vs 0.776) and BBBP, but Graphormer stays ahead on Tox21 (0.759 vs 0.744), GIMLET's ESOL RMSE (1.132) beats CoMPASS (1.201), and GIN beats it on Lipo (0.781 vs 0.790). Those baselines were not retrained on the same data splits, so the comparison is indicative rather than controlled.
Is code and data available for CoMPASS?
The abstract links code at github.com/littlepeachs/CoMPASS, and all eight benchmarks (BACE, BBBP, ClinTox, CYP450, HIV, Tox21, ESOL, Lipo) use public fixed DeepChem splits. However, the paper text does not list the gate's internal constants (floors, caps, scales, evidence-map weights), so full reimplementation likely requires the repository.
What are the main limitations of CoMPASS?
Per the authors: LLM inference cost restricts it to offline screening rather than billion-molecule libraries; misleading retrieval neighborhoods can cause over-correction; and results use fixed benchmark splits rather than resampling. A reviewer would add: external baselines were not retrained under the same split, the GAT anchor may be under-tuned, only three seeds are averaged, and gate hyperparameters are not fully specified.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-4 are referenced in prose but their rendered content (heatmap cells in Fig. 2, confidence-bin curves in Fig. 3, molecule case values in Fig. 4) is absent from the extracted text, so figure-based statements rest on the paper's prose descriptions only.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.