A budget-dependent crossover between coverage- and response-based training-set selection for machine-learned interatomic potentials
cs.LG
Quick verdict
Claims a budget-dependent crossover in MLIP training-set selection; careful paired-seed evidence, but only two datasets and one architecture.
Summary
The paper asks whether compact training sets for machine-learned interatomic potentials (MLIPs) should preserve structural diversity or target configurations on which models disagree, and shows the answer depends on the retained-data budget. Using fixed 1400-structure panels from GAP-20 Carbon and pooled revised MD17 (1000 train/200 validation/200 test), the authors compare random sampling, a validation-selected FPS/DPP coverage selector on MACE-MP-0 embeddings, and a retrospective response witness that targets disagreement between a coverage-trained model and a full-data reference, at 1%, 5% and 20% retained budgets, training MACE on each subset with five paired seeds. At 5%, FPS/DPP beats random on all eight absolute-gap endpoints. The witness is worse at 1% and 5% but reverses at 20%, where it lowers direct held-out force errors by 0.46-5.89% relative to coverage across all eight dataset-endpoint pairs, six endpoints falling below the full-data reference. Complementary analyses show frozen MACE-MP-0 error largely tracks embedding novelty, error-based selection underperforms embedding coverage on Carbon-defect fine-tuning, and a learned similarity metric preserves the coverage ranking. Limitations: two datasets, one architecture and one coverage representation; coarse budget sampling leaves the crossover only bracketed between 5% and 20%; and the witness requires a full-data reference model and labels, making it retrospective rather than an acquisition rule.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Retained-data budget changes the selector ordering: FPS/DPP is closer to the full-data outcome at 1% and 5%, while the response witness is closer at 20% (the crossover lies between 5% and 20%).
- At 20%, witness-trained models lower direct held-out force errors by 0.46-5.89% relative to FPS/DPP across all eight dataset-endpoint pairs, with six endpoints below the full-data reference.
- At the 5% budget, FPS/DPP reproduces the full-data outcome more closely than random sampling on all eight absolute-gap endpoints (all paired intervals above zero).
- Frozen MACE-MP-0 force error largely tracks embedding novelty (Spearman 0.86-0.91), and direct error-based selection yields higher Carbon-defect force MAE than embedding coverage (314.1 vs 221.2 meV/A).
- A learned chemical/physical similarity metric preserves the coverage ranking (Spearman 0.98-0.99), indicating the measured attributes carry little information beyond structural coverage.
Strengths
Budget-resolved experimental design
Three matched budgets (10/50/200 structures) with identical splits, architecture and protocol directly test the coupling between subset size and selector choice; the crossover claim is supported on both reference-agreement gaps (Fig. 2) and direct held-out DFT errors (Table 2).
Disciplined, pre-declared protocol
FPS-vs-DPP comparator chosen on validation force MAE with documented tie-breaks (Table S0 Panel D); endpoint masks and transformations fitted only on the 200-structure calibration partition; the witness is built from separate seed-0 models so outcome seeds 1-5 cannot leak into selection.
Honest negative and trade-off reporting
Exceptional reproducibility provision
Informative complementary controls
Weaknesses
Narrow evidence base
Subset-construction variance unquantified
Retrospective witness limits practical scope
Small mean-endpoint effects against correlated endpoints
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Preliminary Review: A budget-dependent crossover between coverage- and response-based training-set selection for MLIPs
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 4 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does the paper's 'budget-dependent crossover' mean?
It means the best way to pick a small training set for a machine-learned interatomic potential changes with how much data you keep: structural-coverage selection (FPS/DPP) approximates the full-data model better when keeping 1% or 5% of the data, but a response-guided selector that targets configurations where a coverage-trained model and a full-data model disagree becomes better at 20% retained data — and at that budget it also gives lower real held-out prediction errors.
Can I use the response-witness method to build a smaller training set from scratch?
Not directly. The witness is retrospective: it requires a model already trained on the full dataset and full DFT labels for the whole pool. The authors position it as a diagnostic for compressing an already labelled pool, not as an acquisition rule for deciding which new calculations to run (that is the role of conventional active learning).
Is code and data available for this paper?
Yes. The authors archive source-data tables for every figure and table, a shared MACE training configuration, a data dictionary and a reproduce.sh script (Zenodo DOI 10.5281/zenodo.22263567; development repo github.com/JIABI/QUESTS_physical_kernel) that regenerates the paired statistics and figures without retraining. The datasets (GAP-20 Carbon, revised MD17, HEA25, MACE-MP-0) are all public.
How large are the improvements from response-based selection?
At 20% retained data, witness-trained models reduced held-out force errors by 0.46-5.89% versus coverage selection (0.164-0.167 meV/A on the mean endpoints), with larger gains on high-force-tail and high-novelty-mask endpoints. Six of eight endpoints fell below even the full-data reference, though by small margins (about 0.11-0.93%). At 1% and 5% the witness was clearly worse than coverage.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with two marked [...sampling gap...] regions (within Methods Section 2.3 endpoint definitions and within Discussion Section 4.2-4.3); this review only asserts claims backed by captured content.
- Figures appear only as extracted captions and axis labels; trend judgments rely on the reported table values (Tables 1-2, S0-S13), which were captured.
- One extraction artifact noted: the bootstrap resample count renders as '55' in the text, evidently '5^5' with the superscript caret lost; numeric claims here avoid propagating it.
- The abstract in the provided metadata ends mid-sentence at 'These findings establish'; the completing clause appears only in the body text and was not assumed.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.