An open benchmark for machine learning-based polymer property prediction
Robert W. Learsch, Nicholas Liesen, Daniel S. Levine, Anna M. Hiszpanski, Evan R. Antoniuk
cs.LG · cond-mat.mtrl-sci · cs.AI
Quick verdict
Claims graphs beat LLM/descriptor models on new 250k-polymer benchmark; solid open resource, but Task 4 favors PC by construction and most data is simulation.
Summary
The authors introduce PolyBench26, an open benchmark of nearly 250,000 polymer-property datapoints spanning eight properties and four architectures (homopolymers plus alternating, random, and block copolymers), aggregating VIPEA, polyVERSE, and PolyMetriX with 155,930 newly extracted datapoints from OPoly26 molecular-dynamics trajectories. Five model classes are compared on four tasks: zero-shot GPT-4o/GPT-o3-mini, graph-based Polymer Chemprop (PC) and PPG, polyBERT fingerprints, and random forests on RDKit descriptors. Graph-based models achieve the lowest RMSEs in five-fold cross-validation (e.g., Tg 34.7°C for PC versus 70.4°C for GPT-o3-mini), keep this advantage from 300 datapoints upward, and remain flat as repeat-unit complexity grows from five to ten monomers, where descriptor models degrade. On architecture transfer to random/block copolymers, PC wins by two- to threefold, but the baselines were reduced to molar-ratio-weighted averaging of repeat-unit predictions, so the comparison partly reflects input format rather than learned transfer. Most datapoints are simulated rather than experimental; agreement with experiment is only moderate (r² 0.40-0.79), and radius of gyration is admittedly non-physical. Statistics rely on t-tests across five folds without multiple-comparison correction. The full dataset, splits, and training code are open-source, with a hidden test set and leaderboard planned; the resource is valuable, though conclusions favor graph models partly by construction and all models use default hyperparameters.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- PolyBench26 is the first open benchmark combining homopolymers and alternating, random, and block copolymers, with nearly 250,000 datapoints across eight properties from experiment, DFT, and MD.
- Graph-based models (Polymer Chemprop and PPG) deliver the lowest in-distribution prediction errors, as in five-fold CV on Tg (34.7 °C for PC vs 37.9-39.7 °C for descriptor models and 70.4-73.5 °C for zero-shot GPT models).
- Graph-based models retain their advantage from 300 training datapoints upward and stay robust as repeat-unit complexity grows from five to ten distinct monomers, where polyBERT and RF error rises.
- Polymer Chemprop, via weighted-graph (wPSMILES) encoding of architecture, generalizes best to held-out random and block copolymer architectures, with baselines showing roughly twofold (polyBERT) to threefold (PPG, RF) higher RMSE.
- Zero-shot LLM property prediction substantially underperforms polymer-specialized models under the tested prompting conditions.
Strengths
Scale and architecture coverage beyond prior resources
Nearly 250,000 datapoints across eight properties and four architectures (Table 1), combining VIPEA, polyVERSE, PolyMetriX plus 155,930 newly extracted from OPoly26 MD trajectories — the first open benchmark spanning homo-, alternating, random, and block copolymers, addressing the homopolymer-only gap the Introduction documents in VIPEA/OpenPoly/PolyMetriX.
Physics-aware data work beyond RadonPy defaults
The quantum-corrected Cv workflow (phonon DOS via TRAVIS, Methods) reduces RMSE against experimental Cp from 1.9 to 0.4 J/g/K (~79% reduction, Figs S5/S6), and trajectory filtering (no ring spears, no solvent, ~5,000 atoms) trims 94,436 OPoly26 cells to 38,933.
Task design probes distinct capabilities with honest reporting
Representation-sensitivity analysis
Full open release and anti-overfitting plan
Weaknesses
Task 4 comparison is asymmetric by construction
Benchmark is dominated by simulated data with only moderate experimental validation
Shallow LLM evaluation inflates the three-way framing
Thin statistical inference
No hyperparameter tuning and editorial inconsistencies
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
PolyBench26 (arXiv 2609.27036; LLNL + Meta FAIR) is a benchmark/dataset paper targeting the documented absence of standardized open polymer-property benchmarks. It aggregates ~250,000 polymer-property datapoints across eight properties (Tg, EA, IP, refractive index, density, Cp, Cv, radius of gyration) and four architectures — homopolymers plus alternating, random, and block copolymers (Table 1). Sources: PolyMetriX (7,362 experimental Tg), polyVERSE (~738 DFT homopolymer EA/IP), VIPEA (42,966 copolymer points each for EA and IP), and 155,930 newly extracted from OPoly26 MD trajectories. Five model families are compared on four tasks: zero-shot GPT-4o/GPT-o3-mini, graph-based Polymer Chemprop (PC, weighted PSMILES) and PPG (periodic graphs), polyBERT fingerprint + FNN, and random forest on RDKit descriptors. Central claims: graph models win in-distribution (Task 1), across training sizes from 300 points (Task 2), and under repeat-unit complexity growth (Task 3); only PC natively represents all architectures and transfers best to held-out random/block copolymers (Task 4); zero-shot LLMs substantially underperform.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is PolyBench26?
PolyBench26 is an open-source machine-learning benchmark for polymer property prediction: nearly 250,000 polymer-property datapoints covering eight properties (e.g., glass transition temperature, electron affinity, density, heat capacity) and four polymer architectures (homopolymers plus alternating, random, and block copolymers). It combines existing datasets (VIPEA, polyVERSE, PolyMetriX) with 155,930 new datapoints extracted from OPoly26 molecular-dynamics simulations, and defines four evaluation tasks. Code and data are on GitHub (rlearsch/PolymerBenchmark2026).
Which models perform best on PolyBench26?
Graph-based models win: Polymer Chemprop (PC) and PPG achieve the lowest errors in five-fold cross-validation (e.g., Tg RMSE 34.7 °C for PC vs 37.9-39.7 °C for polyBERT/random forest and 70.4-73.5 °C for zero-shot GPT-4o/o3-mini), keep that lead from 300 training points upward, and stay stable as repeat-unit complexity grows. PC is also the only model that natively handles random and block copolymers, and it transfers best to those held-out architectures.
Is the PolyBench26 data experimental or simulated?
Mostly computed. Only the glass transition temperature set (7,362 points, PolyMetriX) is experimental; the rest are DFT calculations (electron affinity, ionization potential, refractive index) or values extracted from molecular-dynamics simulations (~155,930 points from OPoly26). The MD-derived values agree with experiment only moderately (r² between 0.40 and 0.79 depending on property), and the radius-of-gyration target is explicitly a chain-statistics test rather than a physical observable.
Can I reproduce or extend PolyBench26 results?
Yes, in principle. Datasets, train/test splits, preprocessing scripts, and model training/evaluation code are public on GitHub, training runs on CPU (e.g., ~274 s for PC on 1,800 density points; about 9.5 h for the 37,133-point copolymer density set), and the authors plan a hidden MD-based test set with a leaderboard to enable unbiased comparison of new models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 and S1-S7 could not be inspected; judgments about scaling curves, complexity trends, and Fig 5 error bars rely on captions and the extracted tables.
- Table S2 came through garbled in text extraction; the per-encoding variance numbers cited from it should be treated as approximate.
- The GitHub repository contents (splits, code, license) were not verifiable because network access was disabled.
- Internal cross-references are inconsistent in the source itself (GPT prompts said to be in Tables S4/S5, which are actually training-time tables), so some SI items may be missing or mislabeled.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.