Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
Yiming Huang, Ziche Liu, Zhuohang Wu, Yiqian Wang, Junxia Cui, Xinkai Zou, Linjun Mao, Nan Huang, et al.
cs.AI · cs.LG
Quick verdict
Claims frontier LLMs can't yet reliably discover scientific laws; benchmark is well-grounded, but hidden laws are LLM-synthesized and BoN is single-model.
Summary
The paper introduces SCILAWS-BENCH, a benchmark asking whether large language models can genuinely discover scientific laws rather than reproduce memorized formulas. It comprises 118 problems drawn from 381 papers across six disciplines, covering 291 candidate laws and roughly 8 million real data points. Each problem is instantiated twice: SCILAWS-REAL asks models to propose closed-form laws from fixed real observations, scored by held-out predictive fit (SN) relative to published baselines and by source-grounded scientific validity (SV); SCILAWS-PARALLEL converts each problem into an actively queryable, residual-calibrated world whose hidden law is an LLM-synthesized structural variant of a published formula, scored by ordinal structure recovery (SS). Nine frontier models run in a shared ReAct harness with a Python sandbox and 30-turn budget. GPT-5.5 leads all aggregate metrics (SN 50.77, SV 81.84, SS 58.26), yet fit and validity agree on only 54.9% of 3,616 model pairs. A cold-recall audit shows the tasks are far less memorized than AI-Feynman (26.8% vs 55.9% on a matched eight-model panel); published-form recovery falls from 0.85 on canon tasks to 0.59 on moat tasks while full-law recovery stays near 0.08. Best-of-N experiments on GPT-5.4-mini show oracle gains of 4.5–5.3 points from N=5 to N=20 while self-selection gains under 2 points. LLM judges are validated against five human experts (κ 0.77–0.84). Main limitations: LLM-synthesized ground truths, judge overlap with the evaluated panel, no confidence intervals in the main table, and a single-model selection study.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- SCILAWS-BENCH provides 118 paper-grounded law-discovery problems (~8M real data points, six disciplines) whose target formulas are far less memorized than AI-Feynman's (26.8% vs 55.9% cold recall on a matched eight-model panel).
- Predictive fit and scientific validity diverge: tie-aware concordance is only 54.9% across 3,616 model-submission pairs, so fit alone is an insufficient evaluation signal.
- Memorization aids reproducing published forms (0.85→0.59 recovery from canon to moat tiers) but not discovering added structure — full hidden-law recovery stays near 0.08 across tiers.
- Best-of-N pools keep improving (oracle +4.5/+5.3 points from N=5 to N=20) while model self-selection captures under 2 points, making selection the bottleneck.
Strengths
Real-data grounding with measured, not assumed, memorization
Roughly 8M real data points from 381 papers; the cold-recall audit (Section 3.3, Appendix E) quantifies contamination — 30.7% of 1,062 task-model cells recalled, 47.5% of tasks in a 'discovery moat' — and calibrates against AI-Feynman (55.9% vs 26.8% on the same eight OpenAI models; GPT-5.5 71.0% vs 34.7%), backed by a 21,888-call integrity check (E.8).
Fit and validity evaluated separately with an anti-hacking gate
SN is reference-relative (Eq. 1: the best published formula scores 0.5); SV comes from frozen source-grounded rubrics executed by a code-enabled judge, with a global anti-hacking item hard-gating SV to 0 for lookup-table or degree-of-freedom smuggling (C.3). The 54.9% concordance over 3,616 pairs (C.4) empirically justifies the split.
Judge validation meets human–human agreement
Unusual protocol transparency
Weaknesses
Parallel-world ground truth is LLM-synthesized
Judge model sits inside the scored panel
No uncertainty on the headline leaderboard
Selection-bottleneck claim rests on one model
Thin canon stratum
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.01552; UCSD with UCI/UCSB collaborators) introduces SCILAWS-BENCH, a benchmark asking whether LLMs can genuinely discover scientific laws rather than recite memorized formulas. Its claims: (1) 118 problems from 381 papers across six disciplines, 291 candidate laws, ~8M real data points; (2) two complementary settings — SCILAWS-REAL (fixed-record discovery scored by held-out fit SN and source-grounded validity SV) and SCILAWS-PARALLEL (active querying of residual-calibrated worlds scored by ordinal structure recovery SS); and (3) three empirical findings — fit–validity divergence, memorization aiding reproduction but not novel structure, and a best-of-N selection bottleneck.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SCILAWS-BENCH?
A benchmark of 118 scientific law-discovery problems built from 381 published papers across six disciplines, with 291 candidate laws and roughly 8 million real data points. Each problem comes in two settings: SCILAWS-REAL (propose laws from fixed real observations, scored on held-out predictive fit and literature-grounded scientific validity) and SCILAWS-PARALLEL (actively query a residual-calibrated simulated world and recover a synthesized hidden law, scored by ordinal structure match).
Which LLM performed best at discovering scientific laws?
In the paper's nine-model evaluation, GPT-5.5 leads all three aggregate metrics: numeric fit 50.77, scientific validity 81.84, and structure recovery 58.26 (percent scale). No other model consistently ranks second, and domain leaders differ — for example, Gemini 3.5 Flash leads biology fit while GPT-5.5 falls to seventh there.
How does the benchmark avoid LLM memorization of known formulas?
The authors run a data-free cold-recall audit: each model must write the published formula from variable descriptions alone, five samples each, counted as recalled only if at least three structurally match (judged by GPT-4.1). Only 30.7% of the 1,062 task-model cells are recalled; 56 of 118 tasks are recalled by no model ('discovery moat'), versus 55.9% recall on AI-Feynman under a matched eight-model panel (26.8% for this benchmark).
Can LLMs actually discover new scientific laws, according to the paper?
Partially. Models often beat published baselines on tasks whose formulas they cannot recall — GPT-5.5 beats the reference on 54% of moat tasks versus 21% of canon tasks — but recovering a synthesized hidden law's added terms is rare: full-law recovery stays near 0.08 across memorization tiers and reaches only 21% even for GPT-5.5, and self-selection among candidate formulas captures little of the best-of-N headroom.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The provided text is a sampled excerpt with four marked middle gaps: the end of Section 3.1 into C.1, the end of B.4 into C.2, mid-G.1 into G.2, and E.5–E.6 were skipped; judgments depending on those regions were avoided.
- No explicit code/data-release statement appears in the excerpt; only a project-page URL is quoted. A release statement may exist in a sampling gap, so codeAvailable reflects the stated project page.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.