How Good Are Frontier Models at Physics? Expert Re-Grading Reveals Broken Evaluations and Near-Saturation of Leading Benchmarks
Ali Ansari, Haoran Sun, Andy Zeyi Liu, Mark Jabbour, Yongshan Ding, Steven Girvin, Yu He, Sohrab Ismail-Beigi, et al.
cs.AI
Quick verdict
Expert audit: 95% of LLM 'failures' on four physics benchmarks are benchmark/grader defects; convincing, but corrected scores swap question set and grader.
Summary
The paper asks whether frontier LLMs truly struggle with physics, as low benchmark scores suggest. A large Yale and Jump Trading team evaluated GPT-5.6-Sol, Claude Fable 5, and Gemini 3.1 Pro on six physics benchmarks (PHYBench, PRISM-Physics, UGPhysics, HLE-Physics, CMT-Benchmark, CritPt), with faculty and graduate physicists auditing problem statements, reference solutions, and model responses. Of 250 audited rejections across four benchmarks, 57.2% were attributed to benchmark defects and 38.0% to grader errors; only 4.8% (12 cases) were genuine model errors. After correcting solutions and repairing or excluding flawed questions, GPT-5.6-Sol's mean@4 rises from 47.28% to 78.66% on HLE-Physics and from 61.00% to 87.24% on CMT-Benchmark, with 94.44% pass@4 on 54 retained CritPt challenges. The authors conclude the benchmarks are near saturation and understate frontier capability. Limitations: pre- and post-correction scores use different question subsets; corrected grading uniformly uses an HLE-adapted LLM judge; pre-adjudication reviewer disagreement was 28.57%, and 54 of 250 items had only one review; CritPt references were newly derived with AI assistance permitted; Gemini's tool-free corrected scores (e.g., 54.63% on CritPt) are far lower, qualifying near-saturation. No confidence intervals are reported. Anthropic's independent CritPt audit (Fable 5.1, mean@16 88.4%) corroborates the direction.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Of 250 audited model rejections across four benchmarks, 95.20% stem from benchmark defects (57.20%) or grader errors (38.00%); only 12 cases (4.80%) are genuine model errors (Table 2).
- After expert correction, GPT-5.6-Sol's mean@4 rises from 47.28% to 78.66% on HLE-Physics and from 61.00% to 87.24% on CMT-Benchmark, and pass@4 reaches 94.44% on the 54 retained CritPt challenges (Table 1).
- Rule-based evaluators are the dominant failure source on PHYBench and PRISM-Physics (grader errors are 71.43% and 64.86% of audited rejections), chiefly rejecting algebraically equivalent answers.
- Frontier models are near saturation on closed-ended, problem-set-style physics tasks, so current benchmarks cannot measure the remaining gap; new, harder expert-validated evaluations are needed.
- Anthropic's independent CritPt-Corrected audit (Fable 5.1, mean@16 88.4%) corroborates that expert correction dramatically changes measured CritPt performance.
Strengths
Large genuine expert audit with full accounting
250 rejected cases across four benchmarks were audited by ~40 named faculty and graduate auditors (Appendix G), with per-benchmark attribution counts (Table 2), review coverage, and complete conflict-resolution bookkeeping (Tables 3-4: 446 annotations, 196 double-reviewed items, all 56 disagreements adjudicated by a third review).
Audit set is the hardest possible test of the thesis
The pooled audit (Section 2.3, B.3) is restricted to questions GPT-5.6-Sol High failed on every attempt — the subset maximally favorable to the model — yet 95.20% still reduce to benchmark (143) or grader (95) defects, only 12 to model error (Appendix C).
Concrete, checkable defect examples
Independent external corroboration
Honest scoping and self-limitation
Weaknesses
Headline deltas bundle three simultaneous changes
No uncertainty quantification anywhere
Label noise is disclosed but unresolved
Potential circularity in ground truth
Near-saturation is model- and harness-specific
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether low scores on six physics benchmarks — PHYBench, PRISM-Physics, UGPhysics (public-source) and HLE-Physics, CMT-Benchmark, CritPt (expert-authored) — reflect genuine limits of frontier models (GPT-5.6-Sol, Claude Fable 5, Gemini 3.1 Pro) or defects in the benchmarks. Three central claims: (1) 238 of 250 audited rejections (95.20%) across four benchmarks are benchmark defects (143) or grader errors (95), and only 12 are model errors (Table 2); (2) after expert correction, GPT-5.6-Sol High's mean@4 rises 47.28%→78.66% on HLE-Physics and 61.00%→87.24% on CMT-Benchmark, with 94.44% pass@4 on 54 retained CritPt challenges (Table 1); (3) leading benchmarks are therefore near saturation and understate capability. Scope is text-only, closed-ended problems with verifiable final answers; the authors do not claim end-to-end research ability and report that agentic attempts on open physics problems failed (Section 5).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs really fail at physics benchmarks?
According to this study, mostly no: expert audits of 250 cases where GPT-5.6-Sol was marked wrong across four physics benchmarks attributed 95.2% of the failures to flawed questions or graders rather than the model. After corrections, scores jump by 20-80 percentage points, though corrected scores are computed on retained question subsets.
Which physics benchmarks were audited in this paper?
Six: PHYBench, PRISM-Physics, and UGPhysics (drawn from public sources), plus HLE-Physics, CMT-Benchmark, and CritPt (expert-authored). The pooled 250-case audit covers rejected items on the first four; CMT (all 50 questions) and a 56-challenge CritPt subset were audited in full.
What are GPT-5.6-Sol's corrected scores?
mean@4 rises from 47.28% to 78.66% on HLE-Physics, from 61.00% to 87.24% on CMT-Benchmark, and to 87.50% (pass@4 94.44%) on the 54 retained CritPt challenges. Note the corrected numbers use retained or repaired question subsets and an HLE-adapted LLM judge, so they are not directly comparable to the pre-audit numbers.
How reliable were the expert labels in the audit?
Partially: 196 of 250 audited items were reviewed twice with 71.43% initial agreement; all 56 disagreements were resolved by a third review, and 54 items had only one reviewer. The paper also permits AI assistance to auditors, which a careful reader should weigh when interpreting the attribution percentages.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Provided body text is complete through Appendix G (page 19) and the full reference list; the only truncated text is the final sentence of the metadata abstract, which is present in full in the body.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.