Measuring the Creativity of Frontier LLMs in Automated Research
Yiheng Zhao, Mengzhuo Chen, Chengming Hu, Pengyi Liao, Yihan Huang
cs.CL
Quick verdict
Proposes interpretable valueness/novelty metrics for LLM research agents; sensible framing but headline correlation rests on n=6 with p=0.063.
Summary
This paper proposes metrics for quantifying the creativity of LLM-based automated research agents along two dimensions: valueness, the fraction of iterations whose proposed code modifications are kept, and novelty, assessed at three granularities — Exact-Match P-Novelty (the same variable-value modification not proposed before), Variable-level P-Novelty (the modified variable set never explored before), and H-Novelty (the idea not explicitly attributed to an external reference in the model's reasoning). Grounded in cognitive-science accounts of creativity (Boden), the framework is applied to six frontier models (Claude Opus 4.8, GPT-5.5, GLM-5.2, Kimi K3, Qwen3.7-Max, DeepSeek V4 Pro) running Karpathy's autoresearch loop — single-GPU LLM training judged by validation BPB — for 100 iterations each, with novelty judged by GPT-5.6-Sol and manually spot-checked. Valueness (0.22–0.28) and Exact-Match P-Novelty (0.84–0.96) are similar across models; Variable-level P-Novelty varies widely (0.22–0.48), and H-Novelty is high (0.85–0.94) where measurable. Variable-level P-Novelty correlates most strongly with final BPB improvement (r=0.788, p=0.063, n=6), and variable-level novel ideas (37.0% of iterations) account for 62.8% of total improvement. Main limitations: a single task, a correlational design with no causal test, no code release, and reliance on an LLM judge without reported agreement statistics.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 18 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Frontier LLM research agents achieve similar Valueness (0.22–0.28) and Exact-Match P-Novelty (0.84–0.96) on Karpathy's autoresearch task.
- Models differ substantially in Variable-level P-Novelty (0.22–0.48), which reflects breadth of research-space exploration.
- Variable-level P-Novelty correlates most strongly with final BPB improvement (r=0.788, p=0.063, n=6), and variable-level novel ideas (37.0% of iterations) account for 62.8% of total improvement.
- H-Novelty is consistently high (0.85–0.94) among the four models whose reasoning traces can be evaluated.
Strengths
Well-grounded, operational metric design
Section 3.3 turns Boden's novelty/valueness account into four precisely defined, computable metrics; decomposing P-Novelty into Exact-Match vs Variable-level granularity cleanly separates 'recombining known levers' from 'opening new directions'.
Variable-level granularity reveals signal where exact-match saturates
Table 1: Exact-Match P-Novelty is nearly uniform (0.84–0.96) while Variable-level P-Novelty spreads 0.22–0.48; Table 3 further distinguishes failure modes — GPT-5.5 stalls via 64 Continue moves (longest streak 13) whereas DeepSeek V4 Pro cycles with 60 Return moves at similar low PV.
Idea-level contribution analysis goes beyond correlation
Honest scoping of what can be measured
Weaknesses
Headline statistical claim is underpowered
Single run, single task, no variance estimates
LLM-judge evaluation without reliability statistics
Unspecified variable extraction and closed artifacts
H-Novelty interpretation risk and editorial sloppiness
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv:2609.14057, cs.CL, ~4 pages) asks how creative frontier LLMs are when acting as autonomous research agents. It operationalizes creativity — defined after Boden (2004) as jointly necessary novelty and valueness — into four computable metrics (§3.3): **Valueness** (fraction of iterations whose code modification is kept), **Exact-Match P-Novelty** (no identical variable–post-value modification proposed before), **Variable-level P-Novelty** (the modified variable set never explored before), and **H-Novelty** (the idea not explicitly attributed to an external reference in the reasoning trace). Central claims: (1) Valueness and Exact-Match P-Novelty are similar across six frontier models while Variable-level P-Novelty differs substantially; (2) H-Novelty is high where measurable; (3) Variable-level P-Novelty correlates most strongly with final performance (Table 2), and variable-level novel ideas drive most of the improvement (§4.4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What does this paper measure about LLM creativity?
It measures two creativity dimensions of LLM research agents: valueness (the share of proposed code changes that actually improve the baseline and are kept) and novelty at three levels — whether the exact variable-value modification was proposed before, whether the modified variable set was explored before, and whether the idea is explicitly attributed to external prior knowledge in the model's reasoning.
Which model was the most creative in the study?
Kimi K3 scored highest on Variable-level P-Novelty (0.48), meaning it explored the most new research directions, while GPT-5.5 scored lowest (0.22) despite having the highest Valueness (0.28). Valueness was similar for all six models (0.22–0.28), and GPT-5.5's low exploration came from repeatedly continuing the same variable set (64 of 100 iterations).
Is the link between novelty and research performance statistically significant?
Not strictly. The strongest correlation (Variable-level P-Novelty vs final improvement) reached r=0.788 with p=0.063 across only six models, which misses the conventional 0.05 threshold, and four correlations were tested without correction. The supporting idea-level finding — 37% of ideas driving 62.8% of improvement — is suggestive but pooled across models and untested for significance.
What task were the models actually doing?
Each model ran Karpathy's open-source 'autoresearch' loop for 100 iterations: propose a code modification to a single-GPU LLM training program, apply it, and keep it only if validation BPB (bits per byte, lower is better) improves from the starting baseline of 0.993123.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's per-model improvement trajectories are not recoverable from the text extraction (only the 0.993123 baseline and legend ordering are visible), so final per-model improvements could not be checked.
- A verbatim duplicated paragraph in Section 3 and duplicated affiliation numbering suggest extraction artifacts; metric formulas may be imperfectly transcribed.
- The paper appears to be a ~4-page short/workshop-format paper with no appendix in the extracted text; additional reliability details may exist but were not visible.
- The judge prompts for GPT-5.6-Sol are not included in the text, so judge-bias risk is assessed only from the described procedure.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.