WhatWorkedBench: Benchmarking Experimental Understanding in AI Agents
Jingjie Ning, Xueqi Li, Yibo Kong, Dongting Li
cs.AI · cs.LG
Quick verdict
Benchmarks agents' predictions of component effects vs exhaustive execution; rigorous, reproducible design, but evidence limited to one provider's two models.
Summary
WhatWorkedBench evaluates whether AI agents acquire quantitative knowledge from budgeted computational experiments. Agents inspect workflow code, purchase measurements of binary option configurations (d=4 or 6), and submit a complete response surface; exhaustive CPU execution of all 1,248 configurations across 36 tasks (30 sources, 8 workflow families) supplies exact conditional-effect references. The benchmark scores effect MAE/recovery, pair interactions, configuration regret, and artifact validity, with a strict reconstruction criterion at 10% of response range. Across 108 core DeepSeek Flash/Pro episodes and 4,206 numerical-control records, exact optimum selection coexists with weak effect recovery: at B=8, pair ridge makes 15 exact choices but only 3 strict reconstructions. Refitting a Gaussian process to the agents' own observations raises recovery from 0.632 to 0.698 (original Flash cohort) and 0.621 to 0.720 (additional cohort); on six beat-detection/link-prediction submissions, shared GP lifts 0.303 to 0.455. Hash-verified code equivalences raise six-factor GP recovery from 0.248 to 0.462 at B=20. Diagnostic episodes show agents correctly report inactive-option rules yet submit tables violating them. Limitations: evidence rests on one provider's two models, no significance testing is reported, tasks are binary and deterministic, and the original-cohort GP analysis is retrospective.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Exact configuration choice can coexist with arbitrarily weak conditional-effect knowledge, so optimization success and experimental understanding are distinct evaluation targets.
- Refitting a Gaussian process to the same agent observations raises effect recovery from 0.632 to 0.698 (original Flash cohort) and 0.621 to 0.720 (additional cohort).
- Encoding hash-verified code equivalences raises six-factor GP recovery from 0.248 to 0.462 at B=20, and shared GP raises six added-family submissions from 0.303 to 0.455.
- Agents correctly report inactive-option rules yet deliver prediction tables that violate them; post-hoc equivalence projection raises recovery from 0.338 to 0.507.
- 35 of 36 task conditions contain sign reversals of conditional component effects, making average effects and single preferred configurations inadequate summaries.
Strengths
Exhaustive, executable reference with exceptional reproducibility
All 1,248 native configuration outcomes are executed, hash-checked on cold replay, and replayed by a standalone prepared-input pack in 127.87 s on one thread; separately implemented metrics and target-perturbation checks verify scoring boundaries (Sec. 4, Reproducibility Statement).
Shared-estimator protocol cleanly decomposes acquisition vs inference
Table 3 holds acquired observations fixed while swapping D1/D2 ridge and GP, cleanly isolating measurement policy from numerical inference — a control most agent benchmarks lack.
Conceptual contribution: optimization success ≠ intervention knowledge
Unusually honest execution accounting
Hash-verified program-structure controls
Weaknesses
Model scope limited to one provider's two models
No inferential statistics on headline gains
External validity gap to realistic experimentation
Adaptive, partly retrospective study design
Delivery and rate-limit confounds
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
WhatWorkedBench (arXiv 2609.27490) is a benchmark for *experimental understanding*: after budgeted experimentation on a frozen computational workflow, can an agent quantitatively predict how component changes affect outcomes? Agents inspect code, receive two free anchor measurements, purchase up to B additional configurations (binary masks, d=4 or 6), and must deliver a complete response surface. The reference is exhaustive native CPU execution: 1,248 configuration records over 36 task conditions (30 sources, 8 workflow families), yielding 3,392 conditional effects and 342 mean pair interactions. Central claims: (1) exact configuration choice and effect knowledge are separable targets (Sec. 6.1; analytic construction in App. F.2); (2) shared GP refitting on fixed observations raises recovery 0.632→0.698 and 0.621→0.720 across dated Flash cohorts; (3) hash-verified code equivalences raise six-factor GP recovery 0.248→0.462 at B=20; (4) agents report inactive-option rules yet deliver tables violating them (post-hoc projection 0.338→0.507).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is WhatWorkedBench?
It is a benchmark that measures whether AI agents learn how component changes affect experiment outcomes. Agents inspect workflow code, buy measurements of binary option configurations under a budget, and must submit a table predicting the score of every configuration; grading compares predictions against exhaustive execution of all configurations.
What are the main results of the paper?
DeepSeek Flash agents reached 0.632 conditional-effect recovery at budget B=8; refitting a Gaussian process to the same observations raised it to 0.698 (0.720 in a second cohort). Code-equivalence encoding raised six-factor GP recovery from 0.248 to 0.462, and shared GP lifted six beat-detection/link-prediction submissions from 0.303 to 0.455.
Why does the paper separate optimization success from understanding?
An agent can select the best configuration while holding a wrong model of the system. The paper gives a construction (Appendix F.2) and empirical cases — 15 exact optimum choices but only 3 strict reconstructions for pair ridge at B=8 — showing optimization success and effect knowledge are separate metrics.
Is code and data available for WhatWorkedBench?
The paper states that public tasks, references, episode tools, prepared input packs, and a standalone engine reproducing the 4,206 stored control records accompany the work; the exhaustive native references replay in about 128 seconds on one CPU thread.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-5 are available only as embedded text descriptions; visual content (curves, distributions) could not be analyzed directly.
- Review is based on the extracted text alone; dense appendix tables (F.6, K, L) may contain minor extraction noise.
- Public repository/homepage availability is asserted by the paper but could not be independently verified (no network access in this review).
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.