Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humans
Akshay K. Jagadish, Younes Strittmatter, Nori Jacoby, Eric Schulz, Nathaniel Daw, Thomas L. Griffiths, Suyog H. Chandramouli
cs.AI
Quick verdict
Runs an LLM discovery loop on the Centaur simulator; surfaced theories beat canonical ones on held-out humans — promising, but one task and one run.
Summary
The paper asks whether psychological theories discovered by an automated LLM-agent discovery loop running entirely on a behavioral foundation model generalize to humans, or merely characterize the simulator. The authors run AUTOCOG — a four-stage loop in which LLM agents design theory-discriminating experiments, collect responses, arbitrate between rivals, and synthesize successors — for 25 cycles in a two-option multi-attribute choice task (up to five cues, validities 0.5–0.9), seeded with Take-The-Best and Tallying and simulated entirely by Centaur. The loop surfaced two new theories, Relative Contextual Salience Lexicographic (RCSL) and Contextual Relative Advantage Normalization (CRAN), recombining known mechanisms. On ten held-out human experiments, evaluated generatively, RCSL (MSE_p(B) = 0.021 ± 0.002) and CRAN (0.034 ± 0.004) clearly beat the seeds (TTB 0.110 ± 0.010; Tallying 0.161 ± 0.013) and were rivaled only by two theories the same loop discovered with human participants (both 0.018), which — as the authors disclose — were derived from these very experiments. The central argument: an arbitration-based discovery loop demands less of its simulator than estimation does; it need only capture the regularities that distinguish theories. Limitations include a single task family, one unreplicated 25-cycle run, generative evaluation sensitive to parameter-range choices, no significance testing, and experiments staying near Centaur's training distribution.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Theories discovered by AUTOCOG running entirely on Centaur-simulated data generalize to ten held-out human experiments, outperforming the canonical seed theories Take-The-Best and Tallying.
- These simulator-discovered theories are rivaled only by theories the same loop discovered with human participants, which were derived from the very experiments used for evaluation.
- A discovery loop that arbitrates between competing theories demands less of its simulator than estimation does: the simulator need only capture the regularities that distinguish the theories, not reproduce behavior precisely.
- Imperfect simulators can therefore widen the search over candidate theories, with human data reserved for testing whether surfaced theories generalize (a hybrid in silico / human workflow).
Strengths
Direct, controlled test of a timely open question
The same AUTOCOG loop and the identical evaluation pipeline (200 simulated participants, parameters drawn once from declared ranges, seed 0) produced the held-out numbers for both Centaur-discovered and human-discovered theories (Appendix, Run details: 'like for like'), isolating the sim-to-human generalization question cleanly.
Unusual transparency of the artifact
The appendix prints the complete executable code, parameter ranges and verbal mechanism statements of all four theories (TTB, Tallying, CRAN, RCSL), plus agent configuration (gemini-3.1-pro-preview, temperature 0.7, 8,096-token reasoning budget) and Centaur inference details (4-bit Llama-3.1-70B adapter, 25 synthetic participants, temperature 1.0 single-token choices).
Honest disclosure of the strongest competing explanation
Substantive effect size on held-out human data
Weaknesses
Single unreplicated run of a stochastic system
Generative evaluation conflates mechanism with parameter-range priors
No inferential statistics anywhere
In-loop decision quality is unauditable from the paper
Generality beyond one near-training-distribution task family is untested
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This short paper (4 pages plus appendix) addresses a sharply posed question: when a closed-loop LLM discovery system runs entirely on a behavioral foundation model, do the theories it finds describe humans or only the simulator? The authors run the Automated Cognitive Scientist (AUTOCOG) — experimental design, simulated data collection, arbitration, program-synthesis theory revision — for one 25-cycle run in a two-option multi-attribute choice task (up to five cardinal cues, validities 0.5–0.9), seeded with Take-The-Best (TTB) and Tallying, with Centaur (the released 4-bit Llama-3.1-70B adapter) as the sole data source. Four claims: (1) the loop surfaced two novel theories, RCSL and CRAN; (2) on ten held-out human experiments these generalize — RCSL 0.021 ± 0.002 and CRAN 0.034 ± 0.004 MSE_p(B) versus TTB 0.110 ± 0.010 and Tallying 0.161 ± 0.013; (3) they are rivaled only by two theories the same loop found with human participants (both 0.018), which the authors disclose were derived from these very ten experiments; (4) conceptually, an arbitration-based loop demands less of its simulator than estimation does — the simulator need only capture regularities that discriminate theories, echoing multifidelity reasoning (Peherstorfer et al., 2018).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is Centaur in this paper?
Centaur is a foundation model of human behavior: a Llama-3.1-70B language model fine-tuned on trial-level data from 160 psychology experiments to predict human responses. Here it acts as a synthetic participant, answering every trial of experiments designed by the LLM agents.
Did theories discovered purely on simulated data actually match human behavior?
In this study, yes to a striking degree: on ten held-out human experiments, the simulator-discovered theory RCSL achieved a mean squared error of 0.021 ± 0.002 versus 0.110 ± 0.010 for the canonical Take-The-Best heuristic, coming close to theories discovered using real human participants (0.018).
Why can an imperfect simulator still produce useful theories?
The authors argue that a discovery loop which only arbitrates between competing candidate theories demands less of its simulator than parameter estimation does: the simulator merely needs to capture the regularities that distinguish the theories from one another, not reproduce human behavior exactly — analogous to multifidelity methods in computational science.
What are the main limitations of this study?
It covers one task family (two-option multi-attribute choice) near Centaur's training distribution, reports a single unreplicated 25-cycle run of the discovery loop, uses generative (not fitted) evaluation that is sensitive to declared parameter ranges, and reports no significance tests.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1C/1D are available only as caption text plus the quoted endpoint means; per-cycle trajectories and underlying per-experiment values could not be checked beyond those numbers.
- In-loop materials (agent transcripts, arbiter decisions, the fifty designed experiments) are absent from the extracted text, so arbitration quality was assessed only from the authors' description.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.