Induction and Inquiry via Probabilistic Reasoning over Language and Code
Wasu Top Piriyakulkij, Sam Acquaviva, Cassidy Langenfeld, Joshua Tenenbaum, Kevin Ellis
cs.AI
Quick verdict
LLM-guided SMC over language+code hypotheses matches human induction/inquiry data; promising and broad, though statistical rigor and repro details are thin.
Summary
The paper proposes a computational model of human inductive learning and active inquiry in which hypotheses ('mental programs') pair a natural-language description with a Python implementation. Beliefs are maintained as a particle set and updated with LLM-SMC-S, a Sequential Monte Carlo scheme whose proposal kernel is GPT-4; questions and experiments are chosen to maximize expected information gain over the particle posterior. Across three behavioral paradigms reusing published human data — Rule et al.'s 250 list-function algorithms, Thaker et al.'s number-concept game, and Bramley et al.'s Zendo experiments — the model reproduces sequential signatures such as anchoring and garden-pathing, gives the best MSE fit to trial-by-trial accuracy on 70 of the 100 easier algorithms, beats the specialized HL model using 5 proposals versus 500,000, outperforms a hand-built Bayesian model on Zendo test log-likelihood (−1478.82 vs −1539), and shows a U-shaped relation between compute budget and human-model fit consistent with bounded rationality. A shopping-assistant proof of concept with a finite hypothesis space generates more informative questions than vanilla LLM prompting or ReAct. Main limitations: human-likeness may inherit from GPT-4's pretraining rather than the Bayesian architecture; statistical reporting lacks error bars, confidence intervals, and significance tests; the noise parameter is fit to the same human data; and no code or prompt release is stated.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Human sequential induction is well captured by Bayesian updating over 'mental programs' that combine a natural-language description with a Python implementation.
- With GPT-4 as proposal distribution, LLM-SMC-S reproduces anchoring and garden-pathing and fits trial-by-trial human accuracy better than specialized models at far lower search budgets (5 proposals vs. HL's 500,000).
- Expected-information-gain inquiry (eq. 2) lets the model choose experiments (Zendo) and questions (shopping task) that match or beat prior custom models and LLM prompting baselines.
- Model-human fit is U-shaped in compute budget, supporting a bounded-rationality account, while pure LLMs fail to match human behavior on these tasks.
Strengths
Cross-domain unity
The same LLM-SMC-S machinery is evaluated on Rule et al.'s 250 list functions (§3.1), Thaker et al.'s number game (§3.2), Bramley et al.'s Zendo (§4), and a question-asking task — rare breadth for a cognitive model, with model-vs-model comparisons on identical public human data.
Compute-matched comparisons
Fig. 4c: 5 proposals predict human accuracy across 250 algorithms better than HL with 500,000-program search; fig. 6: 70 particles (20 under load) fit anchoring better than Thaker et al.'s custom model using more particles; fig. 7d: Zendo test log-likelihood −1478.82 vs. Bramley et al.'s −1539.
Bounded rationality analysis
Sequential processing is shown to matter
Honest scoping of LLM-only failure
Weaknesses
Pretraining confound unaddressed
Statistical reporting thin
Parameters fit to evaluation data
Shopping task weakens the headline claim
Reproducibility gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
Paper: 'Induction and Inquiry via Probabilistic Reasoning over Language and Code' (arXiv 2609.01815, cs.AI; Piriyakulkij, Acquaviva, Langenfeld, Tenenbaum, Ellis; Cornell/MIT). It proposes a unified computational account of human sequential induction and active inquiry. Core claims: (1) hypotheses are 'mental programs' pairing a natural-language description with a Python implementation (§2); (2) beliefs are maintained as particles and updated with LLM-SMC-S, an LLM-augmented Sequential Monte Carlo scheme building on the authors' NeurIPS 2024 work (ref [11]), targeting the posterior of eq. (1); (3) the model reproduces human sequential-learning signatures (anchoring, garden-pathing) and fits human data better than specialized models at far lower compute; (4) inquiry — experiments or questions — is chosen by expected information gain (eq. 2). Evaluation reuses previously published human datasets; the authors collect no new behavioral data.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is a 'mental program' in this paper?
A hypothesis about the data represented as two parts: a natural-language description (e.g., 'multiples of 3') and an executable Python implementation. The authors argue human hypotheses lie between pure logic and pure language, so language supports flexible idea generation while code gives an exact, checkable grounding against observations.
Did the authors run new experiments with human participants?
No. All human data are reused from prior published studies: Rule et al.'s list-function benchmark, Thaker et al.'s number-concept game, and Bramley et al.'s Zendo experiments. The shopping question-asking study is a proof of concept without new human comparison data.
Does the paper train a new AI model?
No. It uses GPT-4 off the shelf inside a Sequential Monte Carlo loop: GPT-4 proposes and revises candidate hypotheses, and Bayesian reweighting and resampling score them against observations. The contribution is the inference algorithm, not a new neural network.
Is code available for this paper?
The paper text contains no code, data, or prompt release statement. The algorithm and per-domain hyperparameters are described in Appendix A, but the prompts are not shown, so exact replication would require reimplementing the pipeline.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- PDF extraction interleaves figure graphics/captions with body text (e.g., around Figures 1–8, Figure 4's caption merges axis labels and a stray task index), so figure-internal values are cited only where legible in captions; no claim here relies on unverifiable figure internals.
- Body text, references, and Appendix A run to the end (Algorithm 1) with no evident truncation of substantive content; no missing-content claim is made beyond figure interleaving.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.