CogGym: Towards Large-Scale Comparative Evaluation of Human and Machine Cognition
Lance Ying, Jinzhou Wu, Yingshan Susan Wang, Shivam Aarya, Luca M. Schulze Buschoff, Harry Chen, Katherine M. Collins, Andrea de Varda, et al.
cs.AI
Quick verdict
CogGym unifies 258 cognitive experiments; LLM-human alignment scales with size but remains far below human reliability — a rigorous, well-validated benchmark.
Summary
CogGym is a benchmark and infrastructure effort that standardizes 258 cognitive-science experiments drawn from 100 published papers into a shared Experiment Markup Language (EML), then administers the same trials to humans and to 50 large language models (April 2024–April 2026) under matched conditions. Experiments are converted by an AI-agent pipeline with human-in-the-loop review, including audits by original authors; text, image, and video paradigms span theory of mind, causal and physical reasoning, moral judgment, pragmatics, and more. Model fit is scored as R² between human and model mean responses per experiment, plus a normalized distributional divergence from verbalized distributions, calibrated against random baselines and Spearman-Brown-corrected human split-half reliability. The best models (Gemini 3.1 Pro: 0.59 text, 0.58 image, 0.43 video) remain well below human ceilings (0.93/0.95/0.92); alignment rises with model size (r ≈ 0.75–0.81 per Figure 5) and less consistently with release date; physical reasoning and video are weakest. A 15-experiment Prolific replication (521 participants) yields replication-vs-original R² = 0.84 versus 0.62 for the best model, suggesting the gap reflects genuine model-human differences. Limitations: only ~6% of experiments received fresh human validation, video results come almost entirely from one vendor family, and the comparison with formal-reasoning benchmarks is qualitative.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A semi-automated, human-in-the-loop EML pipeline can standardize hundreds of heterogeneous cognitive experiments into executable, matched human-model specifications (validated behaviorally on 15 experiments).
- Model-human behavioral alignment on commonsense-reasoning experiments rises with model size and, after size control, mostly not with release date among open-weight models.
- Even the best models remain far below human split-half reliability: best R² = 0.59 (text), 0.58 (image), 0.43 (video) versus corrected human ceilings of 0.93/0.95/0.92.
- Low alignment reflects genuine model-human differences rather than standardization artifacts (replication R² = 0.84 vs model R² = 0.62 on the 15-experiment subset).
- Alignment gains lag far behind near-saturated accuracy gains on formal benchmarks (GPQA, MATH, HumanEval), and physical-reasoning plus video experiments are the weakest areas.
Strengths
Unmatched scale with matched-trial design
258 experiments from 100 papers across 30+ labs and 10 topic groups, administered to 50 models under identical prompts (10 runs/trial, §3), replacing the fragmented single-study literature the introduction correctly criticizes.
Behavioral validation of the standardization pipeline
The 15-experiment Prolific replication (521 participants, Appendix F/Table 4) directly tests the standardization-artifact confound: replication-vs-original R² = 0.84 versus 0.62 for Gemini 3.1 Pro — the right control experiment, rarely run in this genre.
Calibrated reference points instead of bare scores
Metric robustness checks
Honest scoping and per-item diagnosis
Weaknesses
Thin validation coverage
Internal numeric inconsistencies
Cross-benchmark 'slower progress' claim is not on a common scale
Video conclusions rest on one vendor family
Single prompt configuration, no sensitivity analysis
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CogGym (arXiv 2609.21259) is a benchmark-and-infrastructure paper from a 50+ author collaboration. It converts 258 cognitive-science experiments from 100 papers (text, image, video; theory of mind, causal and physical reasoning, moral judgment, pragmatics, etc.) into a task-agnostic Experiment Markup Language (EML), then administers matched trials to humans and 50 LLMs. Three claims anchor the paper: (1) the EML pipeline standardizes heterogeneous paradigms faithfully enough for matched human-model comparison; (2) model-human alignment rises with size and recency but far more slowly than accuracy on formal benchmarks; (3) even the best models (R² = 0.59 text / 0.58 image / 0.43 video for Gemini 3.1 Pro) remain well below Spearman-Brown-corrected human split-half ceilings (0.93/0.95/0.92).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is CogGym?
CogGym is a benchmark and platform that converts published cognitive-science experiments (258 of them, from 100 papers, spanning text, image, and video) into a shared machine-readable format called EML, then runs the very same trials on both human participants and large language models so behavior can be compared trial by trial.
How well do AI models match human judgments on CogGym?
Not fully, even for the best models. The strongest model (Gemini 3.1 Pro) reaches R² of 0.59 on text, 0.58 on image, and 0.43 on video experiments, while human split-half reliability — the practical ceiling — is 0.93, 0.95, and 0.92. Alignment improves with model size, but physical-reasoning and video tasks remain the weakest areas.
Is CogGym's data and code publicly available?
The paper states that the experiment and dataset release, documentation, and contribution information are available at coggym.org, and that the MEML specification workflow for generating experiments lives in a separate public repository. Re-running the model evaluations requires API access to the 50 evaluated models, several of which are proprietary.
Do AI models diverge from humans because they are smarter than humans?
The authors argue there is no evidence for that: on the 11 experiments that report objective accuracy, humans (0.79) slightly outperform Gemini 3.1 Pro (0.73), and within that subset models with higher relative accuracy actually fit human responses better (r = .72). Most CogGym tasks are subjective judgments with no ground-truth answer key.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are text-rendered in the extraction; individual data points and some axis values could not be independently verified against the plots.
- The full MEML specification and harness live in external repositories referenced but not included in the text; exact scoring-code behavior could not be inspected.
- Several internal numeric inconsistencies were flagged (Fig. 5 vs text, §4 vs Table 1, Appendix H rates) but could not be resolved from the text alone.
- The paper names 2026-vintage models (e.g., Gemini 3.1 Pro, Claude Opus 4.7); their published status and reported figures could not be cross-checked offline.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.