Thinking effort aligns between humans and reasoning models in abductive reasoning
cs.CL · cs.AI
Quick verdict
Finds human–LRM effort alignment in abduction; solid methods but the LRM-vs-LLM contrast is n.s. and stochastic gains may partly reflect averaging.
Summary
The paper tests whether the reasoning cost of large reasoning models (LRMs) tracks human thinking effort in abductive reasoning, where item difficulty cannot be read off formal structure. 120 human participants (recruited via Prolific, US) chose between two hypotheses explaining two observations on 160 items from the ART dataset, filtered so the two hypotheses differ by at most one word. The author correlates per-item reasoning-token counts from seven LRMs and one non-reasoning baseline (DeepSeek-V3, prompted with 'let's think step by step') against human reaction times using partial correlations controlling for prompt length. Under greedy decoding all models correlate significantly, from GPT-OSS-120B (r=0.552) down to DeepSeek-V3 (r=0.197), and error patterns align: items failed by 75%+ of models score 48.7% human accuracy versus 84.6% for consensus-correct items (consensus–human partial r=0.668). The key LRM-versus-baseline contrast, however, is not significant (R1 vs V3, p=0.217). Averaging 10–25 stochastic runs at provider-recommended temperatures raises correlations (e.g., GPT-OSS-20B from 0.41 to 0.55), while temperature 2.0 is unstable. Limitations: the decoding effect conflates temperature with multi-run averaging; V3's effort metric differs from LRM thinking-token counts; no multiple-comparison correction; and pretraining contamination of ART cannot be excluded.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Reasoning-token counts of all seven tested LRMs correlate significantly with human reaction times on 160 ART abduction items after controlling for item length (greedy partial r = 0.229–0.552).
- Models and humans err on overlapping items: model-consensus accuracy tracks human accuracy at partial r = 0.668.
- Stochastic decoding averaged over many runs at provider-recommended temperatures increases token–RT alignment relative to greedy decoding (e.g., GPT-OSS-20B from r = 0.41 to 0.55).
- The alignment advantage of LRMs over the non-reasoning baseline DeepSeek-V3 was not statistically significant at this sample size (R1 vs V3: Fisher z = 1.23, p = 0.217).
- Because abductive difficulty cannot be recovered from formal structure, this task offers firmer ground than deductive tasks for claims of shared thinking effort, and CoT length can serve as a proxy for cognitive effort.
Strengths
Heuristic-resistant item design
§2.4: only ART test-set items whose two hypotheses differ by at most one word were retained (68 equal-length, 92 one-word-apart), duplicates removed, sampling seeded (42); answer key balanced (87 A / 73 B); adversarially filtered items demand real discrimination from both humans and models.
Transparent null and robustness reporting
§3.1 reports the pivotal R1-vs-V3 contrast as non-significant (Fisher z=1.23, p=.217) despite the headline narrative; Spearman partials replicate all Pearson results (App. D); word- vs token-count covariate and outlier removal change no conclusions.
Effect size calibrated against a noise ceiling
Reusable convergence protocol for stochastic decoding
Shared-error analysis argues against memorization
Weaknesses
Headline LRM-vs-LLM claim outruns the statistics
Temperature confounded with multi-run averaging
Baseline operationalization mismatch
'No shortcut in abduction' premise asserted, not tested
No multiplicity control; vague exclusions
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This paper extends de Varda et al. (2025, PNAS), which reported that reasoning-token counts of large reasoning models (LRMs) track human reaction times (RTs) across reasoning tasks. Three contributions: (1) re-testing token–RT alignment on abductive reasoning, argued (§1.2) to resist 'memorized structural difficulty' shortcuts because abduction's difficulty is not recoverable from formal structure; (2) showing that models and humans fail on overlapping items; (3) showing that stochastic decoding, averaged over runs, strengthens alignment. Most claims are calibrated; one Discussion sentence outruns its own statistics (below).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do reasoning models spend about the same amount of 'thinking effort' as humans?
This paper finds that on 160 abductive multiple-choice items, the number of reasoning tokens produced by seven large reasoning models correlates significantly with human reaction times after controlling for item length (partial r from 0.229 to 0.552), and models tend to fail the same items humans find hard. The alignment is partial, though: the strongest model explains only about a third of the reliable item-level reaction-time variance, and the reasoning models' edge over a plain LLM baseline was not statistically significant in this sample.
What task and dataset does the paper use?
A forced-choice adaptation of the ART (Abductive Natural Language Inference) dataset from AllenAI: participants see two observations (the beginning and end of a short story) and pick which of two hypotheses — differing by at most one word, to block length-based guessing — better explains what happened in between. 120 humans answered 160 such items, and the same items were posed to seven reasoning models through an API.
Does the decoding temperature affect human–model alignment?
Yes, in this study. Averaging many sampled runs at provider-recommended temperatures (0.6–1.0) raised token–reaction-time correlations relative to greedy decoding — for GPT-OSS-20B from r=0.41 to 0.55 — whereas a very high temperature (2.0) gave unstable, slower-converging and generally weaker alignment. The authors themselves note the gain may largely reflect noise reduction from averaging rather than genuine exploration of multiple reasoning paths.
Is the code and data for this paper available?
The paper states that code and data are available at an anonymous repository (anonymous.4open.science/r/CostAbductiveReasoning-7052). It also documents the exact prompts, per-model hyperparameters including provider defaults, the item-sampling seed, and statistical convergence criteria, which eases replication — though the repository's contents were not verified in this review.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 3–7 are available only through captions and embedded axis labels in the extracted text; visual distribution shapes (e.g., the token-spread pattern in Fig. 7) could not be inspected.
- The linked anonymous.4open.science repository was not accessed for this review, so the released code/data contents could not be verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.