Does Reasoning Improve Psychological Depth in Large Language Models? It Depends on Who's Judging
Ruichen Zheng, Yihe Wang, Fabrice Y Harel-Canada, Sara Khosravi, Zeynep Senahan Yildiz, Amit Sahai, Nanyun Peng
cs.LG · cs.CL
Quick verdict
Audits a PDS judge (dev ρ=0.646) that flips to 89% pro-reasoning on frontier story pairs where humans split; well-hedged but tiny: 20 premises, 7 readers.
Summary
The paper asks whether an LLM-as-a-Judge evaluator that agrees well with human ratings on its development set remains valid when deployed on shifted, closely matched outputs. Using the Psychological Depth Scale (PDS) rubric, the authors build an ensemble judge from Llama 3.1/3.3 70B and Qwen 3.5 configurations routed per dimension, achieving ρ=0.646 on the original 97-story dataset's 80% selection split. Seven blinded undergraduate readers and the judge then evaluated 60 prompt-matched story pairs (400–600 words) from GPT-5 vs. GPT-4o (auto and high reasoning effort) and DeepSeek-R1 vs. DeepSeek-V3, drawn from 20 premises. Human preferences showed no universal reasoning advantage: GPT-5 was modestly preferred (60.0% and 62.9% overall), DeepSeek-R1 trailed V3 (42.9%), and inter-reader agreement was near chance (Krippendorff's α=0.070). The judge, in contrast, favored reasoning outputs in 89.0% of 300 dimension-level comparisons and 59 of 60 pairs on aggregate PDS, uniformly across all five configurations, with scores correlated to surface features (engagement vs. mean sentence length r=−0.70; negative MATTR associations on four of five dimensions). The authors conclude that development-set correlation is insufficient evidence of deployment validity, while carefully labeling the work a context-specific audit. Key limitations: 20 premises, one retained generation per condition, seven readers from one university, a shared selection/validation split, and no code or data release stated.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 3
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A PDS LLM-judge ensemble with development-set ρ=0.646 preferred reasoning-model stories in 89.0% of 300 dimension-level comparisons and 59 of 60 aggregate-PDS pairs, uniformly across all five configurations.
- Seven human readers showed no universal reasoning advantage: GPT-5 was modestly preferred over GPT-4o (60.0% and 62.9%) while DeepSeek-R1 trailed V3 (42.9%), with inter-reader agreement near chance (Krippendorff's α=0.070).
- Judge scores are associated with surface features — engagement vs. mean sentence length r=−0.70 and negative MATTR associations on four of five dimensions — offering a candidate mechanism for the pro-reasoning skew.
- Reader disagreement is structured (recurring dimension-weighting patterns) rather than random, so a point-estimate judge cannot represent the heterogeneous human target.
- Development-set correlation is insufficient evidence of deployment validity on a shifted distribution; target-matched human checks and independent validation are needed.
Strengths
Precise framing of a consequential validity question
Targets the gap between development-set agreement and deployment validity for LLM judges on subjective, closely matched outputs (Sec. 1), grounded in documented judge biases (position, verbosity, self-preference) rather than anecdote.
Unusually candid self-audit
Appendix B states the routed ρ=0.646 is a selection-set estimate, not performance on an untouched test set; Sec. 4 calls the study a context-specific audit rather than independent evidence of transfer failure; Table 4 explicitly separates human overall preference from judge aggregate PDS as different targets.
Careful statistical treatment for the sample size
Annotator-quality due diligence
Concrete mechanism candidate, hedged as correlational
Weaknesses
Near-chance human target limits interpretation
Judge configurations are not independent
Small, single-site deployment set with unquantified generation variance
No held-out validation even on the source distribution
Surface-feature evidence carries no uncertainty
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper audits a specific failure mode of LLM-as-a-Judge evaluation: a judge tuned for agreement with human ratings on its development set may produce invalid measurements when deployed on a shifted distribution of closely matched outputs. The testbed is psychological depth in short fiction, measured by the five-dimension Psychological Depth Scale (PDS: authenticity, emotion provocation, empathy, engagement, narrative complexity) from Harel-Canada et al. (2024). Three central claims: (1) an ensemble PDS judge with development-set ρ=0.646 emits a near-uniform pro-reasoning signal — 89.0% of 300 dimension-level comparisons and 59 of 60 aggregate-PDS pair wins, uniform across all five configurations — where seven human readers are near chance (Krippendorff's α=0.070) and split by family (GPT-5 modestly preferred over GPT-4o at 60.0% and 62.9%; DeepSeek-R1 trailing V3 at 42.9%); (2) judge scores correlate with surface features such as mean sentence length (engagement r=−0.70) and lexical diversity (MATTR), a candidate mechanism; (3) reader disagreement is structured, so point-estimate judges cannot represent the human target. The authors explicitly frame the work as a context-specific audit, not proof of general transfer failure.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of this paper?
An LLM judge tuned to match human Psychological Depth ratings (ρ=0.646 on its development split) systematically preferred reasoning-model stories in 89.0% of 300 dimension-level comparisons and 59 of 60 pairs across 60 frontier story pairs, while seven human readers were near chance overall (Krippendorff's α=0.070) and split by family — mildly preferring GPT-5 over GPT-4o (60.0–62.9%) but preferring DeepSeek-V3 over DeepSeek-R1 (R1 at 42.9%).
Do reasoning models write psychologically deeper stories?
Not according to this study's human readers: there was no universal advantage. GPT-5 was modestly preferred over GPT-4o (60.0% and 62.9% under two reasoning-effort settings), but DeepSeek-R1 was preferred less often than DeepSeek-V3 (42.9%), and the seven readers disagreed with each other almost at chance (Krippendorff's α=0.070), so preferences were highly subjective.
Why did the LLM judge favor reasoning-model outputs?
The authors found judge scores correlated with surface features: engagement scores vs. mean sentence length at r=−0.70, and lower lexical diversity (MATTR) associated with higher scores on most dimensions. Reasoning-model stories had shorter sentences and slightly lower MATTR, consistent with the judge rewarding stylistic artifacts rather than depth. The evidence is correlational, not causal.
Is the judge's 0.646 correlation trustworthy?
It is a development-set estimate: the same 80% data split was used both to select and route judge configurations and to report the correlation, so it is not an untouched-test figure. The paper's central argument is precisely that such development-set performance does not guarantee validity when the judge is deployed on a shifted distribution.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures appear only as captions and embedded axis-label text; the plots themselves are not renderable from the extraction, though all figure-derived numbers used in this review are duplicated in the tables and body text.
- The extracted title line reads 'Depends on Who's Judging' while the metadata title reads 'It Depends on Who's Judging' — a minor extraction artifact with no effect on content.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.