Implicit Personality Representations in Humans and LLMs
Yilin Geng, Omri Abend, Eduard Hovy, Lea Frermann
cs.AI
Quick verdict
Claims Qwen-2.5-7B's internal trait geometry mirrors human implicit personality structure (Mantel r=0.765); credible and well-controlled but single-model.
Summary
The paper asks whether the internal trait representations of a large language model reproduce the relational structure of human personality impressions. Using the Statistical 'Which Character' Personality Quiz (SWCPQ; 3.4M raters, 77.4M ratings over 2,125 fictional characters), the authors filter to 385 valid bipolar trait scales and build a human implicit-personality matrix as Pearson correlations between trait rating profiles across characters. On the model side they extend the contrastive persona-vector method: scenarios are generated with Gemini-2.5-flash, role-played responses come from Qwen 2.5-7B-Instruct anchored by top/bottom SWCPQ characters, a GPT-4.1-mini judge filters strong-pole exemplars, and layer-20 activation pole-differences serve as trait directions. The resulting 385x385 correlation matrix aligns with the human matrix at Mantel r=0.765 (p<1e-4; 0.708 after controlling synonym clusters; per-trait median 0.80). Two principal components (59% of variance) qualitatively match the social-warmth and intellectual-competence axes of classic person-perception research. Projecting held-out Wikiquote dialogue for 256 characters onto the directions yields moderate agreement with human profiles (median r=0.403 on 195 aligned traits; 93% positive). Limitations: a single model, structural (non-causal) evidence, an imperfect LLM-judge filter (precision 0.79, recall 0.82) validated by only four colleague annotators, and the absence of a generic text-embedding baseline.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- The relational structure of Qwen 2.5-7B-Instruct's internal trait directions aligns strongly with the human implicit-personality matrix built from 77.4M SWCPQ ratings (Mantel r=0.765, p<1e-4; 0.708 after controlling synonym clusters).
- Two dominant principal components of the model's trait space (37% and 22% of variance) correspond to the intellectual-competence and social-warmth dimensions known from human person-perception research.
- Trait directions extracted from role-played scenario responses transfer to held-out Wikiquote dialogue, agreeing with human character ratings at median r=0.403 on 195 aligned traits (93% of 256 characters positive).
- The alignment is robust to synonym clusters, prompt framing, layer choice, similarity metric, exemplar re-selection, and human-data cell-count thresholds, and holds trait-by-trait (median per-trait r=0.80).
Strengths
Comprehensive, reliable human reference
385 bipolar trait scales grounded in 77.4M SWCPQ ratings over 2,000 characters — far beyond the handful of ad-hoc traits in prior persona-vector work; split-half reproducibility is reported (Mantel r=0.998388 over 100 session splits, App. A) and raising the per-cell rating floor from 5 to 20 barely moves alignment (0.765 to 0.758).
Unusually thorough statistical controls
Permutation Mantel tests (10,000 joint permutations, null centered 3.1e-5), a synonym-cluster control (r drops only 0.765 to 0.708 over 58,623 remaining dyads), metric invariance (Pearson vs cosine, r=1.000, App. D), mean-direction and PC1-removal diagnostics (App. E: 0.7645 and 0.6135), and judge-filter sensitivity (Table 4).
Prompt-method and layer ablations
Honest held-out transfer evaluation
Transparent validation of the pipeline
Weaknesses
Missing generic-embedding baseline
Single model
Thin judge validation with colleague annotators
Qualitative PCA axis identification
Anchor circularity partially mitigated only
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a single-model interpretability study asking whether the internal trait geometry of Qwen 2.5-7B-Instruct reproduces the relational structure of human personality impressions — the implicit-personality-theory tradition of Rosenberg et al. (1968) and Fiske et al. (2007). Three central claims: (1) the model-side 385x385 trait-correlation matrix aligns with the human matrix at Mantel r=0.765, p<1e-4 (abstract rounds to 0.77); (2) the two dominant principal components (37% + 22% = 59% of variance) correspond to intellectual competence and social warmth; (3) trait directions transfer to held-out Wikiquote dialogue at median per-character r=0.403 on 195 aligned traits. The framework extends Chen et al. (2025) persona vectors from a handful of ad-hoc traits to 385 crowd-sourced bipolar scales, grounding everything in human data rather than model self-report.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did the study find that LLMs have the same personality structure as humans?
For the one model tested (Qwen 2.5-7B-Instruct), largely yes: its internal trait directions formed a relational structure matching crowd-sourced human ratings at Mantel r=0.765 (0.708 after removing synonym clusters), with social warmth and intellectual competence as the two dominant axes. Only a single model was tested, and the match is structural evidence, not proof the model uses this geometry.
What is the SWCPQ dataset used in this paper?
The Statistical 'Which Character' Personality Quiz, hosted at openpsychometrics.org: volunteers rate fictional characters on bipolar trait scales (e.g., cowardly-brave). The November 2023 release contains about 3.4 million raters, 77.4 million ratings, 2,125 characters, and 499 trait scales, released under CC BY-NC-SA 4.0.
Can the extracted trait directions read a character's personality from dialogue?
Moderately well. On 256 held-out Wikiquote characters (at least 300 words of quoted lines), the correlation between dialogue-projected and human-rated profiles had a median r=0.403 (95% CI [0.358, 0.445]) on 195 high-quality trait axes, positive for 93% of characters. Failures occur when a character's speech persona diverges from their rated disposition.
Is code or data available for this paper?
Yes. Footnote 1 lists a public GitHub repository (yilin-geng/llm_implicit_personality) with code, processed data, prompts, and testing scenarios. The underlying SWCPQ dataset is publicly available from openpsychometrics.org under a CC BY-NC-SA 4.0 license.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure-internal details (PCA scatter in Fig. 4, per-trait distribution in Fig. 3, profile panels in Figs. 5/8-10) cannot be inspected from the text extraction; judgments about them rely solely on captions and in-text statistics.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.