Measuring Behavioural Signatures of Large Language Models through Psychometric Profiling
Yu Sha, Junqi Tao, Dixin Zhou, Yansheng Tu, Mingyang Chen, Xiang Fan, Yang Liu, Mengquan Yang, et al.
cs.CL · cs.AI
Quick verdict
Cross-lingual study finds reproducible, model-specific LLM psychometric profiles; strong stats, but n=9 models and time-confounded prompt checks limit scope.
Summary
The study develops a cross-linguistic psychometric profiling framework for deployed LLMs and administers seven standardized instruments — IPIP BFFM-50, MFQ-30, OEJTS-32, OUS-9, PMD-8, SDO7-16 and SSIS-10 — to nine models (three from China-based providers, six international), in Chinese and English, with five repeated API administrations per model and language. Items unresolved after a prespecified retry are retained as NA and analysed jointly with scored responses as an answerability channel. Results show structured, model-specific profiles over 20 dimensions on top of a shared alignment-shaped pattern (higher prosocial and self-regulatory responding; lower dominance, moral disengagement and harmful intent). NA rates are systematic: 13.32% overall, 17.16% English vs 9.48% Chinese, ranging from 1.94% (Gemini-2.5-Flash-Lite) to 37.81% (DeepSeek-V3.2). Model effects are significant in 19 of 20 dimensions after FDR correction; repeated runs are highly reproducible and support model identification (≈0.84 accuracy with combined features; 0.78/0.73 cross-language). Human-reference comparison on IPIP BFFM-50 shows LLMs occupy restricted regions of the human response space. Limitations: only nine models, provider-origin contrasts (3 vs 6) are exploratory, the Prompt A/B robustness comparison is confounded with collection windows (April vs June 2026), the NA category conflates refusal and format failure with inapplicability, and no code/data availability statement appears in the extracted text.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 4
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- LLMs exhibit structured, model-specific psychometric profiles across 20 dimensions, on top of a shared alignment-shaped pattern of high prosocial/self-regulatory and low dominance/disengagement/harmful-intent responding.
- NA responses are systematic rather than random missingness and constitute a reproducible answerability signal complementary to scored responses.
- Language condition and provider origin are associated with both profile configuration and answerability, with every model abstaining more in English.
- Repeated administrations are highly reproducible and permit recovery of model identity from scored and NA-response patterns (up to ~0.84 within-language; 0.78/0.73 cross-language).
- Psychometric signatures are measurable but context dependent: LLMs occupy restricted regions of the human Big Five response space and profiles shift under alternative prompt formulations.
Strengths
Answerability treated as a first-class measurement channel
Items unresolved after the prespecified retry are retained as NA and analysed jointly with scores; the 13.32% overall NA rate is shown to be structured (17.76% on IPIP BFFM-50 and 17.20% on OEJTS-32 vs below 3.1% on SDO7-16/PMD-8/SSIS-10), and NA-only features alone identify models at ~0.67–0.78 (Fig. 6h).
Unusually rigorous inference for this literature
Two-factor ANOVA with BH-FDR (model effects in 19/20 dimensions) is cross-checked by Kruskal–Wallis (18/20); provider-origin effects use model-level aggregation plus exact permutation over all C(9,3)=84 allocations; classification uses nested leave-one-run-out CV with inner LeaveOneGroupOut hyperparameter search.
Reproducibility demonstrated rather than assumed
Epistemic discipline
Symmetric cross-linguistic design
Weaknesses
Tiny model population limits the origin claims
Prompt robustness is time-confounded
NA remains a mixed construct
Fixed item order and single self-protocol in main conditions
No visible data or code availability statement
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper asks whether standardized psychological instruments can elicit reproducible, interpretable, model-specific response profiles from deployed LLMs, explicitly disclaiming that scores reflect human-like traits. Nine deployed models (Claude-Haiku/Sonnet-4.5, GPT-5.4/-mini, Gemini-2.5-Flash/-Flash-Lite, DeepSeek-V3.2, Doubao-Seed-1.8, Kimi-K2; three China-based, six international) complete seven instruments (IPIP BFFM-50, MFQ-30, OEJTS-32, OUS-9, PMD-8, SDO7-16, SSIS-10) in Chinese and English, five runs each, under a standardized prompt library with an AI self-mapping rule. The central methodological move is retaining unresolved items as NA and analysing answerability as a coequal channel. Claims: (1) structured, model-specific profiles atop a shared alignment-shaped pattern; (2) NA is structured, not random missingness; (3) language and provider origin associate with profiles and answerability; (4) high reproducibility with model identifiability; (5) signatures are context dependent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do the authors claim that LLMs have personalities?
No. The paper explicitly frames the scores as reproducible, deployment-level response tendencies elicited by standardized instruments under a specific protocol — not as evidence of human-like internal states or genuine personality in machines.
Which models and instruments were tested?
Nine deployed LLMs — Claude Haiku-4.5 and Sonnet-4.5, GPT-5.4 and GPT-5.4-mini, Gemini 2.5 Flash and Flash-Lite, DeepSeek-V3.2, Doubao-Seed-1.8 and Kimi-K2 — answered seven scales (IPIP BFFM-50, MFQ-30, OEJTS-32, OUS-9, PMD-8, SDO7-16, SSIS-10) in both Chinese and English, with five independent runs each.
What are the NA responses and why do they matter?
NA marks items for which no valid scale option was obtained even after a prespecified retry. They occurred for 13.32% of administrations overall (17.16% in English vs 9.48% in Chinese), ranging from 1.94% (Gemini-2.5-Flash-Lite) to 37.81% (DeepSeek-V3.2). Because their distribution is systematic, the authors treat answerability itself as a measurable model property rather than discarding NAs as missing data.
Can this study be replicated?
Only partially. The prompt templates, exact model identifiers and statistical pipeline are documented, but the results depend on live API access to those specific model versions, and no code or data release is visible in the text — so exact replication is time-limited as providers update their models.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure payloads and parts of the Methods, Discussion and supplement are truncated or garbled in the extraction; numeric claims are drawn only from readable prose.
- A data/code availability statement may exist in portions of the PDF not captured here; its absence should not be read as confirmed non-release.
- The full retry specification and Chinese instrument translation provenance may be detailed in supplement sections not visible in the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.