Do Personality-Tuned LLMs Make Better Social Agents?
cs.CL · cs.AI
Quick verdict
Fine-tuned 7-8B LLMs fail to beat instruction-tuned baselines at MBTI personality role-play, but low judge agreement (α<0.8) weakens the null result.
Summary
The paper asks whether personality-aware fine-tuning makes small LLMs better at role-playing conditioned personalities than instruction prompting alone, motivated by social simulation and interactive robotics. The authors LoRA fine-tune Qwen2.5-7B-Instruct and Ministral-8B-Instruct-2410 (ranks 16 and 32) on roughly 582,000 samples combining MBTI-labelled Kaggle forum posts with DailyDialog dialogues pseudo-labelled by RoBERTa classifiers, augmenting the imbalanced N/S dimension via backtranslation. Each of six model variants generates 225 dialogues across three urgency-graded scenarios and five personality pairs; three LLM judges (Qwen3-32B, Qwen3-30B-A3B, Mistral-Small-3.1-24B) perform Mayring-style deductive coding before inferring MBTI labels. Results: judges classify baseline outputs significantly better than fine-tuned outputs (macro F1 up to 0.820 for the Ministral base on F/T versus at most 0.738 for its fine-tuned variants; McNemar significant except Qwen rank-16 on F/T), so fine-tuning does not improve personality fidelity. Krippendorff's alpha stays below 0.800 for all final judgements, limiting interpretation. Fine-tuning raises Qwen's English rate from 88.35% to about 92% and lexical diversity (Distinct-2 from 0.427 to about 0.552). Key limitations: fidelity is an unvalidated judge proxy, agreement is low, training labels are noisy pseudo-labels with a social-media-to-dialogue domain shift, only LoRA rank and alpha were varied, and no code or prompts are released.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Personality-aware LoRA fine-tuning does not improve MBTI role-play fidelity over instruction-tuned baselines for two 7-8B models; judges classify baseline outputs significantly better (McNemar, except Qwen rank-16 on F/T).
- Inter-rater agreement among the three LLM judges stays below Krippendorff's 0.800 threshold for all final judgements, limiting confidence in the fidelity results.
- Fine-tuning is linguistically safe: Qwen's English rate rises from 88.35% to about 92% and Distinct-2 from 0.427 to about 0.55, while Ministral remains above 99% English.
- Scenario urgency does not systematically determine personality fidelity (Table IV shows no monotonic relationship).
- The null outcome is attributed primarily to training-data domain mismatch (social media posts versus dialogue) and label noise.
Strengths
Honest negative result with reliability caveats built in
Abstract, section IV.C and the discussion explicitly state that fine-tuning does not help and that Krippendorff's alpha below 0.800 blocks robust conclusions - rare transparency compared with the usual fine-tuning-win narrative.
Evidence-based multi-judge evaluation design
Three judges from two model families including one MoE (Qwen3-32B, Qwen3-30B-A3B-Instruct-2507, Mistral-Small-3.1-24B) perform Mayring-inspired deductive coding with quoted text segments before labelling (Figs. 4-5), not bare label matching.
Reliability is measured, not assumed
Data-construction diligence
Side-channel quality metrics catch real artifacts
Weaknesses
Headline metric is an unvalidated proxy measured by unreliable judges
Detectability confound favors baselines
Noisy training pipeline, unquantified
Narrow ablation space
Thin statistical and reproducibility reporting
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.21857, cs.CL/cs.AI) tests one question: does personality-aware fine-tuning yield more consistent, controllable personality-conditioned dialogue than instruction prompting alone (Section I)? The context is LLM-driven social simulation and socially interactive robotics, framed via Anthis et al.'s 'alienness' concept. The authors LoRA-tune Qwen2.5-7B-Instruct and Ministral-8B-Instruct-2410 and evaluate four tuned plus two base variants across three urgency-graded scenarios. Central claims: (1) LLM judges classify baseline outputs significantly more accurately than fine-tuned outputs, i.e., fine-tuning does not improve personality role-play; (2) judge agreement (Krippendorff's alpha) stays below 0.800 for all final judgements, so results warrant caution; (3) fine-tuning is linguistically safe - Qwen's English-rate and lexical diversity improve, Ministral stays above 99% English; (4) the null is attributed to training-data domain mismatch. Claim (1) is honestly stated as a negative result, which is commendable; its epistemic weight, however, is capped by the evaluation design discussed below.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Did fine-tuning improve personality control in LLMs?
No. In this study, LoRA fine-tuning on MBTI-labelled data made both Qwen2.5-7B and Ministral-8B less recognizable to three LLM judges than their instruction-tuned baselines; McNemar tests favored baselines on nearly all dimensions (the only exception was Qwen with the smaller LoRA rank on the Feeling/Thinking dimension). The authors attribute this to training-data domain mismatch (forum posts versus dialogue) and stress that the evaluation itself has low inter-judge reliability.
Is MBTI a valid personality framework for LLM experiments?
The authors acknowledge that MBTI's reliability and validity are criticized in psychology, but chose it because its four binary dimensions are simpler to classify and evaluate than the Big Five's five continuous traits. Importantly, they treat MBTI labels as behavioral conditions for testing controllability, not as psychological ground truth.
How reliable was the LLM-as-a-judge evaluation in this paper?
Weak. Krippendorff's alpha among the three LLM judges was below the recommended 0.800 for every final personality judgement across all six model variants. On the identical text segments that judges happened to code, alpha ranged from 0.600 (F/T, only n=3) to 0.847 (J/P, n=9). The authors flag this as a core limitation of their negative result.
Is code or data available for this paper?
The paper does not announce a code release. The training datasets are public (the Kaggle MBTI dataset under CC0 and DailyDialog), the base models are open-weight, and LoRA hyperparameters are tabulated, but the judge prompts, exact scenario texts, decoding parameters and seeds are not fully published, which limits replication.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- The abstract is truncated mid-sentence in the extracted text ('the baseline models offer the best...'); any concluding section beyond the discussion may be missing.
- Figures 1-5 are images; exact scenario wordings, the pipeline diagram and the judge prompt exemplars could not be verified beyond the prose descriptions.
- Reported as arXiv v1 (Sep 2026); no appendix was available, so some implementation details may exist outside the extracted text.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.