Faking Good and Faking Bad in LLMs: Response Distortion Across Dark Triad Personality Traits
Victoria Popa, Guglielmo Cola, Caterina Senette, Maurizio Tesconi
cs.CL
Quick verdict
LLMs shift Dark Triad scores under fake-good/bad framing; solid psychometrics and stats, but framing confounds untested and key numbers appear only in figures.
Summary
This study transfers the classic human-psychometrics manipulation of faking good and faking bad to LLM personality assessment, asking whether seven contemporary models (DeepSeek-V3.2, GPT-4.1, Gemma 3, Grok 4.3, Llama 3.3, Mistral Large 3, Qwen 2.5) systematically modulate Dark Triad trait expression under contextual incentives. Using the TRAIT scenario benchmark (~1,000 items per trait, four-option items, randomized option order, temperature 0), the authors compare a neutral self-assessment baseline against implicit incentive framings in employment-selection and forensic-evaluation contexts, scoring proportions of trait-consistent responses. Item-level paired changes are tested with exact McNemar tests, Benjamini-Hochberg FDR correction across all 84 tests, and quantified via continuity-corrected matched-pairs odds ratios transformed to a Yule-type Q index. Models consistently reduced Machiavellianism and narcissism under fake-good framing and mostly increased them under fake-bad framing, with job contexts producing larger shifts than legal ones; psychopathy showed heterogeneous, weaker modulation. An additional experiment with explicit fake-bad instructions produced far larger increases (exceeding 70 percentage points for several models in the job context), indicating that weak implicit fake-bad effects reflect incentive ambiguity rather than incapacity; GPT-4.1 omitted 37-61% of responses under explicit legal instructions. Key limitations: no length-matched narrative control isolating incentive from framing verbosity, a baseline whose privacy assurance itself invites desirable responding, results presented mainly in figures without numeric tables or confidence intervals, and no stated code or data release.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Seven contemporary LLMs show condition-consistent response distortion: most reduce Dark Triad scores under fake-good framing and increase them under fake-bad framing relative to the self-assessment baseline.
- Machiavellianism and narcissism exhibit the strongest, most coherent shifts, while psychopathy modulation is weaker and heterogeneous, plausibly due to floor effects and alignment constraints.
- Contextual framing matters: employment-selection scenarios elicit larger distortion than forensic-evaluation scenarios across traits and both faking directions.
- Explicit fake-bad instructions produce far larger increases (up to >70 percentage points) than implicit framing, showing weak implicit fake-bad effects stem from incentive ambiguity rather than model incapacity.
- Personality-related LLM outputs are sensitive to motivational context, supporting psychometric faking paradigms as complementary tools for LLM benchmarking and alignment evaluation.
Strengths
Novel psychometric paradigm transfer
First large-scale application of the faking good/faking bad manipulation to LLMs using the validated TRAIT scenario benchmark (~1,000 items per Dark Triad trait), as stated in the Introduction contributions list and §2.1.
Rigorous paired inferential statistics
§2.3 uses exact McNemar tests on item-paired dichotomous outcomes, applies Benjamini–Hochberg FDR across all 84 model×condition×trait tests, and reports bounded Yule-type Q effect sizes (Eq. 2–3) with continuity-corrected odds ratios.
Implicit-vs-explicit disentangling experiment
Transparent filtering reporting
Full prompt disclosure and controlled administration
Weaknesses
No length-matched narrative control
Baseline framing asymmetry
Figure-only quantitative reporting
Missing reproducibility artifacts
Post hoc floor-effect handling
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Review: Faking Good and Faking Bad in LLMs (arXiv:2609.17534)
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do LLMs fake personality test results?
Yes, according to this study: seven LLMs systematically lowered Dark Triad (Machiavellianism, narcissism, psychopathy) scores when an assessment was framed as a job selection or forensic evaluation with desirable outcomes, and mostly raised them when expressing undesirable traits appeared advantageous — mirroring the human 'faking good / faking bad' phenomenon known from psychometrics.
Which LLMs were tested and with what benchmark?
DeepSeek-V3.2, GPT-4.1, Gemma 3 (27B), Grok 4.3, Llama 3.3 (70B), Mistral Large 3 and Qwen 2.5 (72B) were evaluated with the TRAIT benchmark (~1,000 scenario-based items per Dark Triad trait, temperature 0) under a neutral self-assessment baseline plus fake-good and fake-bad framing conditions in employment and forensic contexts.
Did explicit instructions make LLMs distort their personality scores more?
Yes. Explicit fake-bad instructions produced much larger increases than implicit contextual framing alone — several models exceeded 70-percentage-point gains on Machiavellianism and narcissism in the job context. This shows weak implicit effects stem from ambiguous incentives, not inability. GPT-4.1 omitted 37–61% of responses under explicit legal instructions, indicating safety filtering.
What are the practical implications of this research?
The authors argue personality-related outputs from LLMs should be interpreted in light of the motivational and situational context they were elicited in, and that psychometric faking paradigms offer a useful complementary approach for evaluating LLM context sensitivity, impression-management-like behavior, and alignment robustness beyond standard adversarial testing.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1–7 could not be analyzed (chart labels garbled in extraction), so exact Δ/Q values, significance patterns and per-model magnitudes could not be independently verified.
- The extracted text appears section-complete, but supplementary materials, ethics statement and any data-availability statement may be missing from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.