Personas Differ from Native-Language Generation: Language Pathways Shape LLM Interpersonal Advice
cs.CL · cs.AI
Quick verdict
Large study (600 Qs, 13 languages, 8 LLMs): native-speaker personas ≠ native-language generation for LLM advice; main caveat is machine-translation confound.
Summary
The paper tests whether two common cross-lingual elicitation strategies for LLM advice are interchangeable: generating advice in the target language and translating it to English (NL), versus prompting in English with a native-speaker persona (NP). Using 600 Interpersonal Skills Stack Exchange questions (150 each on work, friends, relationships, family) translated into 12 languages via a back-translation-filtered NLLB pipeline (mean similarity 0.928), and eight LLMs (five open-weight, three proprietary), the authors generate and annotate roughly 235,200 responses. Compared to NL, NP raises lexical social cues (affiliation, prosocial language, positive tone) but lowers concreteness, social attunement, emotional expressiveness, and formality, and consistently reduces behavioral scaffolding: all eight BCT dimensions decrease (composite NP–NL −0.44, Table 9). In forced choice, NP selects confrontation more often (32.3% vs 28.8%) and redirection less often (53.4% vs 57.6%) than NL, with NL departing further from the English baseline. Effects vary by language, topic, and model. Robustness checks cover persona wording, a translation-quality filter (similarity ≥ 0.95), and a second judge (claude-sonnet-5). Main limitations: all measurement is automated (LIWC plus a gpt-4o judge that is itself a tested generator); NL responses are machine-translated before analysis, so translationese is a possible confound; source questions come from an English-language forum; there is no human validation; and no code or data release is stated.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Native-speaker persona prompting (NP) is not interchangeable with native-language generation plus translation (NL) for multilingual interpersonal advice.
- NP amplifies lexical social cues (affiliation, prosocial language, positive tone) while reducing concreteness, social attunement, emotional expressiveness, and formality relative to NL.
- NP provides less behavioral scaffolding than NL across all eight BCT dimensions (composite NP–NL −0.44, Table 9).
- NP shifts forced-choice recommendations toward confrontation (32.3% vs 28.8%) and away from redirection (53.4% vs 57.6%), with effect sizes varying by language, topic, and model.
- These are properties of model elicitation, explicitly not evidence about real language communities.
Strengths
Matched paired design at genuine scale
600 questions × 12 languages × 8 models × 4 pathways ≈ 235,200 annotated rows (App. B.2 denominators), with NP–NL tested within matched prompt–model pairs (Sec. 3.6), controlling question difficulty and model idiosyncrasy.
Four-layer measurement separates style from substance
LIWC lexical cues, gpt-4o five pragmatic dimensions (App. B.1), eight BCT scaffolding clusters, and forced-choice actions converge: NP is lexically warmer yet less concrete, attuned, and actionable (Fig. 3, Fig. 5, Table 9 composite −0.44).
Auxiliary pathways localize the mechanism
Robustness battery is substantive, not decorative
Explicit non-claims about culture
Weaknesses
Translation confound sits inside the NL condition
Judge–generator overlap
Large-N significance without effect-size framing
No artifact release
Redirection is a catch-all action category
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets a narrow but consequential methodological question in multilingual LLM evaluation: when teams want culturally flavored interpersonal advice, is asking a model to answer as a native speaker (NP) interchangeable with having it generate in the target language and machine-translating the answer to English (NL)? The evidence base is 600 Interpersonal Skills Stack Exchange questions (150 each across work-environment, friends, relationships, family; Table 1), 12 target languages plus English, and eight models (Gemma-4-E4B/26B-A4B/31B, Qwen3.6-27B/35B-A3B, GPT-4o, GPT-5.5, Claude Opus-4.6), yielding 2,400 records per model per non-English language and 235,200 annotated rows (App. B.2). Central claims: NP and NL diverge on linguistic style, behavioral scaffolding, and action choice; NP amplifies lexical social cues while reducing concreteness, social attunement, emotional expressiveness, and formality; NP shifts forced choice toward confrontation. The authors explicitly scope results as elicitation properties, not claims about real speakers (Sec. 1, Limitations) — a commendable non-claim in this literature.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
Do 'answer as a native speaker' prompts make LLMs give culturally different advice?
According to this paper, such persona prompts do not reproduce what models produce when actually generating in the target language. Across 600 advice questions, 12 languages, and 8 models, personas added warmer lexical cues (affiliation, positive tone) but produced less concrete, less socially attuned advice with less actionable guidance, and shifted forced-choice recommendations toward confrontation (32.3% vs 28.8%).
Which is better: persona prompting or generating in the target language?
The paper deliberately does not claim either pathway is better — validating them against real native speakers is left to future work. Its finding is that they are not interchangeable: the choice changes both how advice is phrased and which actions the model recommends.
Which models and languages were tested?
Eight LLMs — Gemma-4-E4B, Gemma-4-26B-A4B, Gemma-4-31B, Qwen3.6-27B, Qwen3.6-35B-A3B, GPT-4o, GPT-5.5, and Claude Opus-4.6 — across 13 languages: English plus Japanese, Korean, Chinese, Arabic, Persian, Turkish, German, Dutch, Swedish, Spanish, Italian, and Portuguese.
How were the translated questions validated?
Each question was translated by NLLB-200 with 32 candidate translations; each candidate was back-translated to English four times and scored with MiniLM cosine similarity; the candidate with the most stable back-translations among the five most similar was kept. Mean similarity was 0.928 overall (0.876 Japanese to 0.976 Spanish).
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 4's Arabic and Persian instruction cells render as garbled characters in the extracted text (PDF font-encoding artifact); their content could not be verified.
- Values embedded in figures (Figures 3–8, 11–24) are not recoverable from text; figure-based judgments in this review rely on captions and the appendix tables.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.