CHARM: Character Hallucination for Multicultural Role Play Benchmark
Sunkyung Han, Nahyeon Park, Gaeun Seo, Seunghyun Yoon, JinYeong Bak
cs.CL · cs.AI
Quick verdict
CHARM 发现角色扮演中的幻觉主要是合规性失败，而非认知缺乏；这是一个可靠的多文化基准测试，但存在一些不一致的表格。
Summary
CHARM 是一个包含 40 个真实与虚构角色的基准测试，涵盖了五个文化语言区域（英美区、西班牙、中国、韩国、印度尼西亚；每个区域 8 个，在真实状态和时期上保持平衡），用于诊断角色扮演 LLM 中的角色幻觉。它通过启用弃权选项的选择题，探究了时间边界（20 个历史角色 vs 30 个现代概念）和跨宇宙边界（角色叙事宇宙之外的实体）。一个两阶段框架区分了边界感知（明确的“是/否”识别）和边界遵从（五个选项中的弃权），并通过三步检查（感知、事实正确的非弃权、在被提示为目标时给出正确答案）来验证参数化覆盖。在六个 LLM（GPT-4o、GPT-5.5、Gemini-3.5-Flash、Llama-3.1-8B、Gemma-3-12B、Qwen3-8B）上的实验中，合规缺口在识别失败中占主导地位（GPT-4o：72.1% vs 8.9%），且 50.5–100% 的事实回答缺口案例被验证为参数化覆盖；跨宇宙边界始终比时间边界更难。西方区域的角色显示出比韩国和印度尼西亚角色（约 39%）更高的缺口率（西班牙为 58.5%）。每个区域的两名母语审查员验证了全部 2,748 个项目（Cohen's κ 0.55–0.79）。主要局限性：缺乏显著性检验或运行层面的方差、正文表 4 与附录表 17/18 之间存在无法解释的数值不匹配、干扰项相似性分析（表 14）使用了主研究中未出现的模型，以及选择题弃权范式的固有限制。
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Across six LLMs, character hallucination is driven predominantly by boundary-compliance failures rather than boundary-recognition failures (e.g., GPT-4o: 72.1% compliance gap vs. 8.9% recognition failure, Table 2).
- For five of six models, 78-100% of factually answered compliance-gap cases verify as parametric overrides: the model holds the fact but fails to suppress it under role constraints (Tables 3/10); the full range across all six is 50.5-100%.
- Cross-Universe boundaries produce consistently higher compliance-gap rates than Temporal boundaries for all six models (Table 6).
- Compliance-gap and override rates vary systematically by cultural region, with Western (EN/Spain) characters failing more than Korean and Indonesian ones (Table 4).
- Independent-run measurement of BA and BC is a conservative lower bound on compliance, since sequential BA-to-BC context raises GPT-4o's Cross-Universe BC accuracy from 10.4% to 80.6% (Appendix M).
Strengths
Genuine diagnostic decomposition
The BA-BC matrix (Table 1) separates Compliance Gap from Recognition Failure; the honest Appendix M control (GPT-4o Cross-Universe BC accuracy 10.4% to 80.6% when BA precedes BC in-context, N=732) reframes independent runs as a conservative lower bound rather than hiding the context confound.
Multi-step override verification with controls
Three conditions (BA=True, factually correct BC, correct KVQ as the target; Appendix G, Table 11) plus a no-role control (Table 16: 64.4-99.5% accuracy) triangulate that models hold the fact but fail to suppress it, going well beyond outcome-only hallucination counts.
Documented multicultural construction
Headline numbers are internally consistent
Candid limitations
Weaknesses
No statistical inference
Main-text Table 4 inconsistent with appendix
Table 14 provenance problem
Abstention is the only rewarded behavior
Single-template BA probe
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CHARM (arXiv 2609.01352) is both a benchmark and a diagnostic framework for character hallucination: role-playing LLMs leaking knowledge the assigned character cannot possess. The core methodological move is decomposing one outcome (answering an out-of-scope question) into two stages: Boundary-Awareness (BA, explicit yes/no recognition that a target is out of scope) and Boundary-Compliance (BC, abstaining among five MCQ options where abstention is correct). The paper claims: (1) hallucination is predominantly compliance failure (Table 2: GPT-4o 72.1% compliance gap vs. 8.9% recognition failure); (2) most factually answered gaps are parametric overrides, verified in three steps (Tables 3/10: 50.5-100% across the six models); (3) Cross-Universe boundaries are harder than Temporal ones for every model (Table 6); (4) failure rates vary by cultural region, Western characters failing most (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is character hallucination in role-playing LLMs?
It is when a model role-playing a character (e.g., Shakespeare) produces knowledge that character could not have - such as answering questions about smartphones or about characters from other fictional universes. CHARM measures how often this happens and diagnoses why.
What is a parametric override in the CHARM benchmark?
A case where the model acknowledges a fact lies outside the role character's knowledge, yet still answers correctly in-character, and also answers correctly when prompted as the target character - showing it holds the fact but fails to suppress it during role-play.
Which models were evaluated in CHARM?
Six: GPT-4o, GPT-5.5, and Gemini-3.5-Flash (closed-source), plus Llama-3.1-8B-Instruct, Gemma-3-12B-IT, and Qwen3-8B (open-source), each answering 2,748 yes/no and five-choice questions across 40 characters from five cultural regions.
Is the CHARM dataset publicly available?
The paper links a GitHub repository (github.com/sunkyoung19/CHARM) for the dataset; all prompts, model settings, and validation criteria are documented in the appendices.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1's embedded text is garbled by PDF extraction; its logic was reconstructed from the caption and Section 2 body text. All tables and appendices (through Table 22) extracted cleanly.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.