CompanionSim: Synthetic Data for Evaluating Anthropomorphism in Human-AI Relationships
Jacy Reese Anthis, Mark Díaz, Renee Shelby
cs.CY · cs.AI · cs.CL · cs.LG
Quick verdict
Releases 2,240 simulated dialogues; studies (N=628, 3,646) find chatbot companionship behaviors reduce likability and trust; solid data, observer-only scope.
Summary
The paper introduces CompanionSim, a synthetic-data pipeline producing 2,240 multi-turn human–chatbot conversations crossing 16 elicited chatbot behaviors (13 companionship behaviors, two anti-companionship behaviors, one control) with seven use cases from the Taxonomy of User Needs and Actions. Conversations were generated in June 2025 with Gemini 2.5 Flash, seeded with fixed first turns, style examples from WildChat/ShareGPT/LMSYS/r/replika/EmpatheticDialogues, and randomized context facts. Two annotation studies collected Godspeed-derived likability and humanlikeness ratings plus affective/cognitive trust items: Study 1 with 628 U.S. participants (Merkle panel, preregistered) and Study 2 with 3,646 participants across the U.S., U.K., India, and Nigeria (Qualtrics). Generalized linear mixed models (random effects for conversation and participant) show companionship behaviors significantly reduced likability and cognitive trust in Study 1 and all four ratings in Study 2; the largest negative effects came from user relationship claims and invalidation, and women, older, and less-frequent AI users showed larger reductions. An automated judge confirmed the intended behavior in 84% of prompted conversations, and simulated conversations were rated as natural as or more natural than real-world logs. Limitations: a single simulator model, third-party observers rating transcripts rather than interacting users, small pooled effect sizes, and no IRB (legal/ethics review instead). Dataset and code are public, making re-analysis straightforward.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 4 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Eliciting companionship behaviors in simulated chatbot conversations significantly reduces third-party annotators' likability and cognitive trust (Study 1) and all four ratings including humanlikeness and affective trust (Study 2).
- Effects are heterogeneous: women, older participants, and infrequent AI users show larger reductions across ratings.
- The pooled effect is driven mainly by user relationship claims and invalidation, supporting the disentangling of individual humanlike features rather than treating anthropomorphism as one lever.
- Simulated conversations are perceived as at least as natural as real-world comparison logs, validating synthetic data as a viable instrument for human–AI interaction research.
Strengths
Factorial, behavior-isolating design
16 behaviors × 7 use cases × 20 runs (2,240 conversations) with 140 fixed first turns and 180 randomized context facts (§3.1, Tables 1–2) lets the authors isolate behaviors that are sparse in organic corpora like WildChat/ShareGPT
Large, cross-country human evaluation with preregistration
Study 1 (N=628, U.S. Merkle panel, preregistered at AsPredicted) replicated in Study 2 (N=3,646; U.S./U.K./India/Nigeria via Qualtrics); 44,467 simulated-conversation annotations analyzed with GLMM random effects for conversations and participants
Candid manipulation validation
Full openness enabling re-analysis
Weaknesses
Construct validity of behavior effects
Third-party observer scope vs. headline framing
Small pooled effects carried by two face-valid behaviors
Validation and internal-consistency gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
CompanionSim (arXiv 2609.00250) is a dataset-plus-studies paper in HCI/AI safety. The authors build a synthetic dialogue pipeline crossing 16 chatbot behaviors — 13 companionship behaviors (Linehan validation levels 2–6, relationship claims, five self-attribution types), two anti-companionship behaviors (invalidation, AI disclaimer), and a no-elicitation control — with seven use cases from the Taxonomy of User Needs and Actions. Twenty generation runs per cell yield 2,240 conversations (Gemini 2.5 Flash, June 2025). Two annotation studies measure third-party perceptions: Study 1 (N=628, U.S. Merkle panel, preregistered) and Study 2 (N=3,646 across U.S./U.K./India/Nigeria, Qualtrics). The headline is counterintuitive: companionship behaviors reduced likability (β=−0.08) and cognitive trust (β=−0.13) in Study 1 and all four ratings in Study 2 (β −0.03 to −0.06), with larger reductions among women, older, and less-frequent AI users.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 4 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CompanionSim dataset?
A publicly released corpus of 2,240 multi-turn human–chatbot conversations generated with Gemini 2.5 Flash, crossing 16 chatbot behaviors (13 companionship behaviors, two anti-companionship behaviors such as invalidation, and a no-elicitation control) with seven use cases, plus human annotations of likability, humanlikeness, and affective/cognitive trust from two studies (N=628 in the U.S.; N=3,646 across U.S., U.K., India, and Nigeria).
Does the paper show that AI companionship makes users trust chatbots less?
Not directly. The 4,274 participants were third-party observers who read transcripts rather than conversing themselves. In that observer setting, companionship behaviors lowered rated likability, humanlikeness, and trust; the authors explicitly note that effects on direct users in live interaction may differ and were not measured.
Which chatbot behaviors received the worst ratings?
Chatbots claiming a romantic relationship with the user ('user relationship claim'; β=−0.34 on likability and −0.42 on humanlikeness in Study 1) and those invalidating the user (β=−0.37 on likability, Study 1) produced the largest negative effects; the pooled companionship effect across the other behaviors was much smaller (β −0.03 to −0.08).
Are the conversations real human–AI chats?
No. They are generated by Gemini 2.5 Flash (June 2025), seeded with style examples from real sources (WildChat, ShareGPT, LMSYS, r/replika, EmpatheticDialogues) and validated for naturalness against 70 real conversations; participants rating them were not told the conversations were synthetic.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Regression tables for the §4.2 heterogeneous-effect (gender/age interaction and men/women/older subgroup) models and for Study 1 baseline demographics appear only as prose β values; those tables are not among the embedded appendix material (Tables A1–A6 cover behaviors/use cases and Study 2 baselines).
- Values behind Figures 2–5 exist only as plotted means and error bars; per-figure numbers beyond those quoted in the text/captions are not recoverable from the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.