Assessing the Applicability of Existing Design Recommendations to AI Companion Design: A Multi-Method Study
Soobin Cho, Deveshi Modi, Divya Mavinkurve, Jieqiong Ding, Mark Zachry
cs.HC · cs.AI
Quick verdict
A multi-method study arguing existing design guidance transfers poorly to AI companions; process is well documented, but stats rest on 3-4 raters per product.
Summary
This paper asks why design guidance from adjacent domains does not transfer cleanly to AI companion design, reporting a four-phase multi-method analysis: a purposive literature review yielding 548 recommendations from 41 sources; collaborative coding with 11 HCI practitioners (95 judged inapplicable, 453 applicable); thematic synthesis into 22 design principles across nine areas, probed by an internal heuristic evaluation of Replika and Pi AI with two evaluators each; and a modified heuristic evaluation by 14 external experts rating applicability, clarity, and importance for Replika, Pi AI, ChatGPT, and Microsoft Copilot. Experts rated principles highly applicable (M=4.14) and important (M=4.25) but less clear (M=3.92). Companion systems scored higher on applicability than general-purpose systems (4.27 vs 4.01), though not significantly (p=.224); area-level tests found differences only for Inclusivity (p=.001, g=2.46) and Predictability and Consistency (p=.022, g=1.42). The accepted manuscript is published in the International Journal of Human-Computer Studies. Conclusions: ethical and UX considerations are deeply intertwined, and principles demand context-sensitive application. Limitations, largely self-acknowledged: short one-off engagements, junior practitioners without companion-design experience, only two products in the internal evaluation, no inter-rater reliability metrics, and inferential tests on 3-4 raters per product without multiple-comparison correction.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing design recommendations from adjacent domains (general HCI/UX, human-AI interaction, agent/chatbot) cannot be directly transferred to AI companion design.
- Nine design principle areas serve as an analytic artifact that localizes where recommendation applicability breaks down for AI companions.
- Ethical and UX-oriented considerations are deeply intertwined in AI companion design, with UX principles able to create ethical risk and ethical principles shaping experience.
- Design principles require context-sensitive, flexible application rather than fixed one-size-fits-all prescriptions.
- AI companions present a distinctive risk profile (dependency, manipulation, self-harm contexts) that warrants companion-specific ethical safeguards absent from current guidance.
Strengths
Transparent, staged multi-method design
Each phase produces an intermediate artifact feeding the next (Fig. 1): 548 recommendations → 453 applicable → 22 principles → expert ratings. The limitations section (§8.3) is unusually candid, explicitly framing the nine areas as an analytic artifact rather than validated guidance.
Large, categorized recommendation corpus
Table 1 breaks down 548 recommendations from 41 sources (Human-AI interaction 229; Agent/chatbot 161; HCI/UX 101; AI companion only 57 — itself evidence of the field gap), and §5.3 gives concrete cited rejection examples (e.g., 'Actions should be reversible' rejected as unnatural for relationship-building).
Concrete, checkable product evidence
Expert evaluation spans both system types
Analytic reconciliation of phase disagreement
Weaknesses
Underpowered, uncorrected inferential tests
No reliability reporting for qualitative analyses
Circular internal evaluation
Non-reproducible corpus and pipeline
Modest novelty of headline findings
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a problem-analysis paper, not a design-guideline paper: it asks why existing design recommendations do not transfer cleanly to AI companion design rather than proposing new guidelines. Its claims are that (1) adjacent-domain guidance (general HCI/UX, human-AI interaction, agent/chatbot) cannot be applied as-is; (2) nine principle areas (Table 2) localize where tensions arise; (3) ethical and UX considerations are intertwined; and (4) context-sensitive application is required. Scope is appropriately modest — §8.3 explicitly disclaims the principles as a validated toolkit. The deliverable is the documented four-phase process plus the artifact.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What are the nine design principle areas for AI companions identified in this paper?
Safety; Transparency; Inclusivity; Predictability and Consistency; Interaction Controllability; Adaptation and Personalization; Engagement; Empathetic Response; and Response Length. They were synthesized from 548 design recommendations in the literature, refined via an internal heuristic evaluation of Replika and Pi AI, and rated by 14 external domain experts. The authors present them as an analytic artifact for understanding design tensions, not as a validated design checklist.
Did the study find differences between AI companion apps and general-purpose AI like ChatGPT?
Ratings were broadly similar. Applicability averaged 4.27/5 for companion systems (Replika, Pi AI) versus 4.01/5 for general-purpose systems (ChatGPT, Copilot), a difference that was not statistically significant (p=.224). At the principle-area level, only Inclusivity (p=.001, g=2.46) and Predictability and Consistency (p=.022, g=1.42) differed significantly, both rated more applicable to companion systems. Note the tests used only 3-4 raters per product, so statistical power was low.
Why can't existing design guidelines be applied directly to AI companions?
The study's central finding is that ethical and user-experience considerations are intertwined: for example, frequent AI-identity disclosures support honesty but degrade the relational experience, while highly engaging, adaptive, empathetic behavior improves experience but risks dependency or harmful validation. The authors conclude that principles require context-sensitive application rather than fixed rules, and that current guidance lacks companion-specific ethical safeguards.
What products were evaluated and how, in this AI companion design study?
Replika (v10.0.1) and Pi AI (v1.0.55) underwent an internal heuristic evaluation by two evaluators each. Then 14 external experts each used one of Replika, Pi AI, ChatGPT (GPT-4o), or Microsoft Copilot, rating the 22 principles on applicability, clarity, and importance (5-point scales) in one-off sessions of roughly 60-90 minutes, with open-ended feedback per principle.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 (study design, rating distributions, integrative overview) could not be analyzed from extracted text; judgments rely on captions and body text.
- Reference list shows minor extraction corruption (e.g., Cassell 1999, Boine 2023 entries malformed); citation-quality judgments were avoided on those entries.
- Supplementary materials (corpus, evaluator guides) were not part of the extracted text; availability statements reflect only the manuscript body.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.