Annie, Are You Okay? How Style- and Context-Based Personalization Shape AI-Assisted Decision-Making
Hasibur Rahman, Hasibur Rahman, Hasibur Rahman, Benjamin R. Cowan, Smit Desai, Smit Desai
cs.HC · cs.AI
Quick verdict
Claims context-based personalization sways stock choices with no trust change; preregistered N=240 experiment is well controlled, with minor hypothetical-stakes caveats.
Summary
This preregistered HCI experiment asks whether two forms of AI personalization shape decisions independently of the advice content itself. 240 Prolific participants ranked three anonymized pharmaceutical stocks, discussed them with a GPT-based agent advocating a counterbalanced ranking, and reranked. A 2x2 between-subjects design independently varied style-based personalization (communication matched to Big Five traits) and context-based personalization (arguments framed around demographics and values) while holding the recommendation, evidence, and six-stage argument sequence constant. Context-based personalization raised reconsideration from 35.0% to 59.2% and roughly doubled the odds of movement toward the AI's ranking (OR=2.04); style-based personalization showed no reliable decision effect. Neither form changed seven evaluative ratings of the AI; only perceived influence rose. Exploratory analyses suggest greater susceptibility among lower-expertise participants. Main limitations: hypothetical, single-session stock task; a single model; and self-reported expertise.
Review complete
Assessment at a glance
3
risk flags identified
● high 0
● medium 1
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Context-based personalization reliably changed ranking behavior: reconsideration rose from 35.0% to 59.2% and the odds of movement toward the AI's ranking roughly doubled (OR=2.04, 95% CI [1.23, 3.38]).
- Style-based personalization was perceived but produced no statistically reliable effect on any decision outcome after Holm correction.
- Neither personalization form changed seven evaluative ratings of the AI; only perceived influence increased (d=0.41, pHolm=.015).
- Participants farther from the AI's recommendation moved more toward it while judging its advice less correct (r=-.23), and exploratory analyses suggest lower expertise is associated with greater susceptibility.
- Recognizing personalization did not create resistance, implying awareness-based safeguards alone are insufficient.
Strengths
Clean isolation of two personalization mechanisms
The 2×2 design (Section 3.2, Fig. 3) independently varies style and context layers in the system prompt while holding the assigned ranking, evidence, and six-stage argument sequence constant; counterbalancing rotates all 3!=6 rankings ten times per cell (Section 3.2).
Rigorous manipulation validation
Section 4.1.4 verifies on all 240 transcripts via two LLM judges (GLM-5.2 primary, MiniMax-M3 replication) that evidence coverage is equivalent across cells (TOST, all p≤.008/.024) and the assigned ranking was advocated in 100% of sessions.
Decisions cleanly separated from perceptions
Transparent statistics and deviation reporting
Honest framing of null and exploratory results
Weaknesses
Possible confound inside the context manipulation
Hypothetical, single-session task with no stakes
Weak and single-source moderator measurement
LLM-judged transcript fidelity
Style banding is data-driven and model-specific
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper tests a precisely scoped question: does personalizing *how* an AI communicates versus *what* it emphasizes change users' decisions, independently of the advice itself? Using a preregistered, IRB-approved 2×2 between-subjects experiment (N=240; Section 3.2), participants ranked three anonymized real pharmaceutical stocks, discussed them with a GPT-5.4 agent advocating a counterbalanced ranking (all six orderings rotated, ten assignments per cell), then reranked. The claims are well calibrated to the design: the abstract asserts reliable decision effects only for context-based personalization, nulls on evaluative perceptions, and explicitly labels expertise findings exploratory. The title's Michael Jackson allusion signals the thesis — influence without an evaluative crescendo — and the discussion (Section 5.1.1) earns it with the Genç et al. (negative-affect) and Cheng et al. (sycophancy) comparison, positioning a third influence signature: behavior change with neither evaluative reward nor penalty.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 3 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the main finding of the 'Annie, Are You Okay?' paper?
In a preregistered experiment with 240 participants ranking stocks, AI advice framed around each person's demographics and values ('context-based personalization') made people more likely to change their rankings and move toward the AI's recommendation (OR≈2), without changing how much they trusted or liked the AI. Matching the AI's communication style to the user's personality had no reliable effect on decisions.
What is the difference between style-based and context-based personalization?
Style-based personalization changes how the AI communicates (tone, length, hedging matched to Big Five personality traits). Context-based personalization changes how arguments are framed and emphasized using facts about the user, such as occupation, income, location, and personal values. In this study, only the context type changed participants' decisions; both types were noticed by participants.
Did participants realize the AI was personalizing to them, and did that help them resist?
Yes — participants detected both forms of personalization, and context personalization produced the study's largest manipulation-check effect (d≈1.46). They even reported feeling more influenced. But recognition did not reduce the behavioral shift: rankings still moved toward the AI's recommendation, and evaluations of the AI stayed flat.
Does this study show personalized AI is dangerous for real investing?
Not directly. The task was a hypothetical ten-minute ranking of three anonymized stocks with no money at stake, using one model and one prompt architecture. The authors themselves list consequential, incentive-compatible and longitudinal studies as needed future work; the findings are a controlled warning signal, not evidence about real-market losses.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Several middle sections (parts of Sections 3.1, 3.6–4.1, 4.5–5.1 and Appendices A/C) were truncated in the extracted text, so the exact power analysis, some balance checks, and expertise-analysis details were reviewed only via cross-references.
- Figures are available only as textual descriptions; distribution shapes (Figs. 8, 9, 14) could not be inspected directly.
- Statements about GPT-5.4, the 2026 CHI references, and the June 2026 ChatGPT finance feature reflect the text as provided and were not externally verified.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.