From Memory to Behavior: A Behavior-Aware Role-Playing Framework for Social Media Influencers
Ji-Lun Peng, Yi-Zhen Zhang, Chun-Nan Chou, Yun-Nung Chen
cs.CL · cs.AI
Quick verdict
SIBPersona adds situation-state-behavior personas to ICL role-playing; gains are strong on its own LLM protocol, but human eval shows parity with vanilla RAG.
Summary
The paper proposes SIBPersona, an in-context learning framework for role-playing real social media influencers. Motivated by the Cognitive-Affective Processing System, it adds a third persona dimension beyond memory and speaking style: the reaction process, extracted from history as Situation–Internal state–Behavior triplets mapping situations to internal states and behavioral strategies. A two-stage pipeline (relevance filtering, then extraction with evidence pairs) builds three dimensional corpora per influencer; the most relevant documents are retrieved as demonstrations at inference. The authors also contribute a dataset of six financial influencers from Taiwan's CMoney Forum and a reference-augmented penalty-based LLM evaluation protocol giving the evaluator representative persona documents. Across four persona-extracting LLMs on Influencer A and five further influencers, SIBPersona significantly outperforms Naive, CoT, IMPersona, and Vanilla RAG on speaking-style and reaction-process scores (e.g., average T 59.31 vs 54.48 for Vanilla RAG on Influencer A), while memory scores saturate near 96. Ablations show removing the reaction process hurts significantly; removing the internal state does not. On CharacterEval and RoleAgentBench, SIBPersona rivals a fine-tuned CoSER model. The protocol correlates moderately with human judgment (Pearson 0.561; only 0.289 for reaction process), and human evaluation finds SIBPersona roughly equal to Vanilla RAG (3.68 vs 3.67). Limitations: non-released private data, six influencers, and reliance on the authors' own protocol.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Adding a situation-dependent reaction-process dimension (SIB triplets) to data-variant persona significantly improves ICL-based impersonation on speaking-style and reaction-process scores versus Naive, CoT, IMPersona, and Vanilla RAG across six influencers and four extractor LLMs.
- A reference-augmented penalty-based LLM evaluation protocol achieves moderate correlation with human judgment (Pearson 0.561, Spearman 0.530) and far exceeds traditional metrics for evaluating role-playing of obscure real individuals.
- Ablations show the behavioral-strategy component of the reaction process, not the internal-state component, drives the performance gains.
- SIBPersona generalizes to fictional-character benchmarks, significantly beating IMPersona and rivaling fine-tuned CoSER-Llama-3.1-8B on CharacterEval.
Strengths
Psychologically grounded, interpretable behavioral dimension with honest ablation
SIB triplets derived from CAPS (§3.1) are shown to drive gains: removing reaction process significantly drops T (61.60→57.82) and RP (75.96→73.36) on six influencers with DeepSeek-V3.2 (Table 4), while removing internal state does not — a clean, mechanism-informative decomposition replicated on CharacterEval (Table 8).
Unusually thorough contamination and reference-configuration analysis
Appendix F quantifies evaluator-reference overlap (0.35–4.68% per dimension on Influencer A), re-runs excluding overlaps (Table 18, rankings unchanged), and rebuilds references from the independent demonstration source (Tables 19/21) — conclusions survive all three checks, a rigor rare in LLM-eval papers.
Evaluation protocol validated against humans and traditional metrics
Robustness across models, targets, and benchmarks
Weaknesses
Decisive evidence is self-referential; human study shows parity with the strongest baseline
Weakest human correlation sits on the core dimension
Small, closed dataset limits external validity
Memory dimension is non-discriminative in the social media setting
Token-budget confound versus Vanilla RAG
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper targets role-playing agents (RPAs) for real, non-famous individuals — six financial influencers on Taiwan's CMoney Forum — in the single-turn task of replying to follower comments. It claims three contributions: (1) SIBPersona, an ICL framework adding a reaction-process dimension (Situation–Internal state–Behavior triplets, §3.1) to the usual memory and speaking-style persona dimensions, motivated by the Cognitive-Affective Processing System; (2) a reference-augmented, penalty-based LLM evaluation protocol for obscure individuals (§4); (3) a new comment-reply dataset (Table 1) plus cross-benchmark validation on CharacterEval and RoleAgentBench. Central claims: significant gains over Naive, CoT, IMPersona, and Vanilla RAG on speaking-style (T) and reaction-process (RP) scores; moderate human correlation for the protocol (r=0.561); and gains driven by behavior B rather than internal state I.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SIBPersona?
SIBPersona is an in-context learning framework that lets an LLM impersonate a real social media influencer by retrieving three kinds of persona information from the person's history: factual memory, speaking style, and a new reaction-process dimension encoded as situation–internal state–behavior (SIB) triplets describing how the person tends to react in given situations.
Does SIBPersona beat existing role-playing methods?
On the authors' automated penalty-based protocol it significantly beats Naive, CoT, IMPersona, and Vanilla RAG on speaking-style and reaction-process scores across six influencers and four models (e.g., average score 59.31 vs 54.48 against Vanilla RAG on Influencer A). However, in their small human study it performed roughly equal to Vanilla RAG, so the advantage is mainly established by their own automatic evaluation.
Is the influencer dataset from this paper publicly available?
No. The data comes from Taiwan's CMoney Forum under company authorization, and the paper states no raw influencer or follower content will be released. The code is promised at github.com/MiuLab/SIBPersona, and public benchmarks (CharacterEval, RoleAgentBench) were used for the fictional-character experiments.
How reliable is the paper's automatic evaluation?
It was validated against three human annotators on 30 samples from one influencer, achieving a moderate overall correlation (Pearson 0.561), far better than BLEU or ROUGE. However, alignment is uneven: strongest for speaking style (0.576) and weakest for the reaction-process dimension (0.289), which is exactly where the paper claims its main gains.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Extraction omitted several passages (part of §3.4 retrieval mechanics, portions of Appendix D.1 and G); judgments rest on the remaining text.
- Figures (t-SNE projections, correlation matrix, qualitative example images) are referenced but not visually analyzable from the extracted text.
- Dense appendix tables (22–25) were OCR-extracted; individual cell values were not re-verified against the PDF.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.