Emotion Experience, Expression, and Perception: Emotion Analysis on Multimodal Social Media Posts
Christopher Bagdon, Carina Silberer, Roman Klinger
cs.CL
Quick verdict
Introduces Mult2EMo, a rigorous 7,200-post author-vs-reader emotion dataset; findings credible but models show modality collapse and data is request-gated.
Summary
The paper introduces Mult2EMo, a dataset of 7,200 multimodal social media posts annotated by their authors (1,101 Prolific participants, 1,200 posts per emotion) for the emotion experienced at a triggering event, 21 cognitive appraisal dimensions, the emotion expressed in the post, image content/type, and text-image relations. A 1,440-post subset (240 per emotion) additionally carries reconstructions from three readers each, enabling direct comparison of experience, expression, and perception. Comparing readers and models (fine-tuned CLIP variants; zero-shot Qwen3-VL-8B) against author labels, reconstruction proves possible but hard: individual readers reach F1 .47 and majority vote .48, versus .51 for multimodal CLIP; a paired cluster bootstrap shows models beat individual readers significantly (p<10^-4) but not majority vote (p=.22). Appraisal RMSE favors models, though readers sit near the mean baseline. Key findings: experienced-expressed emotion mismatches (~10% of posts) cost .21-.25 F1; reader-author event-description similarity strongly predicts accuracy (F1 .34 to .61 across quartiles); models fed curated event descriptions reach .60 F1 versus .51 on post text; and T+I models with randomly shuffled images match real-image performance, exposing modality collapse. Limitations: UK/IE native-English authors only, request-gated data release, naive concatenation fusion, single MLLM with one prompt set.
Review complete
Assessment at a glance
4
risk flags identified
● high 0
● medium 2
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- A multimodal post alone permits only partial reconstruction of authors' experienced and expressed emotions and appraisals, by both readers (macro F1 .47-.48) and fine-tuned models (.51).
- Understanding the triggering event is crucial: reader-author event-description similarity strongly predicts reconstruction accuracy (F1 .34 to .61 across quartiles, Table 4), and models given event descriptions reach .60 F1 versus .51 on post text (Table 5).
- Images carry emotion-relevant information (image-only F1 .30, above chance) but simple multimodal models fail to fuse modalities: random-image T+I matches real-image T+I (.51).
- When authors' experienced and expressed emotions mismatch (~10% of posts), reader and model F1 drop by .25 and .21 respectively (Table 3).
- Models significantly outperform individual readers on emotion (paired cluster bootstrap, p<10^-4) but not reader majority vote (p=.22, Table 2).
Strengths
Unique triadic annotation design
Mult2EMo is the first dataset to capture experience, expression, and perception jointly on the same multimodal posts (7,200 posts, 1,101 authors, 21 appraisal dimensions; Sec. 3), going beyond perception-only corpora (Sec. 2.2) and meme-restricted multimodal work (Sec. 2.3).
Unusually careful controls
A shuffled-image ablation isolates modality use (Table 1: T+I .51 with real vs. .51 with random images), an anonymization robustness rerun quantifies privacy costs (Table 8: .51 vs. .50 F1), and the MLLM is run with order-randomized emotion/appraisal lists three times per post (Sec. 4.1.3).
Honest, appropriately powered significance testing
Theoretically grounded error analysis
Transparent limitations and reproducibility-minded reporting
Weaknesses
Multimodal promise unfulfilled by the models
Reader comparison carries acknowledged confounds
Appraisal conclusions rest partly on acknowledged artifacts
Gated data release, no code
Thin LLM comparison and target-emotion sampling
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper models emotion communication on social media as a three-stage process — author experience, author expression, reader perception — and asks three research questions: whether a multimodal post suffices to reconstruct the author's emotions and appraisals (RQ1), how event-post relatedness affects reconstruction (RQ2), and how image type and text-image relation matter (RQ3). To answer, it releases Mult2EMo: 7,200 posts (1,200 per emotion across anger, disgust, fear, joy, sadness, surprise) from 1,101 UK/IE Prolific authors who label event emotions and intensities, 21 appraisal dimensions (Scherer's Component Process Model), post emotions, image content/type, and five text-image relation items; 1,440 posts (240/emotion, platform-balanced) additionally carry reconstructions from 3 readers each (4,320 annotations, 959 readers). Headline claims: reconstruction is possible but hard; event understanding is the bottleneck; images carry signal that current models fail to use.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 4 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Mult2EMo dataset?
Mult2EMo is a dataset of 7,200 multimodal (text plus image) social media posts in which the original authors annotated the emotion they experienced at the triggering event, 21 cognitive appraisal dimensions, the emotion they expressed in the post, image content and type, and text-image relations. A 1,440-post subset also has annotations from three independent readers who tried to reconstruct the authors' answers, enabling study of emotion experience, expression, and perception together.
Can readers or AI models correctly infer what emotion a social media author intended?
Only partially. In this study, individual human readers reached a macro F1 of about .47 and a three-reader majority vote .48 against author labels, while a fine-tuned multimodal CLIP model reached .51. Joy and sadness were easiest to reconstruct; disgust and surprise were hardest. Models significantly beat individual readers but not the majority vote.
Do images actually help emotion recognition models on social media posts?
Images alone carry emotion signal (image-only models scored F1 .30, above chance), but adding the image to the text did not help in this paper's models: models fed randomly shuffled images alongside text scored the same (.51) as those fed the correct images. This modality-collapse finding points to the need for better multimodal fusion methods.
Is the Mult2EMo dataset publicly available?
Not as a public download. The authors released an anonymized version (faces blurred, names removed) for research purposes upon request via the University of Bamberg NLP group project page, citing participant privacy and third-party content concerns.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 and 5-15 are not analyzable from the extracted text; assessment relies on captions, tables, and surrounding prose.
- Table 1's column layout was reconstructed from flattened text; column-to-system assignments were cross-checked against the prose but the original PDF should be consulted for exact alignment.
- Dataset statistics could not be verified externally because the data is request-gated.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.