Chiaroscuro for Emotions: A Contrastive Emotion Benchmark Grounded in Appraisal Theory
Divyesh Bommana, Mohammad Saim, Tianyu Jiang
cs.CL
Quick verdict
Proposes CHIARO, a 1,000-sentence two-agent contrastive emotion benchmark; well-constructed and novel task, but synthetic gold, no significance tests.
Summary
The paper introduces CHIARO, a 1,000-sentence English benchmark for contrastive emotion inference: each sentence describes one shared causal trigger that produces a positive emotion in one agent and a negative emotion in another, labeled from a ten-class taxonomy (five positive, five negative) derived from GoEmotions and motivated by appraisal theory. Scenes were generated with GPT-5.2 from r/AmItheAsshole posts under a 71-term affect-word ban, six programmatic validators, and mandatory per-emotion situational triggers, then labeled by two annotators (Cohen's kappa = 0.827; 1,050 generated, 1,017 survived, 1,000 released). Seven frontier LLMs, evaluated with a joint two-agent multiple-choice prompt, reach 59.9-67.3 macro-F1 (GPT-5.5 best, mean 65.3), well below the 93.0 macro-F1 that annotator agreement implies against adjudicated gold; four off-the-shelf emotion classifiers score 11.8-29.0, near chance. Errors concentrate on positive emotions (joy F1 31.4 for GPT-5.5), and API models score higher on non-physical than physical scenes, matching the human agreement pattern. Fine-tuning RoBERTa-large on CHIARO alone reaches 69.5% accuracy on a held-out split, and a union with a size-matched GoEmotions slice beats both single sources in-distribution (73.5%) and on six of ten external benchmarks. Limitations include fully LLM-generated scenes from a single generator, a two-annotator gold, no significance testing, and restriction to opposite-valence pairs.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- CHIARO is the first dataset requiring joint attribution of opposite-valence emotions to two named agents from a single shared causal trigger.
- The best frontier LLM (GPT-5.5, 67.3 macro-F1) trails annotator agreement (93.0 macro-F1, kappa 0.827) by roughly 26 points, with the gap concentrated in positive-emotion classes.
- Off-the-shelf emotion classifiers trained on single-agent corpora transfer to contrastive agent-attributed inference at near chance (11.8-29.0 macro-F1).
- Physical-trigger scenes are harder than non-physical ones for both LLMs and human annotators, contradicting the concreteness intuition.
- CHIARO combined with a size-matched GoEmotions slice yields a classifier that beats either source alone on CHIARO (73.5% accuracy) and on six of ten external emotion benchmarks.
Strengths
Genuinely novel task framing
Sections 2-3 make a defensible case that no existing corpus requires joint two-agent, opposed-valence attribution from one shared trigger; the six contrastive scenario types (Appendix C: zero-sum, spillover, asymmetric information, unintended consequence, competing preferences, success vs. failure) are each anchored to cited social-science theory (Festinger, Knobe, Akerlof, Merton, Deutsch, Weiner).
Disciplined construction pipeline
A 71-term affect-word ban (Appendix D), six programmatic validators with bounded repair loops (Appendix E), span-consistency checks enforcing the single-cause rule, and per-emotion mandatory triggers (Table 8) produce a controlled corpus; balanced sampling lifted per-class frequency from a skewed start (disgust 20.0%, sadness 3.4%) to within roughly ±1.5 points of uniform (Table 9).
Transparent human reference point
Instructive error and split analysis
Dual utility with a controlled transfer study
Weaknesses
Fully synthetic gold from a single generator
Gold polarity is handed to the evaluated system
No inferential statistics anywhere
Mandatory triggers invite untested shortcuts
Inflated human ceiling and metric switching
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper (arXiv 2609.03394, cs.CL) introduces CHIARO, a 1,000-sentence English benchmark for contrastive emotion inference: each scene describes a single causal trigger producing a positive emotion in one agent and a negative emotion in another, labeled from a ten-class taxonomy (joy, pride, relief, gratitude, excitement / anger, sadness, fear, disgust, embarrassment) derived from GoEmotions (Section 3.1) and motivated by appraisal theory. Three central claims: (1) no existing dataset requires joint two-agent opposed-valence attribution from a shared trigger; (2) frontier LLMs fall about 26 macro-F1 points below human agreement (best GPT-5.5 at 67.3 vs 93.0) while off-the-shelf emotion classifiers transfer near chance (11.8-29.0); (3) CHIARO is a complementary training signal — RoBERTa-large trained on CHIARO plus GoEmotions beats either source alone on CHIARO (73.5% accuracy) and on 6/10 external benchmarks.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the CHIARO benchmark?
CHIARO is a 1,000-sentence English dataset for contrastive emotion inference: each sentence describes one shared event that makes one person feel a positive emotion (joy, pride, relief, gratitude, or excitement) and another person feel a negative emotion (anger, sadness, fear, disgust, or embarrassment). Both feelings must be inferred from situational context because explicit emotion words are banned. It was built by University of Cincinnati researchers and released at github.com/cincynlp/Chiaro.
How well do LLMs perform on CHIARO?
In the paper's evaluation of seven LLMs with a joint multiple-choice prompt, the best model (GPT-5.5) reached 67.3 macro-F1 and the seven-model mean was 65.3, versus 93.0 macro-F1 for human annotator agreement (Cohen's kappa 0.827). Errors concentrate on positive emotions — joy F1 was only 31.4 for GPT-5.5. Off-the-shelf emotion classifiers scored 11.8-29.0, near chance.
Why is the dataset called Chiaroscuro (CHIARO)?
The name refers to chiaroscuro, the painting technique of rendering strong light-dark contrast on one canvas (associated with Caravaggio, Rembrandt, and Vermeer). It is a metaphor for two opposing emotions — one positive, one negative — arising from the same shared event.
Can CHIARO be used to train emotion classifiers?
Yes. The paper fine-tunes RoBERTa-large on CHIARO alone and gets 69.5% accuracy on a held-out split (versus 44.0% for RoBERTa-base on the same data). Combining CHIARO with a size-matched GoEmotions slice gives 73.5% and beats either source alone on six of ten external emotion benchmarks, suggesting the two datasets capture complementary signals.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 2's per-benchmark transfer accuracies exist only as an image; the extracted text gives selected values in prose (69.5%, 44.0%, 73.5%), so split-level transfer claims could not be checked against the figure itself.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.