SocialReasonBench: A Video-QA Benchmark for Social Reasoning with Counterfactual Narrative Videos
Zheyu Huang, Zijing Shi, Haozhe Luo, Huadong Tang, Mingyu Liu, Meng Fang, Ling Chen
cs.CL · cs.CV
Quick verdict
Game-grounded video-QA benchmark (532 items) for social reasoning; careful grounding and stats, but single-game scope and Gemini overlap limit generality.
Summary
SocialReasonBench is a 532-instance video multiple-choice QA benchmark for social reasoning built from branching gameplay of Detroit: Become Human. Because player decisions create alternative storylines that the game itself records, counterfactual and causal-antecedent answers can be checked against flowcharts, scripts, and visible UI state rather than post-hoc human annotation. A three-agent pipeline (Director for clip localization, Tracker for game-signal grounding, Generator for QA synthesis with entity anonymization and five rubric-defined distractor traps) produced 730 candidates; human auditing retained 532 across seven reasoning dimensions and four interaction types, with 56.4% of labels directly verifiable. Evaluating eight LMMs zero-shot, the authors find competent basic social understanding (e.g., GPT-4o reaches 85.1% on intent recognition) but large drops on counterfactual reasoning (22–62%) and causal antecedent (27–71%); a five-annotator author baseline reaches 85.7–93.2%. The perception-versus-causal gap survives Holm-corrected Fisher tests for six of eight models and persists when both Gemini systems are excluded. Audio ablation on Gemini 3 Pro alone costs roughly 6% overall accuracy. Limitations include the single-game, English source; heterogeneous input regimes across models (native video vs 32 keyframes); moral-dilemma labels defined as majority player choice on only 21 instances; and Gemini-family curation models overlapping two evaluated systems.
Review complete
Assessment at a glance
6
risk flags identified
● high 0
● medium 5
● low 1
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Branching narrative gameplay enables counterfactual and causal-antecedent video-QA whose answers are objectively verifiable against the game's own flowcharts, scripts and recorded branch replays, replacing costly post-hoc human annotation.
- A three-agent curation pipeline (Director, Tracker, Generator) with human audit scales to 532 verified instances across seven theory-grounded reasoning dimensions and four interaction types, with 56.4% of labels directly checkable.
- Contemporary LMMs handle basic social understanding (GPT-4o: 85.14% intent recognition) but degrade sharply on counterfactual reasoning (22–62%) and causal antecedent (27.4–70.5%), while a five-person human baseline reaches 85.7–93.2%.
- The perception-versus-causal gap is statistically robust — significant for six of eight models after Holm correction under one-sided Fisher tests, and persisting when both Gemini-family systems (also used for curation) are excluded.
- Model errors are diagnostically structured: removing audio costs Gemini 3 Pro about 6% accuracy, and incorrect answers cluster into interpretable trap types (knowledge/visual traps for strong closed models, logic traps for the small open-weight model).
Strengths
Game-state ground truth replaces post-hoc annotation
Labels are grounded in flowcharts, scripts, recorded branch replays and UI signals; App. J introduces a four-tier verifiability taxonomy (Tier A 19.4%, B 27.8%, AB 9.2%, script-derived 43.6%) with 56.4% directly checkable, and human majority vote matches the reference label on 49/50 overlap items (App. I).
Counterfactuals are actually realized, not imagined
Branch replays in 100%-completion walkthroughs mean the alternative outcome was physically recorded (App. L branch-replay handling; §4.6 case study), a genuine advance over human-authored alternatives such as ACQUIRED, as argued in §2.
Diagnostic error machinery with rubric-constrained distractors
Unusually careful statistics for a benchmark paper
Transparent handling of the generator-family conflict
Weaknesses
Single-game, single-culture source limits construct coverage
Cross-model rows are confounded by input regime
43.6% of labels are script-derived with no per-tier accuracy reported
Moral-dilemma ground truth is player popularity
Key ablations are single-model and the visual contamination channel stays open
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
SocialReasonBench (arXiv:2608.30716) is a 532-instance video multiple-choice QA benchmark for social reasoning, constructed from branching gameplay of *Detroit: Become Human*. The central claim is methodological: because player decisions generate alternative storylines that the game itself records (flowcharts, scripts, branch replays, UI state variables), counterfactual and causal-antecedent questions can be given objectively verifiable answers instead of post-hoc human annotations. Seven reasoning dimensions (Table 2: intent recognition 74, behavior prediction 78, emotional empathy 120, relationship dynamics 94, moral dilemma 21, counterfactual 50, causal antecedent 95) cross four interaction types. Empirical claims: (i) eight zero-shot LMMs handle basic social understanding but degrade on counterfactual reasoning (22–62%) and causal antecedent (27.4–70.5%); (ii) audio matters (~6% drop when muted, Gemini 3 Pro only); (iii) errors are diagnosable via a five-type trap taxonomy.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is SocialReasonBench?
SocialReasonBench is a 532-question video multiple-choice benchmark for evaluating social reasoning in large multimodal models, built from branching gameplay videos of the game Detroit: Become Human. Questions span seven abilities (intent recognition, behavior prediction, emotional empathy, relationship dynamics, moral dilemma, counterfactual reasoning, causal antecedent), and correct answers are verified against the game's own flowcharts, scripts and on-screen state signals rather than post-hoc human annotation.
How well do large multimodal models perform on SocialReasonBench?
In zero-shot tests of eight LMMs, models handle basic social understanding reasonably — GPT-4o reaches 85.14% on intent recognition — but accuracy drops sharply on counterfactual reasoning (22–62% across models) and causal antecedent (27–71%). A five-person human baseline scores 85.7–93.2%, and the gap between perception-style and causal dimensions is statistically significant for six of eight models after multiple-comparison correction, even excluding the Gemini systems that helped build the benchmark.
Why build a social-reasoning benchmark from a video game?
Branching narratives make counterfactuals real: alternative player decisions were actually played out and recorded, so a question like 'what would have happened if the negotiator had refused' has a checkable answer, unlike linear film clips where only one trajectory exists. The game's flowcharts and UI state variables also provide objective ground truth without post-hoc annotation.
Is SocialReasonBench publicly available?
The project page is socialreasonbench.github.io with code and prompts released; annotations are published under CC BY-NC-SA 4.0. However, the video clips derive from copyrighted gameplay, so access to the video portion is governed by dataset Terms of Use rather than free redistribution.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Data behind Figures 1, 4 (ablation bars) and 5 (per-trap TFR distributions) exists only as images, so per-trap TFR values cannot be verified from the extracted text.
- The Table 3 header row is garbled by PDF extraction; column attribution was cross-checked against in-text numbers (e.g., GPT-4o 85.14/50.00/28.57) and appears reliable.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.