MMDS-Bench: Benchmarking Multimodal Large Language Models on Dynamic Stance in Social Media Interactions
Yuzhe Ding, Kang He, Li Zheng, Shengwu Zheng, Teng Shi, Fei Li, Chong Teng, Donghong Ji
cs.CL
Quick verdict
MMDS-Bench is a well-validated benchmark for multimodal parent-reply stance, but extreme label imbalance and zero-shot-only evaluation temper its conclusions.
Summary
The paper introduces MMDS-Bench, a benchmark for multimodal dynamic stance classification: judging how a social media reply responds to its direct parent when both messages contain images. It comprises 3,482 parent-reply pairs collected from X via topical keywords, labeled with a seven-label taxonomy (Agree, Disagree, Elaborates, Query, Neutral, Unrelated, NA); an 800-instance diagnostic subset requiring structured reasoning over parent understanding, reply understanding, and stance inference; and five human-annotated binary challenge factors. Twelve closed- and open-source MLLMs are evaluated zero-shot with final-label metrics plus a reference-grounded LLM-judge protocol (PUS/RUS/SRS) validated against human annotators (LLM-human Spearman 0.93 for stance reasoning). Gemini 2.5 Pro leads Task 1 (79.93% accuracy, 42.22 Macro-F1) and Claude Sonnet 4.6 leads Task 2 Macro-F1 (51.99); Kimi-K2.5 is the best open model. The central finding is that stance-relation reasoning is the bottleneck: PUS/RUS exceed SRS for all models, and Task 1 accuracy falls monotonically from 76.35% to 26.44% as challenge factors accumulate. Models also overpredict Disagree (39.6% of gold Agree, 58.7% of gold NA). Main limitations: extreme label imbalance (Query n=5, Neutral n=14 of 3,482) destabilizes tail-label metrics; evaluation is zero-shot prompting only, without fine-tuned baselines; data are English-only, image-mandatory X interactions; and possible judge-family bias is unexamined.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MMDS-Bench is the first benchmark for multimodal dynamic stance classification, with 3,482 image-bearing parent-reply pairs and a seven-label taxonomy adapted from textual dynamic stance work.
- Across 12 MLLMs, parent and reply understanding (PUS/RUS) consistently exceed stance-relation reasoning (SRS), identifying relational inference rather than message comprehension as the bottleneck.
- Model accuracy degrades monotonically with the number of co-occurring challenge factors, from 76.35% (zero factors) to 26.44% (five) on Task 1.
- Models systematically overpredict Disagree, including 39.6% of gold-Agree and 58.7% of gold-NA instances.
- The reference-grounded LLM-judge protocol approximates human scoring, with LLM-human Spearman correlation of 0.93 for stance reasoning on a 600-case validation set.
Strengths
Clear task gap with diagnostic payoff
Formulates multimodal parent-reply stance beyond target-oriented work (Sec 1-2); Task 2's PUS/RUS/SRS decomposition localizes failure: all 12 models show message understanding far above relation inference (Gemini 4.92/4.89 vs SRS 4.33, Table 2), and incorrect predictions keep PUS 4.51/RUS 4.24 but drop to SRS 2.20 (Table 3).
Judge protocol validated against humans
Three-judge panel with inter-judge Spearman 0.92 on SRS (Table 6) and human validation over 600 cases from 50 instances showing LLM-human correlation 0.93 for SRS and 0.77 for PUS/RUS (Table 7) — more scrutiny than most LLM-judge evaluations provide.
Transparent annotation QC
Multi-angle difficulty analysis
Actionable error phenomenology
Weaknesses
Tail classes are statistically untestable
Zero-shot prompting only
Unexamined judge-family bias
Narrow, skewed slice of interactions
Lowest-agreement factor drives headline error gaps
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MMDS-Bench (arXiv 2608.30903) defines multimodal dynamic stance classification: given a parent post and a reply that each contain at least one image (text optional), label how the reply responds to the parent using the seven labels of Figueras et al. (2023): Agree, Disagree, Elaborates, Query, Neutral, Unrelated, NA. The paper claims three contributions — the first multimodal parent-reply stance benchmark (3,482 instances plus an 800-instance diagnostic subset), a fine-grained evaluation framework (five binary challenge factors and structured reasoning outputs), and a reference-grounded LLM-judge protocol validated against humans — and two findings: relational inference, not message comprehension, is the failure point, and difficulty compounds across factors (Table 4).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is dynamic stance classification and how does MMDS-Bench differ from ordinary stance detection?
Ordinary (target-oriented) stance detection judges whether a single post supports or opposes a fixed topic, such as vaccination. Dynamic stance instead judges how a reply responds to its direct parent message in a thread — agreeing, disagreeing, elaborating, asking, staying neutral, being unrelated, or being unclassifiable. MMDS-Bench extends this to multimodal cases where both the parent and the reply contain images (memes, screenshots, reaction images), not just text.
Which models perform best on MMDS-Bench?
Per the paper's Table 2: Gemini 2.5 Pro is best on Task 1 with 79.93% accuracy (42.22 Macro-F1) and best Task 2 accuracy (72.25%); Claude Sonnet 4.6 achieves the best Task 2 Macro-F1 of 51.99; and Kimi-K2.5 is the strongest open-source model with 65.88% Task 1 accuracy. All 12 evaluated multimodal LLMs score far better on Agree/Disagree than on the rare labels Neutral, NA, and Query.
Is the MMDS-Bench dataset and code publicly available?
The paper links a GitHub repository (github.com/whu-yzding/MMDS-Bench) and publishes all evaluation prompts in its appendix. The data consist of X (Twitter) posts and images collected via the X API, so actual availability may be affected by platform licensing and takedown constraints that the paper does not discuss in detail.
Why are the Macro-F1 scores so low when accuracy exceeds 70% for the best models?
Because the dataset is extremely imbalanced: Agree and Disagree together account for about 87% of instances, while Query (5 instances), Neutral (14), Unrelated (45), and NA (47) are nearly absent. Models score near zero F1 on Neutral and NA, which drags Macro-F1 (the average over all seven labels) down sharply even when accuracy on the dominant labels is high.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures 1-3 and the prompt/case-study screenshots (Figures 4-8) are embedded images; their content was assessed only via captions and numbers repeated in prose, not from the rendered graphics.
- The extracted text ends cleanly at Table 18 with no mid-sentence cut; any content that would follow page 22 is not visible in this extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.