Modality Discrepancy Transformer for Ambivalence and Hesitancy Recognition
Shiyu Luo, Yu Wang, Jiawen Huang, Zhaoxiang Xiao, Chenxi Huang, Qi Zhang, Bin Liu
cs.CL · cs.CV
Quick verdict
Claims 0.7408 Macro F1 on ABAW A/H via 9-token discrepancy fusion; results look real, but the paper's own ablations suggest LoRA and text blending, not discrepancy, drive gains.
Summary
The paper introduces the Modality Discrepancy Transformer (MDT) for recognising ambivalence and hesitancy (A/H) in clinical videos, extending the conflict-aware fusion architecture of Bekhouche et al. It enriches a 6-token input to nine tokens — three modality embeddings plus three absolute-difference and three Hadamard-product discrepancy tokens — processed by a 2-layer Transformer, with FiLM text-conditioned modulation, LoRA fine-tuning of frozen VideoMAE, HuBERT and RoBERTa encoders, and a text-guided late fusion blending a text-only head (alpha=0.6). On the BAH dataset (778 training videos), MDT reports 0.7408 Macro F1 on the labelled test split and 0.7368 on the private leaderboard, beating the strongest published baseline (0.634) by more than 10 points. Notably, the ablations show that removing all discrepancy tokens yields the best ablation score (0.7325), slightly above the 6-token (0.7219) and 9-token (0.7322) variants, indicating the core architectural contribution is modest; the gains largely stem from LoRA and text-fusion strategies. Main limitations are single-run results with no variance or significance testing, and ambiguously described threshold selection that risks test-set contamination.
Review complete
Assessment at a glance
5
risk flags identified
● high 0
● medium 3
● low 2
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MDT's 9-token discrepancy representation (absolute differences + Hadamard products), FiLM modulation and LoRA fine-tuning achieve 0.7408 Macro F1 on the BAH labelled test split and 0.7368 on the ABAW11 private leaderboard, outperforming the strongest published baseline (0.634) by over 10 points.
- Hadamard-product discrepancy features alone yield no Macro F1 gain (0.7322 vs 0.7325 without any discrepancy features) but improve negative-class calibration (F1-NoAH 0.6700 vs 0.6457).
- LoRA fine-tuning is the single most impactful component (+1.47 Macro F1 over layer unfreezing, Table 5), indicating regularisation — not capacity — is the limiting factor on 778 training examples.
- Text is the only modality that independently discriminates the negative class: video-only and audio-only models collapse to all-positive (F1-NoAH 0.0000, Table 4), while text alone reaches 0.7059–0.7185.
Strengths
Strong, doubly reported headline results
0.7408 Macro F1 on the labelled test split (Table 2) and 0.7368 on the ABAW11 private leaderboard, with full training in under 20 minutes on a single RTX 4090 — practical and consistent across the two test sets.
Unusually honest ablations
Table 3 reports that removing all discrepancy features (0.7325) beats the CA-AH baseline (0.7219) and roughly ties the 9-token variant (0.7322), and the Discussion (§4.4) engages with this directly rather than hiding it.
Clear identification of the regularisation bottleneck
Fair per-class analysis
Weaknesses
Core architectural claim weakly supported by own ablations
No variance, no significance testing
Possible test-set contamination via threshold sweep
Shifting baselines across tables
No code release
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
This is a challenge-system method paper for the 3rd ABAW Ambivalence/Hesitancy (A/H) track. The authors propose the Modality Discrepancy Transformer (MDT), extending the conflict-aware CA-AH fusion of Bekhouche et al. [3]: three frozen pretrained encoders (VideoMAE-Base, HuBERT-Base, RoBERTa-GoEmotions) yield pooled modality embeddings; pairwise absolute-difference tokens are augmented with projected Hadamard-product tokens to form a 9-token input (contributions §1); a 2-layer Transformer classifies; FiLM modulation conditions video/audio on text (§3.3); LoRA (r=8) adapts the encoders (§3.4); a text-guided late fusion blends a text-only head at α=0.6. Headline claims: 0.7408 Macro F1 on the labelled BAH test split and 0.7368 on the private leaderboard (§4.2, Table 2), beating the best published baseline (0.634) by 10.7 points and CA-AH (0.715) by 2.6 points.
The paper's own Table 3 complicates its story: removing *all* discrepancy features yields 0.7325 — above the 6-token CA-AH config (0.7219) and marginally above the 9-token variant (0.7322). The authors are candid about this (§4.4 'Why does removing discrepancy help?'), which is commendable, but it means the headline architectural contribution contributes ~0 to the aggregate metric; its value is confined to per-class calibration (F1-NoAH 0.6700 vs 0.6457). Meanwhile text-only inference at α=1.0 already reaches 0.7185, so the full system's margin over a strong text-only blend is only ~2.2 points.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the Modality Discrepancy Transformer (MDT)?
It is a multimodal model for recognising ambivalence and hesitancy (A/H) in clinical videos. It builds three embeddings from face video, audio and the speech transcript, computes pairwise absolute-difference and Hadamard-product 'discrepancy' tokens between them (nine tokens in total), and runs them through a small Transformer classifier. It adds text-conditioned FiLM modulation, LoRA fine-tuning of frozen pretrained encoders, and a late fusion that blends in a text-only prediction (α=0.6).
What results does MDT achieve on the ABAW A/H (BAH) benchmark?
The paper reports 0.7408 Macro F1 on the labelled BAH test split and 0.7368 on the private ABAW11 leaderboard. This is stated to beat the strongest published baseline (0.634) by more than 10 points, while training in under 20 minutes on one RTX 4090 GPU.
Do the discrepancy features actually help in MDT?
According to the paper's own ablation (Table 3), barely: removing all discrepancy tokens scores 0.7325 Macro F1, which slightly exceeds both the 6-token (0.7219) and 9-token (0.7322) variants. The discrepancy design improves negative-class F1 (0.6700 vs 0.6457) but adds little on the headline metric; the largest single gain comes from LoRA fine-tuning (0.7366).
Is there code or data available for this paper?
The paper does not mention any code, checkpoint or configuration release. The BAH dataset comes from the 3rd ABAW Challenge, so data access depends on the challenge organisers; independent verification of the private leaderboard score (0.7368) is not possible from the paper alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figure 1 (architecture overview) is not analyzable from extracted text; descriptions rely on §3 prose.
- Table cell alignment in the extraction is reconstructed from inline numbers; a few decimals were cross-checked for consistency but OCR artefacts are possible.
- The private leaderboard score (0.7368) cannot be independently verified from the text.
- Reference [20] (SamLowe/HuggingFace model card) is cited in the reference list but does not appear in the body text; minor citation hygiene issue.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.