MIS-Bench: Benchmarking Multimodal LLMs for Psychotherapeutic Interpersonal Skills Assessment
Yuhan Lu, Yi Yao, Hua Shen, Katie Aafjes-van Doorn, Zhaonan Wang
cs.CL
Quick verdict
996-video psychotherapy-skills benchmark shows MLLMs lag human raters; solid expert-annotated design, but withheld raw A/V blocks full replication.
Summary
The paper introduces MIS-Bench, a multimodal benchmark of 996 psychotherapy response videos annotated by trained human raters on eight Facilitative Interpersonal Skills dimensions (1–5 scale, one-decimal expert-mean labels), and uses it to test whether multimodal LLMs can approximate expert judgment. Nine MLLMs (seven open-source, GPT-5.4, Gemini-3.1-Pro-Preview) are evaluated under four modality configurations and direct versus chain-of-thought prompting, scored by ICC(3) and MSE. The best average dimension-level ICC is 38.6% (GPT-5.4), below the 43–64% human single-rater ceiling; multimodal input yields inconsistent gains (though audio clearly helps Verbal Fluency), and direct scoring beats CoT on average (+0.047 ICC; lower MSE in 159/160 comparisons). The proposed MIS-RAFT fine-tunes Qwen2.5-VL with a regression-aware objective computing probability-weighted expectations over integer and fractional digit distributions, raising mean ICC from 0.138 (raw) and 0.027 (standard SFT) to 0.311, with 10,000-resample bootstrap CIs. Main limitations: the public release contains de-identified features and transcripts but not the raw audio/video the models consumed, so multimodal evaluations cannot be rerun from the release; MIS-RAFT is validated on a single backbone without seed variance; and raters saw the patient stimulus while models did not, an information asymmetry that may deflate model agreement.
Review complete
Assessment at a glance
6
risk flags identified
● high 1
● medium 2
● low 3
5 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- MIS-Bench is the first expert-grounded multimodal benchmark for psychotherapeutic interpersonal skill assessment, with 996 clinician-response videos scored on eight FIS dimensions by trained human raters.
- Across nine MLLMs, four modality configurations, and two prompting styles, the best average dimension-level ICC is 38.6% (GPT-5.4), below the 43–64% human single-rater agreement ceiling.
- Multimodal input yields inconsistent gains across models and dimensions, with audio specifically improving Verbal Fluency (mean ICC gain 0.136 across three models under direct prompting).
- Direct score prediction is generally more reliable than CoT prompting (+0.047 average overall ICC; lower MSE in 159/160 matched comparisons).
- MIS-RAFT, a regression-aware fine-tuning objective over integer/fractional digit distributions, raises Qwen2.5-VL mean ICC from 0.138 (raw) and 0.027 (standard SFT) to 0.311.
Strengths
Expert-grounded labels with a human agreement ceiling
Labels come from raters trained to ICC>0.8 against expert ratings on six consecutive clips with weekly calibration (Appendix B); Table 2 reports single-rater ICC(3,1) of 43–64% with 95% CIs on the 625 fully rated videos, giving the model numbers real interpretive context instead of bare scores.
Unusually thorough evaluation grid for this domain
Nine MLLMs × four modality configurations × direct/CoT prompting (Fig. 2 heatmaps cover every supported combination), plus auxiliary Pearson/Spearman/Kendall/CCC/C-index heatmaps and prediction histograms (Appendix D), enabling controlled within-configuration comparisons.
MIS-RAFT is a simple, well-motivated fix with uncertainty reporting
Observations tied to construct validity
Transparent ethics, rubrics, and dataset statistics
Weaknesses
Released data cannot reproduce the multimodal evaluation
MIS-RAFT validated on a single backbone with anomalous SFT baseline
The anti-CoT conclusion leans on MSE while its ICC evidence is near chance
No baseline against prior automated FIS scorers
Structural model–rater information asymmetry
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
The paper makes three claims: (1) MIS-Bench is an expert-grounded multimodal benchmark for psychotherapeutic interpersonal skill assessment (996 clinician-response videos, eight FIS dimensions, §3); (2) nine MLLMs evaluated under four modality configurations and two prompting styles show only modest expert agreement — best average dimension-level ICC 38.6% (GPT-5.4), 33.5% (Gemini-3.1-Pro-Preview), 28.3% (MiniCPM-o-4.5), all below the 43–64% human single-rater ceiling of Table 2; (3) MIS-RAFT, a regression-aware fine-tuning objective (Eqs. 9–12), raises Qwen2.5-VL mean ICC from 0.138 (raw) and 0.027 (standard SFT) to 0.311 (Table 4). The claims are scoped carefully, the Limitations section candidly names the model–rater information asymmetry and the narrow post-training sweep, and the Ethics section explicitly bars deployment use. This is a credible benchmark-plus-method paper whose main exposure is reproducibility, not intent.
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 5 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MIS-Bench?
MIS-Bench is a multimodal benchmark for testing whether multimodal large language models can score psychotherapists' interpersonal skills the way human experts do. It contains 996 videos of clinicians responding to standardized challenging patient scenarios, each rated on eight dimensions of Facilitative Interpersonal Skills (verbal fluency, hope/positive expectations, persuasiveness, emotional expression, warmth/acceptance/understanding, empathy, alliance bond capacity, and rupture-repair responsiveness) on a 1–5 scale. The public release provides transcripts, de-identified audiovisual feature representations, and expert scores rather than raw recordings, to protect participant privacy.
What is MIS-RAFT and how does it improve score prediction?
MIS-RAFT is a fine-tuning method that fixes the mismatch between language models' token-prediction training and the numeric scoring task. Instead of training the model to output the exact score string (cross-entropy, which treats '3.1' as equally wrong as '5.0' relative to '3.0'), it splits each one-decimal score into an integer digit and a fractional digit, computes probability-weighted expectations over the digit distributions, and minimizes the squared error between this expected score and the expert rating. On Qwen2.5-VL it lifted mean ICC with human ratings from 0.138 (untuned) and 0.027 (standard fine-tuning) to 0.311.
How close are GPT and Gemini to human experts at rating therapist skills?
Not very close yet. On MIS-Bench the best model, GPT-5.4, reached an average dimension-level ICC of 38.6% with human experts, and Gemini-3.1-Pro-Preview reached 33.5%; all nine tested models stayed below 40% (all open-source models below 30%). Human single-rater agreement itself ranges from 43% to 64% across the eight dimensions, so even the strongest models remain meaningfully below the human consistency ceiling. Direct score prompting generally worked better than chain-of-thought reasoning.
Is the MIS-Bench dataset public and does it include the actual therapy videos?
The benchmark data and code are released on GitHub, but the raw audio and video recordings are not included for privacy reasons (data were collected under informed consent and IRB protocol 1293523 at Yeshiva University). The release contains transcripts, de-identified feature representations (facial action units, landmarks, gaze, Wav2Vec 2.0 embeddings, prosody, spectrograms, MFCCs), rubric descriptions, and the expert FIS scores. This means the paper's multimodal model evaluations cannot be fully re-run from the public release alone.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures (Fig. 2 ICC heatmaps, Figs. 5–15 correlation matrices, radars, and histograms) are referenced but not analyzable from extracted text, so per-configuration ICC values and label distributions could not be verified.
- Appendix A.1 prompt templates are cited but their bodies are absent from the extraction; only the rubric table (Table 5) is present, so prompt-design quality could not be assessed.
- Closed-source model versions (GPT-5.4, Gemini-3.1-Pro-Preview) are fast-moving API snapshots; exact numbers may not be reproducible even with full data access.
- Formulas and table formatting may contain extraction artifacts; numeric claims were cross-checked against both tables and running text where possible.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.