MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
Luyao Zhu, Xun Wei Yee, Wei Li, Mun Thye Mak, Wee Siong Ng
cs.AI · cs.CL · cs.CV
Quick verdict
12-task artistic-image VLM benchmark (2,400 Qs, 30 models) with annotation-first design; informative, but lacks uncertainty estimates and chance baselines.
Summary
The paper introduces MUSE, a benchmark of 12 tasks and 2,400 multiple-choice, numerical, and open-ended questions over 1,174 newly commissioned artistic images (1920x1080), targeting multi-modal understanding for situated, image-based language education. Tasks span five dimensions - visual perception, semantic understanding, affective interpretation, compositional reasoning, and cultural understanding - with deliberate emphasis on Singaporean/Southeast Asian contexts. Construction decouples reusable structured image annotation from rule-based question generation: 127 student annotators label boxes, emotions, activities, and culture under a one-annotate/two-review consensus, after which task-specific questions and distractors are instantiated programmatically. The authors evaluate 30 open-source and proprietary VLMs (GPT-5.6-Sol, GPT-4o, Qwen2.5/3-VL, InternVL3, Gemma3, LLaVA-NeXT, MiniCPM, DeepSeek-VL2, CogVLM2, GLM-4V, Yi-VL) at temperature 0, scoring MCQs by accuracy and open-ended tasks by embedding cosine similarity. GPT-5.6-Sol leads overall; Scene Classification is near-saturated (median 81.0) while Emotion Detection, Relative Position, Remote Interaction, and Jigsaw Puzzle are weak. Correlation analyses against six external benchmarks argue complementarity; error analyses identify region-activity binding failures, a forced-relation bias in spatial reasoning, and cascading grounding-to-affect errors. Main limitations: no confidence intervals or significance testing, no chance baselines (several scores sit below plausible chance), an unvalidated embedding-similarity metric for open-ended tasks, and no inter-annotator agreement statistics.
Review complete
Assessment at a glance
5
risk flags identified
● high 1
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Decoupling reusable image annotation from rule-based question generation enables 12 diverse tasks with controlled format and difficulty at reduced annotation cost.
- Current VLMs are markedly weaker on artistic-image affective interpretation, compositional reasoning, and visual grounding than on scene and activity recognition (Scene Classification median 81.0 versus much lower medians elsewhere).
- Performance on existing general benchmarks transfers only partially to artistic educational imagery, making MUSE complementary to MMBench, AI2D, BLINK, and AICA-Bench.
- Grounding failures cascade into affective errors, and models show a systematic forced-relation bias in viewpoint-dependent spatial reasoning (90.0% and 73.3% assert definite lateral/vertical relations when ground truth is none).
Strengths
Scalable annotation-first construction
Decoupling reusable structured annotations from rule-based question generation (Section 3.2) yields 12 tasks x 200 questions from a single annotation pass, with explicit distractor taxonomies (10 negative-option methods for Activity Description) and explicit difficulty/granularity controls, improving cross-task semantic consistency.
Contamination-resistant, regionally distinctive data
All 1,174 images were newly commissioned from freelance artists (~USD 36/image) centering Singaporean/Southeast Asian multicultural contexts (37.5% of culture labels, Figure 4), mitigating web-scrape leakage for proprietary models and filling a geographic gap left by Western-centric art benchmarks.
Broad and well-documented evaluation
Analysis beyond the leaderboard
Weaknesses
No statistical uncertainty or significance testing
Missing chance baselines; near-floor scores may reflect format failure
Open-ended scoring metric unvalidated
Annotation quality not quantified
Educational validity and cross-task dependency unexamined
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 5 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is the MUSE benchmark and what does it evaluate?
MUSE is a benchmark for evaluating large vision-language models on understanding artistic images in educational settings. It contains 12 tasks and 2,400 questions built over 1,174 newly commissioned artworks (1920x1080), covering five capability dimensions: visual perception, semantic understanding, affective interpretation, compositional reasoning, and cultural understanding, with a deliberate focus on Singaporean and Southeast Asian multicultural content alongside Western art.
Which vision-language models perform best on MUSE?
In the paper's evaluation of 30 models, GPT-5.6-Sol achieves the strongest overall results, beating GPT-4o on 10 of 12 tasks. Among open-source models, Qwen3-VL-32b leads several tasks such as Activity Localization and Scene Classification. Scene Classification is the easiest task (median score 81.0 across models), while Relative Position, Emotion Detection, Remote Interaction, and Jigsaw Puzzle are the hardest, with even the best models far below human performance.
Why does MUSE use artistic images instead of natural photos?
The authors argue that image-based language learning often uses paintings, illustrations, and cartoons, which contain stylized forms, non-photorealistic colors, implicit narratives, and culture-dependent cues that natural-image benchmarks do not test. Using newly commissioned artwork also reduces the risk that models have memorized the images during training, which is a concern when evaluating proprietary models.
Is the MUSE dataset and code publicly available?
The paper includes Code and Dataset links in its header, indicating intended public release. The extracted text does not contain the actual URLs, so availability should be verified on the paper's arXiv page (arXiv:2609.19088). The paper does document inference configurations, prompts, and annotation procedures in its appendices to support reproduction.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Figures are available only as extracted text; correlation heatmaps (Figures 6-7), error-type distributions (Figure 8), and human-comparison plots (Figures 11-13) cannot be fully verified, and some in-figure values may be garbled (e.g., Figure 9 legend says InternVL3-78b while Table 2 lists InternVL3-38b).
- Code and dataset URLs are not present in the extracted text; availability is claimed via header links but unverified.
- Question templates and scoring details are only partially shown, so judgments about scoring implementation (e.g., Relative Position scoring units) are inferred from the task descriptions.
- Author affiliation and corresponding-author markers appear partially garbled in the extraction, though authorship itself is clear.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.