MM-IFEval-Pro: A Multilingual and Attack-Resistant Benchmark for Instruction-Following in Vision-Language Models
Changming Xiao, Zhenliang Ni, Jinhui He, Han Shu, Jie Hu
cs.AI
Quick verdict
Claims a bilingual, attack-resistant multimodal instruction-following benchmark plus RL set; solid design, but generalization claims exceed its own tables.
Summary
The paper introduces MM-IFEval-Pro, a bilingual Chinese-English multimodal instruction-following benchmark with 1,279 test samples spanning 4 task categories (24 subcategories) and 8 constraint categories (52 subcategories), averaging 3.0 constraints per sample. All constraints are scored by deterministic rule-based verifiers rather than LLM judges, motivated by experiments showing judge-dependent ranking flips of several points (Figure 1). A dedicated visual instruction hijacking dimension renders readable fake tasks into images while the genuine query asks for OCR transcription, scored by normalized Levenshtein distance below 0.1. A 13,547-sample RL training set is built with the same pipeline; full-parameter GRPO lifts Qwen3-VL-8B-Instruct from 78.73 to 89.31 and InternVL3.5-8B from 64.64 to 87.55 on the benchmark average (w/ IH), with moderate transfer to MM-IFEval, MIA-Bench and text-only IFEval. Hijacking sensitivity varies widely across six models: MiMo-VL-7B drops from 71.40 to 64.70 while MiniCPM-V4.5 rises from 73.40 to 76.80. Main limitations: no significance testing or seed variance; the text claims InternVL gains across STEM and document tasks while Table 2 shows its average falling from 63.65 to 63.44; augmentation data may overlap evaluated benchmarks; the hijacking subset covers only OCR-style queries; no dataset URL is given.
Review complete
Assessment at a glance
6
risk flags identified
● high 2
● medium 3
● low 1
4 strengths · 5 weaknesses · 5 author questions
AI text signal = Binoculars zero-shot detector score over 20 sampled passages (Fast-DetectGPT co-reported) — absence does not prove human authorship, and a flag does not imply misconduct.
Levels reflect verification complexity — not a quality score. Evolve never rates your paper.
Key claims under review
- Existing multimodal instruction-following benchmarks lack Chinese coverage and adversarial scenarios, motivating a bilingual benchmark with rule-verifiable constraints.
- MM-IFEval-Pro provides 1,279 test samples (avg 3.0 constraints) across 4x24 task types and 8x52 constraint types, including visual instruction hijacking as a first-class, rule-verifiable category.
- GRPO training on the 13,547-sample companion set raises Qwen3-VL-8B-Instruct from 78.73 to 89.31 and InternVL3.5-8B from 64.64 to 87.55 on the MM-IFEval-Pro average (w/ IH setting).
- The training gains transfer moderately to MM-IFEval, MIA-Bench and text-only IFEval, and do not degrade Qwen's general multimodal performance.
- Hijacking robustness varies sharply across models, making it a discriminative evaluation axis.
Strengths
Deterministic, rule-based scoring is well motivated and consistently executed
Figure 1(a) shows judge-dependent flips of 7.8-9.5 points on MIA-Bench (MiniCPM-V4.5: 91.5 under gpt-4o vs 83.7 under gpt-5.2; Qwen3.5-35A3: 92.7 vs 83.2); every constraint is paired with an executable verifier, enabling stable comparison and clean RLVR rewards (fraction of satisfied constraints).
Language-aware bilingual design goes beyond translation (Section 3.1)
A Language Support field gates constraint-language pairings at construction time, and paired CN/EN verifiers encode real linguistic differences (whitespace words vs Chinese characters with atomic digit spans; language-specific sentence segmentation) — a genuine gap, since CFBench relies on checklist LLM judging and M-IFEval lacks Chinese-specific rules.
Visual instruction hijacking is discriminative and cheap to verify
Internally consistent dataset accounting and a fully disclosed training recipe
Weaknesses
Direct text-vs-table contradiction on InternVL generalization
No statistical uncertainty anywhere
Contamination pathway between training augmentation and evaluated benchmarks
The hijacking subset conflates OCR ability with attack resistance
Release claims without any URL; missing ablations and unspecified generators
Full Review Report
$1.99 · one-timeUnlock the complete preliminary analysis Evolve produced for this paper.
Report preview
Scope & Claims
MM-IFEval-Pro (Huawei; arXiv 2609.04859v2) is a benchmark-plus-training-resource paper. Its claims: (1) existing multimodal instruction-following (IF) benchmarks lack Chinese samples and adversarial coverage; (2) the proposed benchmark — 1,279 test samples, 4 task categories / 24 subcategories, 8 constraint categories / 52 subcategories, 3.0 constraints per sample — fixes this with fully rule-based verification; (3) visual instruction hijacking (IH): readable fake tasks embedded in images, is elevated to a first-class rule-verifiable dimension; (4) GRPO on a 13,547-sample companion RL set lifts Qwen3-VL-8B-Instruct from 78.73 to 89.31 and InternVL3.5-8B from 64.64 to 87.55 (benchmark average, w/ IH), transferring to MM-IFEval, MIA-Bench and IFEval. Most claims are internally supported; one prose-vs-table contradiction is not (below).
The remaining sections cover Methodology Assessment, Statistical Evidence, Results & Reproducibility, Limitations & Risks, and Verdict & Recommendations.
- All 4 strengths & 5 weaknesses, analyzed in detail
- 6 risk flags with severity, section & reviewer questions
- Methods & statistical evidence assessment
- 5 questions a peer reviewer would ask
- Full six-dimension evaluation report & recommendations
One-time purchase · this paper only
One-time purchase, this paper only. AI-generated preliminary analysis — not peer review.
Frequently asked questions
What is MM-IFEval-Pro?
A bilingual Chinese-English benchmark for multimodal instruction following in vision-language models: 1,279 test samples with an average of 3.0 constraints each, spanning 4 task categories (24 subcategories) and 8 constraint categories (52 subcategories), all scored by deterministic rule-based verifiers instead of LLM judges, plus a dedicated 'visual instruction hijacking' dimension where readable fake tasks are embedded in images.
What is visual instruction hijacking in this paper?
An adversarial setting where an image contains clearly readable fake instructions while the actual user text query asks for something else — here, OCR transcription of the image text. A robust model should transcribe rather than execute the embedded fake task; success requires the normalized Levenshtein distance between the response and the rendered text to be below 0.1.
How much does RL training on MM-IFEval-Pro improve models?
In the paper, full-parameter GRPO on a 13,547-sample companion set raises Qwen3-VL-8B-Instruct's benchmark average from 78.73 to 89.31 and InternVL3.5-8B's from 64.64 to 87.55, with positive but smaller transfer to the original MM-IFEval (Qwen 58.68→61.73; InternVL 47.13→50.13), MIA-Bench and text-only IFEval; on general multimodal benchmarks Qwen improves slightly while InternVL's average slightly declines (63.65→63.44).
Is the MM-IFEval-Pro dataset publicly available?
The paper states that the 13,547-sample RL training set is released, but the reviewed text contains no repository URL, license, or hosting details, and release of the 1,279-sample test benchmark or verifier code is not explicitly stated — availability should be confirmed with the authors.
About this review
This is an AI-generated preliminary review by Evolve.Pub, produced from the openly available text of the paper. It is not peer review and does not represent the authors' views. The canonical record of the paper is at the publisher link above.
- Table 1 extraction is imperfect in places: the InternVL rows carry two identical trailing values (72.64/72.64; 78.93/78.93) and the MM-IFEval perception column shows suspiciously round values (51, 34, 33). Qwen rows were arithmetically verified (Table 2 averages and Figure 4/6 task counts recompute exactly) and are cited only where they check out; InternVL MIA/IFEval cell assignments in Table 1 are therefore cited cautiously.
- Figure 6(b) training-set constraint-count proportions cannot be read from the extraction (axis ticks present, data values absent). The EN 68.9% / ZH 31.1% split in Figure 1(b) refers to the 13,547-sample training set, not the test set.
- No dataset or code repository URL appears anywhere in the provided text — release is asserted only in prose. The provided text ends after Appendix Tables 4-6 with the reference list and both appendix tables present, so the paper body appears complete; any final pages beyond those tables are not in the extraction.
Go deeper on this paper
Wrote a paper? Stress-test it before reviewers do.
Upload your manuscript for the same structured scrutiny — reasoning, statistics, evidence, figures, and reproducibility.